Stark Raving Finkle

Agentic Engineering: What Agents Can Safely Own

Sun, 30 Aug 2026 00:00:00 +0000

Two posts back, I used Kent Beck’s 3X framework to argue that agentic engineering gets read at different phases of maturity depending on how far you sit from the code. Last post, I argued that more code isn’t enough. Agents compress execution to nearly free, but judgment is still as critical as ever: deciding what to build, owning intent, understanding the result well enough to steer the next loop.

When the cost of coding collapses, lots of other activities become visible. Some of what’s left is critical and shouldn’t be outsourced: judgment, most of all. The rest is process we built up when writing code was the expensive part and a lot of it was never questioned because the coding cost dwarfed it. That’s the part to re-examine.

Will Larson landed in the same place from the org side: very little work is genuinely constrained by execution bandwidth anymore. It’s constrained by approvals, by handoffs, and most of all by missing decisions. His prescription is a roadmap of decisions, not dates — stop scheduling software around calendars, and start accelerating the decisions the work is actually waiting on.

If that’s right, the highest-leverage move isn’t making agents faster. It’s growing the share of work whose decisions are already closed. The work an agent can pick up and run without stopping to ask what you meant. That’s the work that ships in a day. Everything else waits on human decision latency, exactly like it always did. Perhaps that’s the bottleneck we should examine.

Let’s focus on a narrow, practical question: what makes a piece of work agent-shippable, and how do you tell before you hand it over? The failure mode here isn’t an agent that writes bad code. It’s an agent that confidently runs work that was never ready — at volume. Which means the first act of judgment isn’t reviewing the output. It’s deciding what should have been allowed to run unattended in the first place.

I’m in the middle of exploring this at work: a real triage, on a real backlog, with real arguments about which traits actually matter. We don’t have a playbook for this yet, so I’ll revise and update as we go. We’re starting with a set of criteria we’re testing against the work.

The Right Conditions Aren’t New

Same-day start-to-deploy was always possible, but only for work with a specific profile:

Clear requirements, or a problem the developer already understood
Small scope
An unambiguous definition of done — no second-guessing
The developer already held the model of the code — the frameworks, systems, and patterns
The developer had everything needed to build, test, and verify

Every one of those is a proxy for the same thing: reducing the person’s thinking-and-typing load. The constraint before agents was code generation: how fast one person could hold a problem in their head and produce correct code. All five traits focus on that constraint. Small scope shrinks it. Clear requirements remove upstream thinking. Existing context removes ramp-up.

Two things were bundled inside that one person, and they’re the whole reason the profile is changing:

The generator was also the decider. Small judgment calls got made in-flight, at zero latency, by the same person writing the code. Nobody wrote them down, because nobody had to.
Verification was partly internal. An experienced developer carries a running “this looks wrong” sense; formal checks only had to cover the gap that instinct didn’t.

Give code generation to an agent and those two split apart. The in-flight decisions now have to be written down as intent or context, because the agent won’t make them the way you would. The internal “this looks wrong” has to be externalized as mechanical checks, because the agent doesn’t have it. This is why the intent and verification pillars from the last post stop being nice-to-haves. They existed in someone’s head, and now they can’t.

When a developer holds the model of the code, it’s really two things. The first part is judgment: knowing, from principle, which approach is right when the situation is new. That stays with the developer. The other part is patterns: knowing how this codebase does a thing, what the established shape of a change looks like here. Patterns are extractable. Judgment isn’t.

Which makes Patterned a first-class condition that we can leverage with agents. The second instance of anything is cheap, because now there’s a pattern to copy; the first instance is where the judgment comes into play. So build the first instance deliberately and slowly, then fast-track the rest. The first one is you deciding the shape, the rest is the agent matching it.

An agent can go exactly as fast as your feedback loop allows.

That’s also why the same model produces excellent output in one repository and slop in another. Agent-shippability is a property of the codebase, not of the model. It’s something you build, not something you buy. Which is also why swapping harnesses isn’t the silver bullet.

What Qualifies Now

A unit of work can be started and deployed in a day by an agent when it’s:

Decision-closed. No judgment calls remain in-flight, or they’re pre-encoded as written policy the agent can consult.
Mechanically verifiable. “Done” is checkable by a machine, not by taste, not by a human reading the diff carefully.
Blast-radius-bounded. Cheaply reversible, behind a flag, an additive migration, an isolated module.
Context-resident. Everything the agent needs will physically fit in what it can take in.
Patterned. The change matches a shape already in the repo, so the agent has examples to copy rather than a principle to work out.

Those five are all properties of the work. There’s one more, called out in Larson’s post, that’s a property of what stands around the work: approval-clear, meaning no human-latency gate sits between merge and deploy. A change can be decision-closed and patterned and still not ship today because it’s waiting on a security sign-off or another team’s review.

This list is an all-or-nothing requirement. Work must clear every single trait to qualify, not just hit a good average. One specific pairing often trips people up because two of these traits seem completely unrelated. Cognitive load and blast radius are independent. Adding an index to a 400-million-row table, or changing persisted data, could be trivially simple: minimum thinking, maximum clarity — and also a one-way door. “It’s a small change” is an argument about cognitive load; it says nothing about whether you can revert it. Decision-closed and patterned can both be true while blast-radius-bounded is flatly false. The work that feels safest to wave through is often exactly the work that isn’t.

Ambiguity Is a State You Can Change

Reading the list and it sounds like work either qualifies or it doesn’t. Sort the backlog into two piles and hand one to the agents. But the most valuable work almost never starts in the qualifying pile. Interesting work is ambiguous work: the decisions aren’t closed, the definition of done is soft, nobody’s holding the full model yet. If you only ever fast-track what’s already decision-closed, you’ve capped the payoff and it’s usually the least interesting work you have.

The trap is treating decision-closed as a fixed property the work. It isn’t. It’s a state you can move work into, and the move is cheaper now than it’s ever been.

Larson calls this the iterative application of judgment: you don’t resolve a hard, ambiguous decision by thinking harder about it, you resolve it by prototyping until the options collapse. The answers get obvious not because you got smarter but because a working artifact makes the constraints explicit and eliminates the paths that don’t survive contact with them. One big fuzzy decision becomes a series of small clear ones. That’s decision latency attacked at the source, not by deciding faster, but by making the decision easy.

Prototyping was always the right answer to ambiguity. It was just expensive enough that most teams didn’t have time for it. That’s the part that changed. I’ve written before about how agents turn a multi-week probe into an afternoon one. Discovery phases that used to produce documents now produce working scaffolding, and a POC can unblock an architecture decision in days instead of weeks. When a prototype costs an afternoon, “prototype until the decision closes” stops being a luxury and becomes the default way you qualify ambiguous work.

So there are two ways a unit of work becomes agent-shippable. It was already decision-closed, rare, and usually the boring stuff. Or you spend a person’s judgment up front, cheaply, to close it: prototype the ambiguity, encode what you learned as the constraints the agent will follow, and then fast-track the rest. It’s the same as creating a pattern, except here the first instance is the thinking, not the code.

The share of your backlog that qualifies isn’t a number you measure. It’s a number you grow by spending cheap, up-front judgment to move work across the line.

One caveat, because this is exactly where enthusiasm gets people hurt: a prototype that closes the decision is not the thing you ship. The artifact you built to think with hasn’t been verified, isn’t reversible, and hasn’t earned the other traits just because it answered the question. Closing the decision qualifies work to enter the fast path — it doesn’t let it skip the path. The moment a POC that “works” gets rushed straight to production is the moment you’ve traded the discipline for speed.

The Number Worth Growing

Post one made fun of “percentage of code written by AI” as a KPI — a gameable metric that inflates the exact review burden that’s already the bottleneck. Here’s a metric that isn’t gameable: qualification rate — the fraction of incoming work that clears the bar to run agent-shippable, trending up over time.

You can’t fake it by generating more code. The only way it moves is by doing the work underneath it: closing decisions before they reach the agent, building the first instance so the next ten are patterned, tightening verification until “done” is machine-checkable, keeping blast radius bounded. Every one of those is a piece of judgment spent up front.

It’s also a decent answer to the executive question from that first post: is this working? Not “how much code did agents write,” but “how much more of our work is now the kind of work an agent can safely own than was six months ago?” That number going up is a real transformation.

Once you can classify work, and grow the share of work that qualifies, you need a system that actually routes it. Send qualified work onto an agent-fast path, everything else onto the slower, more mindful one, with judgement and decisions. That’s how we get to really start Expanding and Extracting value.

Agentic Engineering: More Code Isn't Enough

Sat, 15 Aug 2026 00:00:00 +0000

In my last post using Kent Beck’s 3X Framework, I argued that agentic engineering is being perceived at different phases of maturity depending on where you sit. Executives believe the systems are mature, in Extract. Practitioners are doing the work of Expand. The practice, the tools, the workflows are still churning in Explore.

Underneath that churn, there are four foundations we know provide value:

Context: What the agent needs to know about the situation — the codebase, the coding patterns, the how-tos for the frameworks and systems we already have.
Intent: What we want the agent to do and why. The clarity of the intention drives the quality of the outcome. Garbage in, garbage out. So much of developer craft is moving to this process.
Code generation: Turning context and intent into working code. The most obvious win, especially with developer-steered agentic coding sessions.
Code review and approval: Checking that the generated code is correct, consistent, and safe to merge. Agentic code reviews are very good, but removing people from the process creates anxiety.

Each of those made it through Explore and these are the baseline for any engineering organization. Developers are definitely able to Extract value and deliver work faster using these foundations.

If you aren’t succeeding with developer-steered agentic coding sessions, you aren’t ready to move on to the next level. Even if you are successful at creating more code, you still need to figure out how to turn that increased code into increased value.

It’s tempting to head toward the “dark factory” model: agents running 24/7, writing more and more code. But ask the pre-agentic version of that question: if we hired three times as many developers and set them loose writing code, would that create three times the value? We already know it wouldn’t. Code production was always the easy part to blame for not increasing business value. Agents removed code production as the bottleneck, so now you more easily see how planning, decisions, judgement, and understanding are the real bottlenecks.

Get Good at the Basics

Getting good at the four pillars with developer-steered sessions — a human in the loop, driving — is a critical milestone and shouldn’t be skipped. Context that yields code matching your patterns. Intent sharp enough that the agent builds the right thing. Generation you can steer. Reviews, human and agent, that catch what your experts would.

It’s the org-level version of something from the last post: autonomy is earned, not granted. If developer-steered coding isn’t reliably producing work you’d actually ship, then dark factories and features-in-a-day are fantasies. You’d only be scaling something that doesn’t work yet and yields no value. You’re ready when a developer and an agent reliably turn a clear intent into shippable, reviewed work and the developer still understands what came out — not when the agent simply produced a lot. Once you hit this milestone, you’re able to separate the hype from the actual work needed to level-up.

The Part Agents Can’t Do

Say you’ve got all four pillars humming. An agent can now execute every one of them. It still won’t produce business or customer value on its own because the value was never purely in the code.

Writing code was always the smaller part of the job; one line of research puts developers at roughly 14% of the week actually writing code. Agents took over that slice and made it nearly free. Deciding what’s worth building, shaping why, judging whether the result is right and what it means for the next move — that is the part that turns code generation into value, and it’s the part that doesn’t really automate.

Three activities still depend on people, and they’re all the same shape:

What to build. Direction. An agent optimizes toward a target it’s handed; it doesn’t decide the target is worth hitting.
Driving intent. The clarity of the why drives the quality of the outcome. You don’t hand off authorship of intent, you participate in it.
Understanding the result well enough to steer the next loop. Not a rubber stamp on generated code. A working model of the system, at the architectural level, good enough to decide what happens next.

Some people really want to ignore or downplay the importance of understanding. Geoffrey Litt describes understanding as a new bottleneck: there are two reasons to understand an agent’s code. You can understand it well enough to evaluate (thumbs up, thumbs down) or well enough to participate, to hold a model of the system that lets you decide what comes next. Agents are getting good at supporting evaluation. Participation is the part that keeps you valuable, because a project is never one loop, it’s many. Participation is what lets you consider feedback from the existing loop and drive the direction of the next one.

Evaluation-only comprehension feels like rigor and good process, but we know that it becomes mechanical especially when under time pressure or volume. You can approve a correctly compiling diff every day and understand the system a little less each time. Margaret-Anne Storey names what accrues an “accumulation of not knowing” — cognitive debt, and it’s the debt you can run up while every check stays green. The model in your head becomes less clear, and at some point you can’t drive intent anymore because intent is more than “is this correct,” it’s “why this approach should be used and not the alternatives.”

What This Sets Up

So “more code isn’t enough” is a constraint statement. Agents made generation nearly free, and none of the critical thinking parts got faster or easier. Don’t confuse moving code faster for progress if it doesn’t create value for the business or its customers. It’s just code churn.

Even the people whose job is measuring this draw the same line. DX frames the goal as capacity, not horsepower, and is careful to wall off engineering capacity from business value, noting the far side is often not an engineering question at all. This post is about that far side. The judgment that turns capacity into value is exactly what an agent can’t do for you.

That human work isn’t a tax on velocity. The output of the high-judgment work is what makes agent volume safe and valuable in the first place. The decision making, pattern creation, and system understanding have clearly become the true craft. Understanding compounds and builds into value throughput.

That leaves three questions: what qualifies for the fast path, what has to stay on the slow one, and what we owe the people who need to keep understanding a system they’re no longer reading line by line.

Agentic Engineering: Where's the Playbook?

Sun, 19 Jul 2026 00:00:00 +0000

Agentic engineering is happening at a breakneck pace, and many are wondering if they are making the “right” decisions and using the “right” approaches. I’ve been a proponent of not chasing the newest fad, but to instead learn about what’s being used across the industry and keeping your options open.

I was listening to a recent Pragmatic Engineer podcast, where Gergely was interviewing Kent Beck. Kent has spent decades helping to shape software development through a variety of methodologies. Toward the end of the podcast, Kent describes our current situation with agentic development: We don’t have a playbook yet.

We don’t have a playbook. But we do have a framework for what it feels like to not have one, and Beck wrote that too.

The 3X Framework

One of the frameworks Kent created while working at Facebook is called 3X: Explore, Expand, Extract:

Explore: The search for product-market fit. Success requires cheap, rapid experimentation to generate many ideas, tolerating failures, and putting on a “hacker” hat to minimize the time from question to answer.
Expand: The hypergrowth stage. When a successful growth loop is found, focus on scaling up and overcoming bottlenecks. Fast, temporary fixes are valued here to survive the rapid scale.
Extract: Maximizing value and ROI. The product matures, and focus shifts to rigorous engineering, optimization, increasing margins, and efficiency.

The industry has been running an Extract playbook for software development for the last 20+ years, and running it fairly well. Agentic coding wipes some that playbook away. People who were very good at running that playbook are having an identity crisis.

Beck says we’re back in Explore, and we’re beginning to write a new playbook. He also makes this great point:

The skill of writing a playbook is completely different than the skill of applying a playbook.

Beck starts applying 3X to agentic engineering in the podcast, but he doesn’t go deep. I wanted to think more about it, because I think it explains why this moment feels so disorienting, and why we’re churning through new tools without making any actual forward progress.

Reframing 3X for a Methodology

In product 3X, the thing you’re searching for is a growth loop: something that compounds with customers. In methodology 3X, the thing you’re searching for is a transferable practice: something that still works when someone other than its inventor does it, on a codebase they didn’t write.

Explore is practice discovery. Lots of small bets, high tolerance for failure, and observations are legitimate evidence. “It worked for me” is a real signal. The cost of a failed experiment is small, and we’re running a lot of experiments.

Expand is diffusion. Taking the discovered practice and scaling it to the rest of the engineering organization. The bottleneck is tribal knowledge — and the deeper problem is that some of it can’t be written down at all. The people who invented a practice are exercising judgment they can’t fully articulate. So what gets copied is the artifact, not the judgment behind it.

Extract is codification. Standards, metrics, compliance, efficiency. The sign of Extract is boring — Extract’s own playbook tells you to choose boring technology. The debates over git, CI, or code review are over. People still write about them, but it’s optimization at the margins, not a rethink of the core practice. When agentic engineering reaches Extract, we’ll stop having conferences about it. Maybe.

Nobody Agrees What Phase We’re In

What makes our current moment more confusing is the fact that people are not aligned about the phase state. The point of view changes by how far you sit from the actual work.

DX ran a panel at their annual conference, and they were direct about where the questions came from: assumptions circulating in executive teams and boardrooms that are shaping strategy right now. The panelists — engineering and research leaders from Etsy, Twilio, GitHub, Google, and Microsoft — were read a series of statements and asked to react.

Here are the statements:

An AI-first SDLC means fewer engineers.
In five years, more than 50% of code will be written by AI.
Leaders need to mandate AI usage to make sure adoption is moving along.
The future of software engineering is managing agents rather than writing code.

Headcount. Volume metrics. Mandates. Role redefinition. Every one of those is an Extract-phase question. Not one of them asks what works, or how to spread it. They all assume that’s settled and ask how to squeeze value out of it.

You can read what the practitioners said back:

Eirini Kalliamvakou couldn’t answer the “managing agents” statement without rewriting it first. The real work, she said, is defining intent, setting constraints and guardrails, providing context, and verifying output: engineering at a different level of abstraction. That is a list of Expand-phase work.

Rafe Colburn on mandates: the push usually comes from outside engineering, from people afraid of falling behind, and it produces shallow adoption. Brian Houck, who is running a study of around 600 engineers and managers, found their single biggest disagreement is that most engineering managers think AI usage is a reasonable individual performance metric — and engineers don’t.

The practice itself is still wide open. We don’t have settled answers on context management, spec-first versus code-first prompting, subagent decomposition, when to let an agent run versus when to interrupt it, or which model to use for which task. We don’t even have a settled answer on how to write a prompt: one study found that semantically equivalent rewrites produced different code 46% of the time, and changed correctness in 28%.

We don’t have best practices yet.

Executives are in Extract. Mandates, metrics, headcount, margin.
Practitioners are doing Expand. Context and knowledge management, guardrails, agent-assisted review & approval, unattended agentic workflows.
Much of the practice is still Explore. Nothing is settled, including how to talk to the thing. We know there is something valuable here, but can’t quite realize the full potential.

The further you are from the code, the later in 3X you think we are.

This is a structural problem. Phase perception is a function of distance from the work — an executive genuinely cannot see the Explore-phase mess, because all they ever see is the output. Some gap between the C-level and the work streams is normal in any transformation. What’s abnormal right now is how wide the gap has become, because the ground moves every week and the layers are drifting apart faster than they can talk to each other.

Executives Rush to Extract

Sooner or later someone asks the question: Is this working?

It’s a fair question, but because most organizations haven’t set up an evaluation loop or a baseline, it’s difficult to answer. When you can’t prove a practice is working, you reach for the only instruments you’ve been trained to use: Metrics. Mandates. Capital-letter Frameworks.

People start using “percentage of code written by AI” as a KPI — a very game-able metric. Bill Gates supposedly said: measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs. Volume metrics simply inflate the exact code review burden that is already our biggest bottleneck. I don’t value solutions implemented with large amounts of code, so why would I be happy with agents that do?

Premature Extract isn’t a separate mistake, it’s a symptom. Twenty years of Extract muscle memory, firing at a problem it can’t see clearly from where it’s standing.

Practitioners Are Stuck in Explore

While executives might be quick to assume agentic engineering is in the Extract phase, we have another problem: Practitioners frequently focus on implementation details, like MCP, Skills, and Harnesses — rather than doing the work to stick with a fundamental concept and move it through Expand.

We have strong evidence that agentic code generation, code reviews, and even code approvals can create tangible benefits. We have seen evidence that contextual knowledge is a huge benefit to agents. Instead of leaning into those concepts, we tend to churn through a variety of implementation experiments that try new technical approaches, but don’t move the core concept forward.

I’ve written about my own experiences with agentic code generation, and how those outcomes yielded 2× to 7× delivery compression, PR cycle time dropping, and experimentation rates increasing. The next steps have to be pushing those outcomes to create business value, not just engineering value. Sometimes it’s more fun to play with the implementation details and new tools.

Here’s what we see teams doing today:

Wiring agents into CI/CD pipelines, IDEs, and ticketing systems. Connecting agent harnesses into your existing SDLC workflow.
Rapid scaffolding, loose security boundaries, and a high tolerance for flaky agent behavior. Velocity is the priority: get the agent opening PRs and writing tests autonomously.
Aggressive adoption of codebase-wide editing agents, automated PR reviewers, and autonomous bug-fixers, to see how much work can be offloaded.

That list shows a desire to move core foundational aspects of agentic engineering into the Expand-phase. We should be looking for bottlenecks as we scale more of the practices into production-level work. The way we’re going about it feels very loose and experimental.

We adopted Expand-phase goals while keeping Explore-phase discipline.

The tools outran the practice. I have to remember that people like myself fueled some of the disconnect: we handed executives the numbers that made them think the work was done.

A recent NBER paper traced AI’s gains through the entire delivery pipeline — lines of code, to commits, to pull requests, to repos, to releases — across more than 100,000 GitHub developers. The gains at the top are enormous, and they grow with each generation of tooling: roughly +40% more commits from autocomplete, +140% from interactive agents, +180% from autonomous agents.

Then they follow those gains downstream through the SDLC. The effect on software actually released tops out around +30%.

180% going in. 30% coming out.

This is true with my own numbers too. Delivery compression, PR cycle time, experimentation rate — all measured at the front of the pipeline. Everything downstream of the merge is barely agentic or automated. The gains pile up at the front of the pipeline and then hit a wall.

Explore Is Not Permission to Be Sloppy

You might think that “just try stuff” means “rigor is optional”, but it’s not.

Explore has its own discipline, but it’s specific to the phase: Cheap to try. Easy to reverse. Comprehended and actually evaluated.

The evidence standard shifts by phase, and this is what people are getting wrong. “It worked for me” is sufficient in Explore. It’s dangerous in Expand. It’s useless in Extract. Most of the arguing happening right now is people applying one phase’s evidence standard to another phase’s claim.

Explore gives you permission to fail. It does not give you permission to not know whether you failed.

Autonomy Is Earned, Not Granted

An agent can only run unattended over work whose correctness can be proven without a human looking. So the path to autonomy isn’t “trust agents more.” It’s “make more of our work machine-verifiable.” Every gap you close turns a class of work into something an agent can safely own.

A phase isn’t a decision you make, it’s a state you demonstrate: Can this practice be verified without you standing over it? Can someone who didn’t invent it run it on a codebase they didn’t write?

So what does earning it look like? It means taking the core pillars of agentic engineering that we know work — management of context, specifying intent, agent code generation, agent code review/approval — and building the actual infrastructure to scale them safely.

Here is what we’re doing to move those pillars from loose exploration into disciplined expansion:

Context: Improve and evolve the context we provide to agents to allow them to perform better. Generated code should be simple, concise, and matching the coding patterns we expect from developers. Code reviews should look for the same issues and checks we expect from our expert developers.
Intent: Provide clear and concise intention of the outcome we want from the agent. The sharper the intent, the faster we get the desired outcome. Capture the gist of what’s been created to deal with cognitive debt as agents write and review more of the code.
Code Generation: Move beyond developer-steered sessions and allow the agent to run a subset of tasks unattended. This requires agent-verification of the work. If an agent can’t verify the outcome, you can’t trust the outcome. You’re not scaling code generation effectively.
Code Review/Approval: We’re already using agentic code reviews. Start scoring each change by blast radius, confidence, and risk. High confidence & low-risk merges itself; everything else goes to a human reviewer.

None of this is a playbook. It’s a set of bets, some of which will look naive in a year. Every one of them is an attempt to earn the next phase rather than trying to skip ahead.

We are being asked to pay out a maturity dividend on a practice that hasn’t matured yet. The cure isn’t a better executive metric; it’s closing the distance. Leaders who actually use these tools on real work stop misreading the phase, because they can finally see the mess.

If we want to write the new playbook instead of just blindly applying an old one, we have to stop treating understanding as a side effect.

Autonomy is earned, not granted.

Next up: Taking those four pillars through Expand; What breaks first in each one, and why understanding turns out to be underneath all of them.

Writing a Field Guide to Agentic Engineering Transformation

Thu, 14 May 2026 00:00:00 +0000

We’ve been building a more agentic engineering organization at League over the last ten months. I’ve been writing blog posts about different aspects along the way. In April, I felt like we’d crossed a threshold.

A significant part of the engineering organization was not only using coding agents in daily workflows, the teams themselves had redesigned their operating models. Not all teams were doing things the exact same way, but many of them had made deep changes in their sprints, rituals, and processes.

We saw meaningful changes in delivery velocity — a 2× baseline on committed milestones, with high-pattern work like migrations and refactors compressing much further. We saw lots more experimentation, and delivery on those experiments. We saw an appetite to take on work that would have been avoided in the past.

It felt like a real success and worth writing about. What we did, what worked, what didn’t.

Field Guide to Agentic Engineering Transformation

I believe software development is mostly a sociotechnical endeavor, so I focus more on the people than the technology. I believe a successful agentic transformation has everything to do with the people and culture. You’ll find a lot of that in the field guide.

I wanted to write the field guide to help other companies going through the same type of transformation. A lot of the information written about agentic engineering focuses on the tools, or worse, the hype. I wanted to provide something that was more grounded.

We’ve crossed a threshold, but we’ve not really finished yet. We have the operational challenges you’d expect when engineering gets meaningfully more productive. We’re working to improve testing automation, local development friction, code review processes, and deployment automation. We’re looking at how we create requirements and designs.

We’ll try some things, learn some things, and I’ll keep writing about it.

The Agentic Coding Payoff: Experimentation Velocity

Tue, 07 Apr 2026 00:00:00 +0000

Most teams measure AI impact on software delivery the same way: before and after comparisons on tasks that were already in the plan. A migration that used to take three weeks now takes two days. A bug that took a week to diagnose gets resolved in an afternoon. These are real wins.

But they’re measuring the floor, not the ceiling.

Speed on planned work is a productivity gain. The ability to attempt unplanned work is a strategic one. A team that ships 20% faster will be faster, sure. A team that runs ten experiments a quarter instead of two will, over time, find things the faster team never looked for. Both matter, but only one of them compounds over time.

In my last post, I argued that coding agents don’t replace platforms — they make building blocks, guardrails, and durable systems more important. That post was about what agents need to work well. This one is about what becomes possible once they do: a shift from delivery speed to experimentation velocity.

What Experimentation Velocity Looks Like

When the cost of trying something drops far enough, people try things they never would have planned.

Our engineering teams have been running structured agentic hackathons and sprints — short, intense blocks where engineers work on projects, with AI agents as full execution partners. What comes out of those sessions isn’t only incremental improvement on planned work. It’s entirely new bets that only exist because the cost of placing them collapsed.

Some of the work included:

POC for managing human & agent interactions: Unblocked the architecture decision in days instead of weeks.
Frontend analytic event migration project: Turned a discovery task into a full delivery.
Backend service: 2-month schedule pull-forward on a foundational platform service.
Product team: Effectively delivered one full quarter of roadmap ahead of schedule.
Another product team: 6-week schedule pull-forward on a major feature, at reduced capacity.
Yet another product team: A backlog item that had no path to delivery got shipped because AI created spare capacity.

These weren’t just shortcuts on existing work. They were bets no one would have made when each one cost weeks of effort.

Once we started looking for this pattern, we saw it happening more and more. Not because we gave people more time, but because each project went from a multi-week investment to an afternoon experiment. Discovery phases started producing working artifacts instead of just documents: working API scaffolding, configuration management tooling, reusable implementation foundations. The validation process itself became the starting point for building.

The Bigger Opportunity: Product and Growth Experimentation

The engineering examples are compelling, but they’re not the most interesting part of this shift.

The real unlock is what happens when product and growth teams start thinking in terms of experimentation velocity. When the question changes from “can we get this on the roadmap?” to “can we just try it?”

One unexpected outcome from our sprints was a cross-team POC that shipped a genuinely new product interaction model — a way for users to interact with AI directly within the screens they’re already using, without switching to a separate chat interface. Context-aware, embedded in the existing product, surfacing alongside whatever the user is already doing. The team built it in a single sprint using the existing backend architecture. It wasn’t on any roadmap. A few people had the idea, the cost of trying it was low enough to just go, and within days they had a working demo on two platforms. Within a week, that demo had reshaped how the product team was thinking about AI surface area and they’re now using that experiment as the foundation for new product plans. What started as an unplanned two-day bet is actively shaping the roadmap.

That sequence (have an idea, validate it quickly, commit if it works) is what high experimentation velocity actually looks like. It’s the sequence that matters most for product and growth teams, because it changes the economics of exploration.

Think about what product teams spend most of their time on: building cases for what to try next. Writing specs. Prioritizing backlogs. Negotiating for engineering capacity. Most of that overhead exists because trying things is expensive, so you need to be very sure before you commit resources. When trying things becomes cheap, the entire prioritization calculus changes. You don’t need a business case to place a small bet. You need a hypothesis and a day.

This is the difference between a product team that ships its roadmap 20% faster and one that tests three ideas for every one it used to test. The first team gets efficiency. The second team gets compounding discovery and some of those discoveries become the roadmap.

The New Bottleneck

Experimentation velocity is only valuable if your organization can act on what experiments reveal.

Once engineering capacity stops being the constraint, a new set of bottlenecks becomes visible and they don’t live in engineering.

Requirements that aren’t ready to act on. Design work that can’t keep pace with implementation. Product decisions that take longer to make than the feature takes to build. These are now the rate limiters. The constraint has moved from “can we build this?” to “do we know what to build, and can we decide fast enough to keep up with what we’re capable of executing?”

The teams pulling the most value from agentic workflows are the ones that treated this as a whole-system change, not just an engineering upgrade. They restructured how requirements get written: specific, testable, agent-ready. They compressed design cycles to match implementation speed. They pushed decision-making closer to the work, with explicit permission to move at 70% confidence and course-correct on real output rather than waiting for perfect plans.

What To Ask Your Teams

If you’re evaluating your AI investment primarily through a delivery lens (features shipped, velocity improved, cycle time reduced) you’re getting value, but you’re probably underestimating what’s available.

The better question is a pair: how much faster are we executing what we planned, and what did we try this quarter that we couldn’t have tried six months ago?

If the answer to the second question is a long list of experiments, prototypes, and bets that came out of nowhere, you’re seeing the real return. If the answer is just “we shipped the roadmap faster,” you’re leaving the more interesting half on the table.

Delivery speed is a good start. Experimentation velocity is the level up. The constraint has moved. The teams winning with AI aren’t just faster, they’re attempting more. That’s the metric worth tracking.

Coding Agents Don't Replace Platforms — They Make Them More Important

Fri, 03 Apr 2026 00:00:00 +0000

There’s a growing belief in engineering circles that because coding agents can generate code so cheaply and quickly, we no longer need to invest in platform frameworks, services, and shared systems. The argument goes something like: code is essentially free now, so why bother with reusable building blocks? Just let agents generate what you need on the fly.

I think this perspective fundamentally misunderstands both how agents work and what makes software development expensive.

Agents Already Use Frameworks

Here’s the irony of the “we don’t need platforms anymore” argument: coding agents themselves rely heavily on frameworks, libraries, and established patterns to produce good output.

When you ask Claude Code or Cursor to build you a web application, it doesn’t reinvent HTTP from scratch. It reaches for React, FastAPI, Express, or whatever framework fits the context. When it writes infrastructure code, it uses Terraform modules or Pulumi components. The agent’s effectiveness comes precisely from having well-understood building blocks to compose, not from generating everything from raw primitives.

Addy Osmani captured this well in his piece on writing specs for AI agents, noting that vague prompts produce vague results while specific constraints and well-defined building blocks dramatically improve output quality.

This maps directly to platform engineering. A well-designed internal developer platform is, at its core, a curated set of constraints and building blocks. Golden paths, opinionated templates, standardized observability, security guardrails — these aren’t overhead. They’re the context that makes both human and AI-driven development faster and more reliable.

Constraints Improve Agent Output

There’s a counterintuitive principle at work here: giving agents fewer choices often produces better results.

When an agent operates with a blank canvas and unlimited options, it pattern-matches against its training data, which includes code of wildly varying quality. The result is what people call “AI slop”: technically functional output that’s tangled, inconsistent, and painful to maintain. But when you provide architectural constraints, approved component libraries, established patterns, and clear specs, the agent’s output improves dramatically.

This is why the emerging best practice for agentic development isn’t “let the agent do whatever it wants”. Instead, teams are investing in AGENTS.md files, architectural constraint validators, spec-driven development workflows, and code health scoring systems that give agents objective signals about quality. As CodeScene’s analysis of agentic coding patterns concludes: coding via agents requires more rigor, more structure, more code quality — not less.

Platform engineering is the organizational expression of exactly this idea. When you give development teams (and their agents) a set of well-maintained building blocks (vetted libraries, standardized service templates, pre-configured CI/CD pipelines, embedded observability) you’re constraining the solution space in productive ways. If you need more building blocks, you can use agents to build those too. The platform grows, but it grows deliberately.

Disposable Code vs Durable Code

Charity Majors wrote a piece that frames this tension well: Disposable Code Is Here to Stay, but Durable Code Is What Runs the World. Her core argument is that software development is separating into two domains with very different cost models and risk profiles.

Disposable code (design mocks, experiments, data processing scripts, prototypes) has exploded since agents made generation cheap. You spin something up, use it, throw it away. Agents are fantastic at this.

But then there’s the code that runs financial transactions, manages medical records, routes airline flights, and powers commerce sites. This is durable code, and its cost isn’t defined by how hard it was to write. The cost is defined by the standards you build to and the entire lifecycle that follows: maintenance, extension, migration, understanding its behavior, and fixing it when it breaks. Disposable code is cheap because you don’t even try to maintain it. Durable code is expensive because you must.

This distinction matters enormously for the platform engineering conversation. Platform systems (the shared services, frameworks, SDKs, infrastructure components) are definitionally durable code. They’re the foundation that many teams and many applications depend on. They need to be reliable, performant, well-tested, observable, and maintained over time. As Majors points out, the cheapest way to build good durable software is by investing in the sociotechnical systems that unlock fast feedback loops: CI/CD, observability, testing. The only companies currently reaping efficiency gains from AI are the ones who already invested in these foundations.

Building is the shortest phase of a software system’s life. Maintenance is where the real cost lives. A platform of durable building blocks doesn’t just speed up initial development — it dramatically reduces the ongoing cost of maintaining everything built on top of it.

Specs, Code, and the Feedback Loop

I see enormous value in using agents for durable purposes, not just throwaway scripts. Drew Breunig’s experiment with a software library with no code is an interesting thought experiment. He released whenwords, a library containing only a spec and tests, no implementation code. You paste the spec into a coding agent, tell it your target language, and it generates a working implementation. It works across Ruby, Python, Rust, Swift, and more.

Maybe we don’t need to share code at all, just specs and tests. Vercel built just-bash (a simulated bash environment in TypeScript) and Pydantic built Monty (a Python interpreter in Rust) using similar approaches — throw a test suite at an agent, make it pass.

But in a follow-up talk, Breunig walked back the premise. His biggest takeaway: “No-code libraries are toys because they are unproven.” The act of implementing code clarifies and communicates intent in ways that specs alone cannot. A spec doesn’t truly work until it’s implemented, just like software doesn’t truly work until it meets the real world.

He reframed spec-driven development from a one-way equation (specs + tests + agent = code) into what he calls the Spec-Driven Development Triangle: a feedback loop where implementing code improves the spec, updated specs drive new tests, and the three nodes must be kept in sync as they evolve. Even with well-defined specs, real-world projects generated 20-comment threads debating the right way to implement something, because no spec is ever complete.

Durable platform building blocks aren’t just code, they’re code that has been through that triangle many times. The spec has been refined by implementation. The tests have been shaped by production behavior. The code has earned trust through time. That feedback loop is precisely what makes platform components more valuable than freshly generated code, no matter how good the agent is.

There’s a middle ground between “generate everything from scratch” and “only use proven platform code” that I find compelling, and it’s directly relevant to platform engineering.

Using Agents to Extend Platform Reach

Consider a server-driven UI system. The real value of such a platform is on the server side: the configuration layout, the integration with data systems for hydration, the logic and filtering for managing delivered content, the APIs for client interaction. This is durable, complex, evolving code that benefits from deep expertise and long-term investment.

The frontend SDK, though, is a different story. It’s written in a specific tech stack (Swift for iOS, Kotlin for Android, React for web) but the behavior it implements is largely mechanical: take the server’s UI models, render components, handle actions, manage state. If you could extract the specs (the API contracts, the UI model definitions, the action behaviors) you could ask an agent to generate a frontend SDK in a new tech stack. React Native, Flutter, whatever your team needs next.

This isn’t “disposable code generated on the fly.” It’s using agents to build durable code from durable specs, extending the platform’s reach to new tech stacks without requiring a team of specialists in each one. The server-side platform remains the source of truth. The generated SDKs are tested against the same contract. The specs are the stable foundation; the implementations are the part that agents can help scale.

You could apply the same pattern to client API wrappers. If you have a well-specified API (clear endpoints, typed request/response models, error handling contracts, authentication flows) an agent can generate idiomatic client libraries for Python, Go, TypeScript, Java, and beyond. Not as throwaway code, but as maintained, tested libraries that follow the spec. When the API changes, you update the spec and regenerate.

This is something like Breunig’s original “library with no code” concept, but bounded and connected to a larger platform system. The spec isn’t floating freely, it’s anchored to a real, maintained, durable platform. These generated implementations feed back into the triangle: building the Flutter SDK might reveal ambiguities in the spec that the Swift SDK never surfaced, improving the platform for everyone. The generated code isn’t pure commodity. It extends the platform’s value into new contexts while strengthening the spec it’s built from.

Google’s recent A2UI project hints at a similar future: a declarative UI protocol where agents send component descriptions that different client renderers (web, mobile, desktop) can display natively. The spec is the stable contract; the renderers can be built per-platform, potentially with agent assistance.

The Real Relationship: Agents and Platforms Together

The emerging picture isn’t agents vs. platforms. It’s agents and platforms reinforcing each other.

Platforms give agents the constraints, context, and building blocks they need to produce high-quality output. Golden paths become agent-friendly paths. Architectural validators catch agent mistakes before they hit production. Standardized templates mean agents start from a proven foundation rather than a blank page.

Agents, in turn, help platforms scale their impact. They can generate implementations from specs across multiple tech stacks. They can help maintain the platform itself, updating templates, running migrations, fixing linting issues across codebases. They can handle the mechanical work of extending platform reach to new contexts.

The State of Platform Engineering Report calls this the “dual mandate” for 2026: platforms must be AI-powered (using agents to enhance developer productivity) and must simultaneously be platforms for AI (providing the guardrails and governance that agents need to operate safely). Ninety-four percent of organizations now view AI as critical to the future of platform engineering.

As the Anthropic Agentic Coding Trends Report puts it, engineers are shifting from writing code to coordinating agents — focusing expertise on architecture, system design, and strategic decisions. That’s exactly what platform engineering has always been about: making the right thing the easy thing, so teams can focus on what matters.

The Takeaway

Code generation getting cheaper doesn’t mean platforms get less important. It means they get more important.

Building blocks improve agent output. Agents with constrained, well-defined component libraries produce better code than agents with a blank canvas. Platform engineering provides those blocks.

Maintenance outlasts building. The cheapest part of a software system’s life is writing the first version. Durable building blocks reduce the cumulative maintenance burden across every application built on top of them.

Agents can extend platform reach. Well-specified platform systems (APIs, UI models, action contracts) can be implemented across new tech stacks with agent assistance, turning platform specs into multi-platform value.

Platforms need to serve agents too. As agents become active participants in the development lifecycle, platforms must provide the guardrails, context, and observability that agents need to operate reliably.

The organizations that will move fastest aren’t the ones letting agents generate everything from scratch. They’re the ones giving agents great building blocks to work with and using agents to make those building blocks available everywhere.

Agentic Coding: Who Will Review All That Code?

Sun, 01 Mar 2026 00:00:00 +0000

League (my day job) recently enabled mandatory GitHub Copilot code reviews on all pull requests. Not that long ago, I would not have supported this idea. Models, and the agent systems that use the models, have improved to the point of frequently finding non-trivial issues with code in a pull request.

We’d had Copilot and Codex reviews enabled for a while, but they were optional. Easy to ignore. For those who actually read the reviews, we watched the quality get better and better. Good enough to make mandatory and see how it works.

As we use coding agents to create more code, it follows that we’re going to need to find ways to review all that code. Before I get side-tracked into talking about agentic code reviews, we should probably figure out what parts of code reviews matter the most.

Code Reviews Don’t Catch Many Bugs

Here’s the thing most engineering leaders don’t want to accept: human code reviews are not very good at finding defects. We all think they are. We’d like them to be. The research has been telling us otherwise for over a decade.

[this is where I asked Claude to summarize some research for me]

The 2013 Microsoft Research study by Bacchelli and Bird — “Expectations, Outcomes, and Challenges of Modern Code Review” — analyzed hundreds of review comments, surveyed 873 programmers and 165 managers, and observed developers in the act of reviewing. They found that only 14% of review comments addressed defects. The most common category? Code improvements: readability, consistency, dead code removal. A follow-up study by Czerwonka, Greiler, and Tilford at Microsoft (2015), bluntly titled “Code Reviews Do Not Find Bugs,” corroborated those findings across 25,000 developers: roughly 15% of reviewer comments indicated a possible defect, while about half concerned long-term maintainability.

And yet, 44% of developers in Bacchelli and Bird’s survey ranked “finding defects” as their number one motivation for doing code reviews. There is a persistent gap between what we believe code reviews do and what they actually do.

Mäntylä and Lassenius found that 75% of defects discovered during code review were “evolvability” defects — issues affecting maintainability and comprehensibility — not functional bugs.

Most code reviews find no bugs at all. This doesn’t mean the bugs don’t exist, only that reviewers didn’t find them.

Someone should point out that finding 15% of the defects is better than nothing, and I’d agree. It would be tragic if something happened to cause that detection rate to drop.

What Are Code Reviews Good At?

Google’s engineering practices documentation (SWE Book) says: “Checking for code correctness is NOT the primary benefit Google accrues from the process of code review.” Instead, Google identifies the core benefits as code comprehension, consistency enforcement, knowledge sharing, psychological ownership, and validation, with correctness listed last.

A study of code review networks at Microsoft, Spotify, and Trivago found that an average participant can spread information to 72–85% of all code review participants within four weeks. Information travels between any two participants in fewer than five hops and fewer than five days. Code review isn’t just a quality gate. It’s the primary nervous system through which engineering organizations maintain shared understanding of their own systems.

The real value of code reviews:

Knowledge transfer: Someone other than the author learns how the system works. This is how organizations reduce bus factor, onboard new engineers, and maintain the ability to debug systems at 2am.
Architectural oversight: Code owners and senior engineers catch changes that work locally but don’t fit the broader system design — wrong abstractions, inconsistent patterns, unintended coupling.
Norm maintenance: Teams calibrate on naming conventions, error handling strategies, testing approaches, and what “good” looks like. This is social, not mechanical. Once a decision has been made, it can become mechanical.
Shared awareness: Reviews keep the team aware of what’s changing and why. This context is invisible until it’s missing.

None of these are things a linter catches. None of these are things you get from a passing test suite. Most critically, these are exactly the things that get lost when AI generates code that no human fully understands. Read what others are writing about the “cognitive debt” that happens with coding agents.

Agentic Coding Makes the Problem Worse

The Cortex 2026 Benchmark Report found PRs per author up 20% year-over-year, but incidents per pull request increased 23.5% and change failure rates rose roughly 30%. The DORA Report documented a 7.2% reduction in delivery stability with heavy AI reliance — even though 75% of developers felt more productive. AI-assisted PRs are 18% larger on average. Median PR size increased 33% in 2025.

So we’re producing more code, faster, in larger chunks, with more subtle defects — and asking humans to review it all. This won’t scale. Faros AI data across 10,000+ developers shows code review time has increased by roughly 91% as PR volume and size increased. An IT Pro/Qodo survey found that nearly half of developers don’t check AI-generated code at all, with 38% saying it takes longer to review than code from colleagues.

Using Agents to Review Agent-Written Code

This is why League enabled mandatory Copilot reviews. We’re using agents to cover some of the review capacity that human teams can no longer provide at the volume AI-generated code demands. And honestly, agents are pretty good at the mechanical layer of review. They catch the things that can be tedious for humans — improper framework usage, inconsistent error handling, basic security patterns. The models have gotten good enough at this to be worth mandating.

The emerging model is layered review:

Real-time IDE feedback — the agent catches issues as you write (or as another agent writes).
Automated PR review — Copilot, Codex, Claude, or similar tools handle the mechanical layer at submission time.
Human review for the hard parts — architecture, security threat modeling, business logic, and the “does this fit our system” judgment calls.

The key insight is that automated review should free up human attention for the things humans are uniquely good at, not replace human review entirely.

Focus More on Code Planning

Here’s a spicy take: we could probably do fewer code reviews if we got better at code planning.

With coding agents, we need to be very clear and explicit with upfront requirements. We’ve even started calling this “spec-driven development.” That extra time spent at the beginning of the process, along with guidance given to the agents through rules files, architecture documents, and the tests used to verify results. It could mean the need for code reviews decreases.

This isn’t a new idea, but agentic coding makes it more urgent. When a human writes code, they carry implicit context: they know the team’s conventions, the system’s quirks, the reasons behind past decisions. When an agent writes code, all of that context has to be explicit. It has to be written down. And the act of writing it down — of specifying what you want, how it should integrate, what patterns to follow, what to avoid — is itself a form of design review that happens before a single line of code is generated.

Addy Osmani’s “PR Contract” framework captures this well. Every PR should declare: what and why in a sentence or two, proof it works via tests or demo, a risk tier and which parts were AI-generated, and one or two specific areas requiring human review focus. The core principle: if you can’t explain why the code works, you shouldn’t be asking someone else to approve it.

The organizations adapting best are investing heavily in what happens upstream of the pull request:

Architecture decision records and design docs that agents and reviewers can reference
Project rules files (.cursor/rules, AGENTS.md, copilot-instructions.md) that encode team conventions so agents generate code that fits the system from the start
Spec-driven development where the specification is reviewed and approved before any code is generated
Test-first workflows where agents must execute and pass tests before submitting for review

When you invest in the inputs, you reduce the burden on the review. The review becomes a confirmation that the plan was followed, not a discovery process for whether the plan was any good.

Where This Is Heading

I’m not saying we should stop reviewing code today, but I do think we’ll stop reviewing as much code in a year.

The answer, I think, involves three shifts: let agents handle the mechanical review they’re now good enough at, invest heavily in what happens before code is written (specs, architecture, rules), and be intentional about preserving the knowledge-sharing function of review even if that means reviewing less code, more carefully, with a focus on system-level understanding rather than line-by-line correctness.

The risk is knowledge erosion and cognitive debt. If humans stop reading code because agents write it and other agents review it, the team loses the shared understanding that code review has always quietly maintained. The Opsera 2026 benchmark found that senior engineers realize nearly five times the productivity gains of junior engineers from AI tools — precisely because they already possess the system-level understanding needed to evaluate AI output. AI amplifies existing expertise. It doesn’t create it.

So the question isn’t really “who will review all that code?” The question is: how do we maintain the knowledge transfer, architectural oversight, and shared understanding that code reviews have always provided — when the volume of code outstrips our capacity to read it all?

The Rise of Hyper Local Software

Tue, 24 Feb 2026 00:00:00 +0000

For years, we’ve been told that software is eating the world. Big, cloud-hosted platforms — Salesforce, Notion, Slack, Jira, Airtable — working to solve every problem for every team, at any scale. And mostly, they delivered. SaaS transformed how businesses work.

Making software and systems that support millions of people, with a variety of needs and requirements isn’t easy. Software adds complexity to deal with the hundreds use cases. It becomes a series of compromises. Software that starts small and well focused, evolves into a villain, despised by users.

Now something is shifting and not enough people are talking about it.

The “SaaS Is Dead” Take Is Half Right

There’s no shortage of hot takes about AI agents killing SaaS. The argument goes: why pay for Salesforce when an AI agent can just build your CRM for you? Why subscribe to a project management tool when an agent can manage your projects? Satya Nadella has suggested that business applications are essentially just “CRUD databases with business logic” that will eventually collapse into the AI layer.

That’s a plausible long-term story for enterprise software, but it’s not the most interesting thing happening right now.

The more interesting story is quieter and more personal. Agentic coding tools are making it trivially fast for individuals and small teams to build software tailored to their situation. Not enterprise replacements. Not startups. Just… tools that fit their needs.

I don’t know if this has a name yet, but I’ve started calling it hyper-local software: apps built for an audience that might be one person, one team, or one community. Apps that would never survive as a SaaS business because the market is too small or too specific, but that are useful to the people they’re built for.

Why Now

The economics of building software have changed. The friction of creation has dropped to the point where it now competes with the friction of discovery. It used to be that if you had a niche software need, you’d spend hours searching for a tool that mostly fit, pay for it, and accept its limitations. Now, in many cases, it’s faster to just build what you actually want.

Hyper-local software doesn’t have the same scale and quality requirements of enterprise SaaS software. When it’s software built for one, or a team, you’re willing to live with papercuts — or fix them as needed. You don’t need to handle scale for millions of users.

The result is a new kind of software culture emerging. Not SaaS. Not open source in the traditional sense. Something more like the early days of the web, when people built things because they wanted them and shared them because sharing was easy.

The Opportunity

Is anyone thinking about how to support this ecosystem?

The conversation around AI and software is dominated by the enterprise angle, which SaaS companies will survive, which will be disrupted, how incumbents should respond. That’s a legitimate conversation, but it ignores an entirely different opportunity.

What would it look like to deliberately cultivate a hyper-local software culture?

The ingredients seem clear enough:

Low-friction creation. The tools are largely here — agentic coding has dramatically lowered the barrier. But they’re still oriented toward professional developers. There’s room for tooling that’s oriented toward domain experts who want to build something that works for their specific context.

Easy sharing and remixing. The most valuable thing about hyper-local apps isn’t just that one person can build one — it’s that patterns can spread. If a teacher builds a great gradebook app for her specific situation, there’s likely another teacher in a slightly different situation who could build on that work. A culture of remixing is essential.

Community around use cases. The social layer matters enormously. When people share what they’ve built, not as products but as solutions to specific problems, it creates a library of starting points. It normalizes the idea of building your own tools. It creates the kind of culture where “I just built something for that” is a normal thing to say.

Graceful hosting. The infrastructure needs to be low-maintenance enough that a non-engineer can deploy and keep running a small personal app without it becoming a second job.

This isn’t a product pitch. It’s a gap observation. The culture of hyper-local software is forming on its own in blog posts, in indie developer communities, in the growing practice of vibe coding. But it’s diffuse and unsupported.

Has anyone thought about giving this ecosystem a home? Maybe I’m not paying close enough attention.

Glitch Was Almost That Home

It’s hard for me to think about this gap without thinking about Glitch.

Glitch launched in 2017 with a simple and radical idea: what if building and sharing a web app was as easy as publishing a blog post? You could create a project, remix someone else’s, deploy instantly, and share with a link. The community that formed around it was warm, creative, and genuinely collaborative. By 2019, the platform hosted over 2.5 million apps. Sadly, in 2025, Glitch shut down project hosting. The architecture had grown expensive to maintain. Abuse had driven up operational costs. The developer landscape had evolved around it.

Looking back, Glitch had most of the right ingredients. Easy creation. One-click remixing. Starter projects that let you go from zero to something real in minutes. A community that celebrated building things, not just shipping products. An ethos¹ of sharing that felt more like a craft community than a startup ecosystem.

Glitch was, in spirit, exactly the kind of place where hyper-local software culture could have flourished. Maybe my own bias is showing.

The best software has always been the kind that fits. The era of hyper-local apps is making that possible for almost anyone. The question is whether we’ll build the ecosystem to support it.

1: Claude helped me with finding the right word

Migrating a 20-Year WordPress Blog to Hugo and Cloudflare

Sun, 15 Feb 2026 00:00:00 +0000

I moved my 20+ year old WordPress blog off Bluehost and onto a modern static site setup with Hugo and Cloudflare Workers. What I expected to be a painful, multi-week project turned into a surprisingly smooth 4-day migration, largely thanks to working with Claude as a coding agent throughout the process.

Why Move Away from WordPress?

WordPress has served me well for two decades, but it felt like overkill for what is essentially a personal blog. I’m not running e-commerce, I don’t need a database for every page load, and I definitely don’t need the constant plugin updates and security concerns. More importantly, Bluehost, while reliable, isn’t exactly a modern hosting vendor. I wanted to explore what a more current tech stack could offer.

This migration gave me an excuse to dive into Cloudflare’s ecosystem and see what they’re really capable of beyond just being a CDN. The cost savings (from ~$150/year to essentially free) was a nice outcome, but the real motivation was modernization and learning.

The Export Process: WordPress to Markdown

The first step was getting content out of WordPress. I used the Simply Static plugin to export everything as static HTML files. This worked well and gave me a complete snapshot of the blog including all images, styles, and 20 years of posts. I decided to keep that as the main archive.

But Hugo uses Markdown, not HTML. I used a Wordpress Export to extract all content into a download XML file. Then ran a WordPress Export to Markdown tool on the export file.

npx wordpress-export-to-markdown

That created individual index.md files for each post with proper front matter. Hugo is flexible about content structure. The WordPress Export to Markdown tool gave me options, and I decided to organize posts using a year/month hierarchy with page bundles:

content/
└── posts/
    └── 2025/
        └── 03/
            └── exploring-llms-as-agents/
                ├── index.md
                └── images/
                    └── screenshot.png

This page bundle approach meant each post lived in its own folder with all its assets. Images could use relative paths like images/screenshot.png in the Markdown, making posts completely self-contained.

I chose the rusty-typewriter theme as a starting point. Hugo’s theme system made it easy to customize without touching the theme files directly — just override partials in my own layouts/ directory when needed.

A few things to note: I added series as a custom taxonomy (great for multi-part posts), and configured a search index output so the theme’s built-in client-side search would work. The search feature was surprisingly fast once I optimized the index to only include post summaries instead of full content.

Fixing Legacy Content

Twenty years of content meant dealing with some interesting legacy issues:

Cross-Post Links

My WordPress URLs followed the pattern /blog/YYYY/MM/post-slug/, but I wanted cleaner Hugo URLs at /posts/YYYY/MM/post-slug/. Internal links needed updating:

# Find and replace across all Markdown files
find content/posts -name "*.md" -type f -exec sed -i '' 's|https://starkravingfinkle.org/blog|/posts|g' {} +

Ancient Blogspot Links

I found posts that still linked to my really old Blogspot blog from 2004. Those needed fixing too:

# Handle multiple URLs on same line (trickier regex)
find content/posts -name "*.md" -type f -exec sed -i '' 's|http://weborama\.blogspot\.com/\([^)]*\)\.html|/posts/\1/|g' {} +

The key was using [^)] in the regex pattern to stop at the closing parenthesis of Markdown links, preventing the pattern from being too greedy when multiple links appeared on one line.

WordPress Shortcodes

WordPress uses [caption] shortcodes that Hugo doesn’t understand. I converted them to Hugo’s native figure shortcode.

I did some of these conversions manually for the handful of captioned images, but Claude created a script would work for larger volumes.

Deploying to Cloudflare Workers

Getting Hugo working locally was one thing; deploying to Cloudflare Workers was where I got to learn new things. Workers can serve static sites directly, which is perfect for a Hugo blog.

Basic Worker Setup

mkdir starkravingfinkle-blog
cd starkravingfinkle-blog
npm init -y
npm install wrangler --save-dev

The wrangler.toml configuration started simple:

name = "starkravingfinkle-blog"
compatibility_date = "2025-02-04"

[assets]
directory = "./public"
not_found_handling = "404-page"
binding = "ASSETS"

Hugo builds into the public/ directory, and Workers serves from there. The binding = "ASSETS" creates an environment variable you can reference in Worker code.

Handling URL Redirects

Here’s where it got more complex. My old WordPress URLs used /blog/ in the path, but I wanted the Hugo site at the root. Breaking existing links was not an option. There has to be a few people who have linked to my posts over 20 years.

I needed a Worker script to handle redirects:

// src/index.js
export default {
  async fetch(request, env) {
    const url = new URL(request.url);
    
    // Redirect /blog/ paths to /posts/
    if (url.pathname.startsWith('/blog/')) {
      // Handle RSS feed
      if (url.pathname === '/blog/feed/' || url.pathname === '/blog/feed') {
        return Response.redirect(url.origin + '/feed.xml', 301);
      }
      
      // Handle blog posts
      const newPath = url.pathname.replace('/blog/', '/posts/');
      return Response.redirect(url.origin + newPath, 301);
    }
    
    // Serve static assets normally
    return env.ASSETS.fetch(request);
  }
};

Updated wrangler.toml to use the script:

name = "starkravingfinkle-blog"
main = "src/index.js"
compatibility_date = "2025-02-04"

[assets]
directory = "./public"
not_found_handling = "404-page"
binding = "ASSETS"

Now old WordPress URLs redirect to the new Hugo structure with proper 301 permanent redirects.

Deployment

The deployment workflow is simple:

# Build the Hugo site
hugo

# Deploy to Cloudflare
npx wrangler deploy

That’s it. The site is live on Cloudflare’s edge network in seconds.

DNS and Domain Migration

Moving from Bluehost to Cloudflare meant changing nameservers and setting up custom domains. The process was straightforward:

Add domain to Cloudflare (it scans existing DNS)
Review imported DNS records
Update nameservers at domain registrar
Add custom domain to Worker (automatic DNS setup)
Update baseURL in hugo.toml
Rebuild and redeploy

DNS propagation took about 30 minutes in my case. I tested first with the *.workers.dev subdomain before pointing the real domain, which gave me confidence everything worked.

One gotcha: Browser DNS caching. Even after DNS propagated globally, my main browser held onto the old Bluehost IP for hours. Testing in Safari (not my main browser) showed the new site was live while Firefox stubbornly showed the old one. Clearing Firefox’s DNS cache (about:networking#dns) fixed it.

Search Functionality

The rusty-typewriter theme includes client-side search using a JSON index. Initially, the search was slow because the index included full post content. Over 20 years of writing made for a huge JSON file.

The fix was simple: generate summaries instead of full content in the search index template:

{{- $.Scratch.Add "searchindex" slice -}}
{{- range $index, $element := where site.RegularPages "Params.indexable" "ne" false -}}
    {{- $.Scratch.Add "searchindex" (dict 
        "id" $index 
        "title" $element.Title 
        "permalink" $element.RelPermalink 
        "tags" (delimit ($element.Params.tags | default slice) " ") 
        "summary" ($element.Summary | plainify)
        "date" ($element.Date.Format ($.Param "dateformat" | default "2 January, 2006"))
    ) -}}
{{- end -}}
{{- $.Scratch.Get "searchindex" | jsonify -}}

Removing the full .Plain content and using summaries made the search index 80-90% smaller and noticeably faster, but still gave very good search results

Working with Claude as a Coding Agent

I need to call out how much easier this migration was because I worked with Claude throughout. I’m not talking about asking occasional questions, I mean having Claude actively participate in the entire migration as a coding agent.

What Claude helped with:

Explaining Hugo’s content structure and conventions
Writing and debugging sed commands for bulk URL fixes
Creating the Worker redirect logic
Troubleshooting DNS and configuration issues
Optimizing the search index
Walking through Cloudflare setup step-by-step

The biggest value wasn’t just getting answers, it was the iterative problem-solving. When something didn’t work, we’d debug together. When I hit a wall, Claude suggested alternatives. When I learned something new about Hugo or Cloudflare, Claude helped connect it to the bigger picture.

This is what AI coding agents excel at: removing the friction of getting started and maintaining momentum when you’re learning multiple new technologies at once. Without Claude, I probably would have spent a week just reading Hugo documentation and Cloudflare guides before writing any code. It might have been enough perceived friction that I just wouldn’t even have started.

Learnings

Static Sites Scale Down Complexity

Moving from WordPress to static HTML eliminated an entire class of concerns: database performance, PHP updates, plugin conflicts, security patches. The site is literally just files being served from Cloudflare’s edge network. Simple is powerful.

Redirects Were Important

Twenty years of links exist out there in the wild. Search engine indexes, social media posts, bookmarks, other blogs — all pointing to old URLs. The redirect strategy needed to be bulletproof. Cloudflare Workers made this trivial with a few lines of JavaScript.

AI Agents Change the Learning Curve

Learning Hugo, Cloudflare Workers, DNS setup, and email configuration simultaneously would normally take weeks. With Claude as an active partner, it took 4 days. The agent didn’t just answer questions, it helped me make decisions, debug problems, and connect concepts across multiple technology stacks.

Legacy Content Has Character

Finding 20-year-old Blogspot links in my posts was a fun reminder of how long I’ve been writing online. Rather than being annoyed by the cleanup work, I appreciated the archeology of it. Every odd WordPress shortcode or malformed URL told a story about what blogging tools were like in 2005.

What’s Next?

Now that the blog is running on Hugo and Cloudflare, I want to explore a few additions:

New content types: Hugo makes it easy to add distinct content types beyond blog posts. I’m thinking about adding quotes and TIL (Today I Learned) sections — short-form content that doesn’t fit the typical blog post format.

Writing workflow: Since the blog content is now local Markdown files (not trapped in a WordPress database), I can work more easily on drafts. The friction of “opening WordPress to write” is gone. It’s just files in a directory.

Analytics refinement: Cloudflare Web Analytics is running, but I want to understand what reports and insights are actually useful versus vanity metrics. The data is there; I need to figure out what questions to ask of it.

Final Thoughts

Migrating a 20-year blog sounds daunting, but it turned out to be one of those projects that’s easier than you think, especially with the right tools. Hugo is powerful but approachable, Cloudflare Workers are remarkably simple to deploy to, and working with Claude as a coding agent removed most of the friction from learning new technologies.

If you’re on the fence about moving away from WordPress, I’d encourage you to try it. Static sites aren’t just faster and cheaper, they’re genuinely simpler to reason about. And with modern tooling (Hugo’s live reload, Wrangler’s instant deploys, AI agents for guidance), the developer experience is excellent.

The web has come a long way since I started this blog in the early 2000s. It’s nice to have my infrastructure finally catch up.

Agent Coding Update: Teaching Agents Our Primitives

Mon, 08 Dec 2025 00:00:00 +0000

A few months ago, I wrote about vibe coding and the challenge of shipping quickly-created prototypes. I ended that post suggesting we could “teach” coding agents about our platform primitives through instruction files and MCP servers. At my day job (League), we’ve been putting this idea into practice, and the results have been very encouraging.

Building a Knowledge Base for Agents

A cross-functional team has created a structured knowledge base repository—a collection of markdown documents covering both frontend and backend aspects of the League platform. Think of it as documentation specifically designed for agent consumption. These aren’t just API references or code comments; they’re curated explanations of our patterns, primitives, and platform conventions.

The knowledge base includes information about:

UI frameworks and component patterns we use
Backend integration patterns and workflows
API structures and authentication mechanisms
Routing and state management patterns
Testing strategies and common utilities

Making Knowledge Discoverable and Actionable

Documentation is only useful if it can be found. The team built an MCP server that gives agents access to some useful capabilities:

Search – Agents can query the knowledge base by topic, finding relevant documents from our extensive collection without needing to know exactly what exists.
Read - Agents can fetch the contents of the relevant documents for use in the current working session.
Write – Most importantly, agents can contribute back to the knowledge base. When an agent, working with a developer, discovers a successful pattern or solves a problem in a novel way, it can propose new documentation via pull request.

This bidirectional flow transforms the knowledge base from static documentation into a living, growing resource. The agents aren’t just consuming our platform knowledge, they’re helping to build it.

Local Workflows Enable Faster Iteration

One of our key learnings has been that making more of the developer workflow “local” dramatically improves agent effectiveness. When an agent can not only write code but also test the changes immediately, the feedback loop tightens significantly. The agent, and developer, learns what works faster and produces better outcomes with less back-and-forth.

This local-first approach also makes agents better at completing tasks. As they experiment and validate patterns, they can immediately capture that knowledge and push it back into the knowledge base. Good patterns get documented while they’re fresh. Failed approaches can be noted before anyone else tries them.

From Prototype to Production, Faster

The combination of searchable platform knowledge and local testing capabilities has transformed how quickly we can go from idea to working feature. Developers, working with agents, now vibe code using our primitives from the start. There’s less “throw away the prototype and rebuild it properly” and more “this is already using our patterns, let’s polish and ship.”

The code that emerges from the agent looks like it could have been written by someone familiar with League’s platform conventions because in a very real sense, it was. The agent had access to our collective knowledge about what good League platform code looks like.

If you think it’s fun to vibe code green-field prototypes built on default frameworks not connected to your own system, I can tell you it’s even more fun to quickly prototype and craft solutions using your own primitives, ready to be added into your products.

What We’re Learning

This approach isn’t without challenges. We’re still discovering:

What level of detail makes documentation most useful for agents versus overwhelming the context window. Developer curation plays a critical role.
How to structure the knowledge base so relevant information surfaces easily. Again, developer curation along with solid search capabilities should help.
When to trust agent-written documentation versus requiring developer review. At this point, we’re always requiring developer review, and that might be the case permanently.
How to balance comprehensiveness with maintainability as the knowledge base grows.

But the core insight holds: when you give agents knowledge of your platform primitives and enable them to both test and contribute, they become far more effective collaborators. The coding isn’t just faster, it’s actually aligned with how you build software.

Where We’re Headed

We’re still early in this exploration, but the path is promising. As the knowledge base grows and agents contribute more successful patterns, we’re creating a virtuous cycle. Each successful feature makes the next one easier. Each documented pattern helps future agents avoid past mistakes. The main goal to help developers get more done without as much cognitive load.

The future of agentic development won’t be about replacing developers. It will be about building systems where developer knowledge and machine capability amplify each other. Where the tribal knowledge that usually lives in senior developers’ heads gets captured, structured, and made accessible at the moment it’s needed.

Vibe Coded to Shipped Feature

Sun, 06 Jul 2025 00:00:00 +0000

AI-assisted coding tools and agents have unlocked an unprecedented level of functional prototyping. Not just for developers. Product managers, Designers, even the C-suite are building functional prototypes of their ideas in a matter of days, or even hours.

There can be a lot of value in creating a functional prototype of an idea so people can really test out the interactions and experience in ways that are not feasible with less interactive approaches.

People like to push boundaries, so I imagine a future, maybe it’s already here, where people begin to ask: How do I ship my quickly created prototype in our product?

Lots of people will tell you that you can’t just ship a vibe coded feature — well, maybe those “people” are developers. There are some good reasons why you should think twice about shipping a vibe coded concept:

Do you understand the code the agent wrote?
What dependencies does the application use?
How does it store data and is it secure?
How does the login system work, if it has one?
… the list goes on

Honestly, the same questions should be asked no matter how the code was created. If the vibe coded prototype is the MVP, then you have many options, perhaps even just shipping the prototype as-is.

Typically, we’re in a situation where we already have an existing platform or application. We can’t just ship the prototype. We need to somehow merge the prototype into the existing code base.

So the more important question could be: How do we integrate this code into our existing platform or application?

Existing platforms and applications have data stores, UI frameworks, routing mechanisms, APIs, and other “primitives” used to create and evolve the systems.

We have some options.

Breakdown and Rebuild

Let’s treat the prototype as exactly that — a quickly created example of what we want to deliver. With this option we should audit the prototype and decompose it into the pieces used to create it. Once we have a breakdown of the system, we can start re-composing the concept using our own primitives.

This approach is what we’ve been doing for years for hastily created prototypes created during hackathons or discovery spikes. The process won’t feel any faster and certainly doesn’t feel “agentic”, but it works.

Vibe Code with Our Primitives

What if we could “teach” the vibe coding agent about the primitives we use? It could use our UI framework and APIs. It could understand our authentication system. It could store data into our data stores.

Agentic code assistants like Cursor and Copilot have rule or instruction files that can provide guidance and details for how the agent constructs the code, and can include code snippets and examples.

Agents can also use MCP servers that can be used to provide more semantic content to the agent about the frameworks and systems already in place.

This approach is how we create the most impact and productivity from code assistant tools. We might not be at this point yet, but we keep getting closer.

Thoughts on AI Assisted Coding

Wed, 02 Jul 2025 00:00:00 +0000

I’ve been using Copilot in VS Code to help build several different projects. I’ve also used Claude, Gemini, and ChatGPT to help start and evolve projects. I find the tools to be very helpful and I feel more productive using them. I also find myself having more fun when using the tools.

I have no doubt that these tools are here to stay and will become a standard part of software development.

What Kinds of Projects

I frequently use code assistance when creating Python-based utility scripts for manipulating data or automating websites. Agents have a solid “understanding” of Python and JavaScript, so I rarely have issues with the generated code itself. I typically need to help with managing the code structure and modularity.

My children and hobbies will sometimes nudge me to create more full-blown web apps. I like to structure those projects as vanilla HTML & JavaScript frontends, with Python-based API backends.

Agents will typically try to use a popular framework, but I push it towards modern JavaScript and Web Components.

My Approach

Make Useful Prompts: Start with an overview of the work you want to accomplish, and then kickoff the initial prompt with the basic starting point. I will make it clear that I want minimal dependencies, and specify any base frameworks we should use.

Get the Right Context: It’s easy for a code assistant to “forget” about existing code and recent changes. Use the context tools to make sure the code assistant is aware of relevant code, and be explicit about the context in that code. If you aren’t mindful of existing context, the code assistant is likely to create duplicate code, or not use existing code. Because I split up frontend and backend code, I will need to remind the code assistant about one or the other when refactoring. Agent-mode can have less issues in this situation.

Prepare for the Journey: Break up the process into tasks/steps. Yes, this means you need to plan out the workflow and have a general idea of the system and architecture. I’ve seen people talk about using a one-shot prompt/agent to build an entire solution, but that has not been my experience. Perhaps a personal bias, but I want to control the development flow. I’m less interested in a one-shot agent building a complete solution.

Options and Suggestions: Even though you have a general idea of the system and architecture, I frequently ask the code assistant for suggestions and options when starting subsequent steps. The code assistant may even recommend one of the suggestions over the others. This is where you need to think about the ramifications of the options. I occasionally veto all suggestions, but use the input to create a new prompt. Other times, I’ll use parts of one of more suggestions, but explicitly define the new approach in the next prompt.

Be Subtractive: Code assistant seem motivated to add code, even expanding beyond the scope of your prompt. Adding new capabilities to your code base can be exciting, but this is a siren song that leads to tech debt and code you no longer understand. I am the editor of this work and I trim and delete parts of the code I do not want at this point. I might want to add something in the future, but not now. One fascinating and frustrating behavior is the code assistant’s desire to always add back the code I edit away. We need to explicitly tell the code assistant we don’t want some parts of the code. Yes, some prompts are just refining the context for the model to avoid future indulgences.

Forest and Trees: Most of the time I end up working with the code assistant on specific pieces and areas of code. Every now and then it’s useful to have the code assistant appraise larger views of the project. This can be easier with agents, but can also be done with context-providing tools too.

Other People’s Thoughts

You might find some of these posts interesting and informative as well:

Building Agents: Creating JSONLogic from Intent

Sun, 27 Apr 2025 00:00:00 +0000

I had a lot of fun working through my “Exploring LLMs as Agent” series. I’m starting to dive a little deeper into some specific Agent use cases, so I’ll move away from “exploring” and start “building” agents.

At work, we’ve started configuring rules outside of code, especially for situations where a non-developer wants to create some rule-based logic. We’ve looked at JSONLogic for this use case, but there are other approaches too.

Rule logic can get complex, which makes it hard for those non-developers (and even developers) to create valid rules. Maybe this is a use case for an Agent?

TLDR: Here’s the code repo for the JSONLogic Agent. It’s part of my ADK code repo. Keeping the ADK Agents in the same repo seemed like a good idea.

JSONLogic Agent

I decided to build an Agent, based on Google’s Agent Development Kit (ADK) since I had some experience with the framework. I had a rough idea of what I thought I wanted to do:

Create a schema for JSONLogic
Create a schema for the dataset (JSON) being evaluated
Expose the schemas to the Agent, with a sprinkle of ReAct planning
Agent create JSONLogic on its own

Side Track

Something I’ve started doing recently is asking Claude or Gemini what they think of the approach, and if they have any other options or suggestions. In this case, Claude suggested having the Agent create an intermediate format, not actual JSONLogic, and then using a separate tool to convert from the intermediate format to JSONLogic.

I didn’t much care for that idea, but I ended up doing just that. Initial experiments showed that the Agent didn’t always create valid or even functional JSONLogic — which doesn’t really have a well defined schema. Now, those problems could have been my own failings, but I began to see opportunities having the Agent parse my intent of the rule, using a higher-level, simpler structure. So the plan became:

Create a schema for the Parsed Intent format
Create a schema for the dataset (JSON) being evaluated
Expose the schemas to the Agent, with a sprinkle of ReAct planning
Expose a tool to convert Parsed Intent to JSONLogic

Parsing Intent

Instead of trying to make sure the Agent knew all the syntax and nuance around JSONLogic, I created a simple format of operations and conditions (which can be nested) to capture the user’s intent of the rule. It looks like this (from the system prompt):

The intent should follow this format:
{
  "operation": "logical operation (AND, OR, NOT)",
  "conditions": [
    {
      "field": "data field",
      "operator": "One of the operators above",
      "value": "The value to compare against (for most operators)"
    },
    // Additional conditions...
  ]
}

For nested conditions, use this structure:
{
  "operation": "logical operation (AND, OR, NOT)",
  "conditions": [
    {
      "field": "data field",
      "operator": "operators from above",
      "value": "the value"
    },
    {
      "operation": "logical operation (AND, OR, NOT)",
      "conditions": [
        {
          "field": "data field",
          "operator": "operators from above", 
          "value": "the value"
        },
        // Additional nested conditions...
      ]
    },
    // Additional conditions...
  ]
}

One of the interesting aspects of using this approach is that I could create high-level operations that are not natively supported in JSONLogic directly, but could be implemented using JSONLogic — like macros. Here are some, again pulling from the system prompt:

When parsing conditions for the intent, use ONLY these operator keywords:

- **Comparison Operators**:
  - Basic: "equals", "notEquals", "greaterThan", "lessThan", ...
  - Range: "between", "betweenOrEqual"
  
- **Logical Operators**:
  - "AND", "OR", "NOT"
  
- **String Operators**:
  - "startsWith", "endsWith", "contains" (contains substring)
  
- **Array Operators**:
  - Empty checks: "empty", "notEmpty"
  - Item checks: "contains" (is value or values in array)
  - Index access: use dot notation to access array elements, e.g., "object.array.0"
  - Length checks: "lengthEquals", "lengthGreaterThan", "lengthLessThan"

Some of these operations are directly supported by JSONLogic. Others are supported if you understand the “Truthy & Falsy” nuance JSONLogic applies. The Range checks are a special case that JSONLogic supports, but using “between” is much more explicit. Item checks for a single item in an array are directly supported, but not checks with multiple items in an array — but that can be implemented using some. Length checks are not directly supported, but can be implemented using reduce, an idea I stole from a co-worker.

Checkout create_jsonlogic_from_intent in [jsonlogic.py](https://github.com/mfinkle/google-adk-agents/blob/main/jsonlogic_agent/tools/jsonlogic.py) to see how I hacked some of these “macro” operations together.

Results

The JSONLogic Agent feels like a success. I include two different test datasets and a series of predefined “questions” for the Agent to act on and output JSONLogic. The command line tool will even execute the JSONLogic against the same datasets and compare to an expected answer. Here’s an example:

Query: Find customers who completed the survey in less than 5 minutes (300 seconds) and reported at least one pain point

Planning:
1. Get the available fields from the schema to identify the correct field names for "time spent" and "pain points".
2. Parse the user's intent into a structured representation. This will involve identifying the conditions for "time spent" being less than 300 seconds and "pain points" array not being empty.
3. Convert the structured intent into a JSONLogic rule using the `create_jsonlogic_from_intent` tool.
4. Validate the fields in the JSONLogic rule against the available fields from the schema using the `validate_fields_from_jsonlogic` tool.
5. Return the complete JSONLogic rule.

The relevant fields are:
- metadata.timeSpent (time spent)
- questions.satisfaction.painPoints.response (pain points)

Parsed Intent:
{
 "operation": "AND",
 "conditions": [
   {
     "operator": "lessThan",
     "field": "metadata.timeSpent",
     "value": 300
   },
   {
     "field": "questions.satisfaction.painPoints.response",
     "operator": "notEmpty"
   }
 ]
}

Generated JSONLogic rule:
{
 "and": [
   {
     "<": [
       {
         "var": "metadata.timeSpent"
       },
       300
     ]
   },
   {
     "!=": [
       {
         "var": "questions.satisfaction.painPoints.response"
       },
       []
     ]
   }
 ]
}

Test result against sample data: False
? PASS - Result matches expected outcome: False

I added some utilities to make it a little easier to work with ADK Agents from mechanisms other than the adk command line tools.

What’s Next

I’ll keep building some Agents using ADK. I want to explore vector search and embeddings, and how those might be used together with an Agent.

Exploring LLMs as Agents: Google Agent Dev Kit

Mon, 14 Apr 2025 00:00:00 +0000

Up to this point, I have been experimenting with my own home-grown agent framework, based on Simon Willison’s great LLM project to handle wrapping the different LLMs. Armed with some experience, I want to start looking at real frameworks. Google released an Agent Development Kit at Google Next. It supports many of the features I have been playing with, including tool calling, planning, MCP, and local models. It also supports some advanced concepts like agent orchestration.

TLDR: Here’s the code repo for my initial ADK exploration.

Features

Google’s ADK has many features that are becoming common in agent frameworks:

Agent with support for multiple LLMs, including local using LiteLLM
Tools that support custom methods, a set of built-ins, 3rd party wrappers (including MCP), and even an easy AgentTool
Session and Memory capabilities with in-memory and persistent examples
Orchestration with different types of Agents to handle parallel, loops, and sequential workflows
Built-in system of Evals for testing your Agents

ADK has a command line interface that is somewhat unique to agent frameworks. It reminds me of the Flask command line system:

adk run <agent>
adk web
adk api_server

ADK also supports a mechanism to deploy your agents to Google infrastructure (Agent Engine or Cloud Run). I’m hoping this could be used to deploy to other infrastructure too.

Converting ToolAgent

I wanted to see how easily I could convert my ToolAgent to use ADK. The core Agent was fairly small and simple:

root_agent = Agent(
    name="multi_tool_agent",
    model="gemini-2.0-flash",
    instruction=SYSTEM_PROMPT,
    generate_content_config=types.GenerateContentConfig(
        temperature=0 # More deterministic output
    ),
    planner=PlanReActPlanner(),
    tools=[
        FunctionTool(func=get_weather),
        FunctionTool(func=get_datetime),
        FunctionTool(func=get_current_location),
        FunctionTool(func=get_zipcode),
        FunctionTool(func=calculate),
        FunctionTool(func=get_appointment_specialties),
        FunctionTool(func=get_available_appointments),
        FunctionTool(func=get_appointment_details),
        FunctionTool(func=book_appointment),
        FunctionTool(func=cancel_appointment),
    ],
)

This code uses a much simpler system prompt because much of the “PLEASE USE TOOLS” prompt is handled by the tool calling system. I use a ReAct planner to get the “thinking” aspects I had in ToolAgent. The tools themselves are still simple, mocked Python methods. ADK, like other frameworks, uses Python Typing and Docstrings to extract the specification details for the tool methods, so I changed my code (for the better):

def get_available_appointments(specialty: str) -> list[dict]:
    """
    Gets the available appointments for the given specialty.
    Args:
      specialty (str): the specialty name
    Returns:
      list: a list of available appointments
    Example:
        >>> get_available_appointments('dentist')
        [{'id': '1', 'date': '2022-01-01', 'time': '10:00 AM'}, ...]
    """
    <IMPL REMOVED>

Here’s a simple exchange that required a few tools:

user: what dentist appointments are available tomorrow?

[multi_tool_agent]: /*PLANNING*/
1. Get the current date using the `get_datetime` tool.
2. Extract the date from the response and calculate tomorrow's date.
3. Get the available dentist appointments using the `get_available_appointments` tool with the specialty set to "dentist".
4. Filter the appointments to only include those that are on tomorrow's date.
5. Respond to the user with the available appointments for tomorrow.

/*ACTION*/

[multi_tool_agent]: /*REASONING*/
1. The current date is 2025-04-13.
2. Tomorrow's date is 2025-04-14.
3. Now I need to find the available dentist appointments.

/*ACTION*/

[multi_tool_agent]: /*REASONING*/
I have the available dentist appointments. I need to filter the appointments to only include those that are on tomorrow's date, which is 2025-04-14.
The available appointments for tomorrow are:
- id: 3, time: 11:00 AM
- id: 4, time: 3:00 PM

/*FINAL_ANSWER*/
The available dentist appointments for tomorrow, 2025-04-14, are:
- id: 3, time: 11:00 AM
- id: 4, time: 3:00 PM

Overall, I thought it was pretty easy to get a basic agent working. Maybe I’ll try to create a WebAgent too. I’ll keep exploring some of the more advanced concepts to see how those can be used.

Checkout the code repo

Exploring LLMs as Agents: Local Models

Sun, 13 Apr 2025 00:00:00 +0000

I finally got around to exploring local models, which is surprisingly simple to set up. I wanted to see how well a local model would perform in ToolAgent compared to the remote models (mostly Gemini) I have been using.

For more context on my explorations , checkout the previous posts: Minimalist Approach, Taking Action, Planning via Prompting, Tools & Benchmarking, WebAgent Gets a Refactor, and Model Context Protocol (MCP). Take a look at the repository to see the code.

Local Models

Local large language models (LLMs) are models that can be run directly on a personal device, such as a laptop, desktop computer, or smartphone, rather than relying on cloud-based servers. They offer features like privacy, security, compliance, offline availability, and lower operating costs.

One of the easiest ways to get started with local models is with Ollama, a system to download, manage, and run models on your laptop. Another frequently used resource is Hugging Face, which I’ve heard called the “GitHub of Models”.

My ToolAgent is based on Simon Willison’s llm project, which has plugins for several different local model system, including Ollama (llm-ollama) and Hugging Face models via MLX (llm-mlx). I was able to use both approaches to test out some local models.

Which Models

There are a mind-boggling number of local models and variations to choose from. Hugging Face has a leader board. Based on what I saw other people talking about, I decided to start with a few:

Ollama mistral-small3.1:24B (4bit) - 15GB
Ollama gemma3:27B (4bit) - 17GB
mlx-community/Qwen2.5-0.5B-Instruct-4bit — 278 MB
mlx-community/Mistral-7B-Instruct-v0.3-4bit — 4.08 GB
mlx-community/Mistral-Small-24B-Instruct-2501-4bit — 13.26 GB
mlx-community/Llama-3.3-70B-Instruct-4bit — 40 GB

You’ll notice a few specifications, like 24B and 4bit. These relate to some performance tradeoffs you need to consider when running LLMs on your laptop.

Model size is specified by the number of parameters the model supports, 7B is 7 billion and 27B is 27 billion. Generally, the more parameters a model supports, the better the model is at reasoning and following instructions.

Quantization (4-bit, 8-bit, 16-bit) is about how precisely the model stores its weights. Lower quantization means smaller files sizes and less RAM is required to run the model. The downside is less accuracy or ability to reason.

Results

Once I had a few local models installed and ready to go, it was time to try them out with ToolAgent! Because ToolAgent is based on the llm library, I just had to pass the model name using the rules for the plugin (llm-ollama or llm-mlx) I wanted to use.

Speed: Local models are slower than cloud-based models. Ollama and MLX use GPU on my Macbook Pro M1 with 64GB, but it’s sluggish. MLX is built for Apple silicon, so I expect the model was working as well as it could on a single machine.

Reasoning: The smaller models (Qwen2.5-0.5B and Mistral-7B) were not able to function as an agent. They couldn’t follow the instructions in the system prompt and they couldn’t call any tool functions. The larger models (Mistral-Small-24B and Llama-3.3-70B) were able to function as agents. They didn’t work as smoothly as the cloud-based models, but they were very functional.

An exchange with Mistral-Small-24B:

You: what is tomorrow's date?

Thought: "I need to determine the current date and time to calculate tomorrow's date.",
"type": "call_tool", "tool": "get_datetime", "param": null

Thought: "I have the current date, which is 2025-04-13. To find tomorrow's date, I need to add one day to the current date.",
"type": "call_tool", "tool": "calculate", "param": "2025-04-13 + 1 day"

Thought: "I cannot calculate dates directly. I need to determine the date by using the current date and adding one day manually.",
"type": "output", "value": "... output text ..."

Agent: Tomorrow's date is April 14, 2025.

You: what dentist appointments are available tomorrow?
Thought: "I need to determine the available dentist appointments for tomorrow. First, I need to check the available specialties to ensure 'dentist' is one of them.",
"type": "call_tool", "tool": "get_appointment_specialties", "param": null

Thought: "I have confirmed that 'dentist' is an available specialty. Next, I need to find the available dentist appointments for tomorrow.",
"type": "call_tool", "tool": "get_available_appointments", "param": "dentist"

Thought: "I have the list of available dentist appointments. I need to filter the appointments for tomorrow's date, which is April 14, 2025.",
"type": "output", "value": "... output text ..."

Agent: Here are the available dentist appointments for tomorrow, April 14, 2025:
Appointment ID: 3, Time: 11:00 AM
Appointment ID: 4, Time: 3:00 PM

It made a bad assumption with calculating the next day, but recovered. Otherwise worked well.

What’s Next

I have started to look at real agent frameworks, like Pydantic-AI. I think I have learned enough of the basics to be dangerous and it’s time to move beyond my own toy framework. Google released an Agent Development Kit at Google Next. It supports many of the features I have been playing with, including tool calling, planning, MCP, and local models. It also supports some advanced concepts like agent orchestration.

Let’s start exploring Google’s ADK!

Exploring LLMs as Agents: Model Context Protocol

Sun, 30 Mar 2025 00:00:00 +0000

I will likely run out of topics to cover in the “LLMs as Agents” series, eventually. I wanted to explore creating a chat UI and what I could do with embeddings but I didn’t do that. For more context , checkout the previous posts: Minimalist Approach, Taking Action, Planning via Prompting, Tools & Benchmarking, and WebAgent Gets a Refactor. Take a look at the repository to see the code.

I’m back to cover the new hotness in the agent space: Model Context Protocol (MCP) from Anthropic

MCP is having its jQuery moment, and that’s not an insult. Even OpenAI is adopting MCP. I’ve learned about hooking tools up to LLM-based agents, and it was always clear that competing standards for how to connect LLM-based agents to tools would create fragmentation and hurt growth. MCP is an open protocol for building agents. The MCP system contains:

MCP Hosts: Programs like Claude Desktop, IDEs, or other tools that contain a client
MCP Clients: Protocol clients that maintain 1:1 connections with servers
MCP Servers: Lightweight programs that each expose specific capabilities through the standardized Model Context Protocol

There is a ton of information available for you to learn about MCP. This post is not teaching you about MCP.

This post is about MCP-enabling the ToolProvider and ToolAgent framework I created! That’s right, I’m exposing ToolProviders as MCP Servers and ToolAgent as an MCP Client — and I’m doing it with zero dependencies on any MCP framework. I like to see how things work under the covers. Having spent time building Browsers, I also believe that any open protocol needs to have multiple implementations, even crappy ones like mine.

Getting Started

Before I started building an MCP Client and Server from scratch, I made a few decisions:

I was going to use Copilot to help do the work. It just seemed silly to try this otherwise.
I copy/pasted many sections of the MCP Specification into a single, long Markdown document. I had to give Copilot the right context. I also used the MCP Schema.
I was going to implement the stdio transport. HTTP SSE and WebSockets (I think) are also available, but a local system was good enough to learn how this works.
I was only going to support “Tools” in my Client and Server for now. You can also support “Resources” and “Prompts”.

ToolProvider MCP Server

Copilot handled all the heavy lifting here. I asked to create an MCP Server, using the MCP specification, that was implemented using ToolProviders. ToolProviders already supported everything an MCP Server needed to support tools, so Copilot just had to map from my basic JSON schema to the JSON schema used by MCP.

Copilot was able to create a basic stdio-based request/response system, and was able to add the JSON-RPC message passing. Copilot was able to glean some of the messages and the message flows from the MCP specification.

I install Claude Desktop to test it out. It didn’t work on the first try. We missed some nuance of the JSON-RPC message passing, and we had to update some message types based on the newer schema. It did not take long before we had my ToolProvider tools running in Claude as an MCP Server.

ToolAgent MCP Client

Again, Copilot did most of the work. We already had a basic stdio-based system working, and we had several other examples of ToolAgent being used in different ways: terminal chat, web-based chat, and some tests — so it didn’t take long to have a a basic system with MCPToolProvider (since ToolAgent connects to ToolProviders, we have to map MCP Servers to ToolProviders) and an ToolAgentMCPClient that wraps ToolAgent itself.

I also installed a 3rd party MCP Server (filesystem) so I could test a Server that wasn’t something I wrote.

We had to tweak the Client stdio request/response mechanism to get things working reliably. Timing, buffering, and flushing are important! Because the Client is spawning Servers as processes, I had to add some code to make sure the Client was cleaning up the spawned Servers on shutdown or in failure situations.

I was able to get both ToolProvider MCP Server and filesystem MCP Server working in my ToolAgent-based MCP Client.

Connecting to MCP server: filesystem
Connected to secure-filesystem-server v0.2.0
Available tools: 11
Connecting to MCP server: mock
Connected to ToolProvider MCP Server v0.1.0
Available tools: 19

Chat with the agent (type 'exit' to quit, 'tools' to list tools):
> What files do I have in my Desktop folder?

Thought: "I need to find out which directories I have access to before I can list the files in the Desktop folder.",
"type": "call_tool", "tool": "list_allowed_directories"

Thought: "I can now list the files in the Desktop folder since I know I have access to it.",
"type": "call_tool", "tool": "list_directory", "param": "/Users/mfinkle-mac/Desktop"

Thought: "I made a mistake in the previous JSON format. The parameter for list_directory should be an object.",
"type": "call_tool", "tool": "list_directory", "param": { "path": "/Users/mfinkle-mac/Desktop"}

Thought: "I have successfully listed the files and directories in the Desktop folder.",

Agent: Here are the files and directories in your Desktop folder:
.DS_Store
.localized
Reactions (directory)
Screen Recording 2024-06-24 at 5.12.08 PM.mov
Screen Recording 2024-07-19 at 10.30.18 AM.mov
Screen Recording 2024-11-04 at 12.53.58 PM.mov
Screen Recording 2025-03-17 at 11.56.51 AM.mov
Stuff (directory)
live-chat-sample.mov

You can see ToolAgent processing the requests, but it’s using the filesystem MCP Server to perform the actions.

The code for both the ToolProvider MCP Server and the ToolAgent MCP Client are in the LLMs Agents repo.

Learnings

MCP is a very handy mechanism to connect LLM-based agents to different types of tools and resources. We’ve seen “broker” or “middleware” systems like this in the industry before and there is usually a lot of benefit for the ecosystem while a dominate player exists. I remember using COM and ODBC.

There are a lot of MCP Servers being created everyday (curated list). Many are of questionable value, but that’s the beauty to the early adopter phase. We’ll start to see some examples that push the boundaries, and that will be interesting.

There are fewer MCP Clients, but that will certainly change. I wonder how people (or IT/Security) will feel about all of this data being moved around. It makes me want to start looking into local models (another topic I failed to start).

What’s Next

I want to refactor the ToolProvider approach to defining tools. The MCP approach is cleaner and would make the mapping code (from ToolProvider to MCP Server) simpler too. The Playwright team released a Playwright MCP Server, which when used in my ToolAgent MCP Client could mean WebAgent doesn’t need to exist anymore.

With the growing number of easy to connect MCP Servers, I want to start investigating ways to orchestrate all those tools into a meaningful and useful Agent.

Also:

Local Models
Chat UI (and non-Chat UI) experiences
Embedding and Vector search

Search

Mon, 24 Mar 2025 23:00:00 -0300

Search

Here you can search for indexable pages, like posts and alike. There’s also support for URL param “q” which then auto searches upon page load event.

It's necessary to enable Javascript

Exploring LLMs as Agents: WebAgent Gets a Refactor

Thu, 20 Mar 2025 00:00:00 +0000

My fifth post on LLMs as Agents. I just it’s a series now.

After refactoring ToolAgent, I decided to do the same with WebAgent, building it on the ToolAgent system. For more context and a refresher on WebAgent, checkout the previous posts:

Minimalist Approach: I kicked off my exploration by making two agents using the bare-minimum dependencies. I wanted to learn the concepts, before using a do-it-all framework.
Taking Action: I decided to focus on the tool_agent and add some read/write tools for the agent to use. I also discovered the need to send clear and consistent context in prompts.
Planning via Prompting: I looked at ways to improve the agent outcomes by using better approaches to planning. I settled on using ReAct, with some Few-Shot prompts, and had the LLM do some Chain of Thought (CoT) output.
Tools & Benchmarking: I refactored the ToolAgent code into a reusable base class, added some additional mock tools, and even some basic benchmarking.

Take a look at the repository to see the code.

WebAgent

After refactoring ToolAgent, I decided to see if I could build WebAgent on top of ToolAgent. I think it turned out pretty nicely. Again, Copilot helped too.

WebAgent now derives from ToolAgent, and PageManager (the wrapper around Playwright for browser automation) became a ToolProvider. All of the wrappers for Playwright automation became tools the agent could use. This worked better than I thought.

Starting task: Navigate to wikipedia.org and find the featured article of the day

Thought: "Okay, I will start by navigating to wikipedia.org. Then I will analyze the page content to find the featured article of the day. Finally, I will extract the text of the featured article and output it."
Tool: "navigate", Param: "https://www.wikipedia.org"

Thought: "Now that I have navigated to the Wikipedia homepage, I need to identify the element containing the 'Featured article'. I will use the 'get_page_content' tool to get the HTML content of the page and analyze it to find the appropriate CSS selector.",
Tool: "get_page_content"

Thought: "The HTML content I retrieved is for the main wikipedia.org page, which is just a language selection portal. I need to navigate to the English Wikipedia to find the featured article. I will click the English link.",
Tool: "click", Param: "#js-link-box-en"

Thought": "Now that I have clicked the link, I need to wait for the page to load.",
Tool: "wait_for_navigation"

Thought: "Now that the English Wikipedia page has loaded, I need to find the 'Featured article' section. I will use the 'get_page_content' tool to get the HTML content of the page and analyze it to find the appropriate CSS selector.",
Tool: "get_page_content"

Thought": "I have analyzed the page content and found that the 'Featured article' is located within a div with the id 'mp-tfa'. I will now use the 'get_text' tool to extract the text from this element.",
Tool: "get_text", Param: "#mp-tfa"

Thought: "I have successfully extracted the text of the featured article. Now I will output the result."

===== TASK SUMMARY =====
Task: Navigate to wikipedia.org and find the featured article of the day
Status: incomplete
Duration: 12.39 seconds
Input tokens: 68486
Output tokens: 827

The token usage was noticeably higher than my ToolAgent experiments. This was almost entirely due to get_page_content which was pushing the entire HTML content of the page into the LLM.

I had to fix that.

Learnings: Managing Context

As I said, my initial implementation of get_page_content just returned the entire HTML content back into the LLM. This wasn’t good. Because the LLM was using the page content to decide what HTML elements it needed to interact with, I could take some steps to limit which HTML elements I returned.

Step 1: Regex

I apologize up front that I even tried this, but I just wanted to see if I could get some quick wins. I asked Copilot to create an implementation of get_page_content that filtered out any elements that were not: Form elements, Elements with IDs, Elements with Roles or Aria-Labels, and a small set of special class names (like .btn).

The result was a much smaller content context, but there were issues. Of course there were issues! I lived through the “Use regex on HTML” era and it’s not good.

Step 2: Playwright

I was already using Playwright, so why not use it in a way similar to how we’d use BeautifulSoup? One well crafted prompt later, and Copilot had a very functional version.

Copilot: You’re absolutely right! Using Playwright directly to select elements gives us much better control and maintains the structure and order of the original page.

Even Copilot was relieved we moved away from regex.

The Playwright-based version had some other benefits:

Elements are extracted in the same order they appear in the DOM, preserving visual hierarchy
Filtering allowed me to only includes important attributes like id, class, role, etc
Executing some code in the DOM allowed me to only include visible elements

These helped the LLM make better decisions about the HTML elements it needed to interact with during the task. It also dramatically reduced the amount of input tokens sent to the LLM.

Step 3: Greedy LLM

The last thing I noticed and corrected was the LLM’s desire to ask for the page content whenever it got confused with CSS selectors. And yes, it gets confused. The LLM would fetch the page content, even when the page had not changed. Token usage goes up, and for the most part, the LLM didn’t zero-in on the right CSS selector. It likely just started guessing.

I added a is_maybe_dirty check in PageManager which was set to false after any call to get_page_content. I set is_maybe_dirty to true after: navigate, click, and type_text.

If is_maybe_dirty=false in get_page_content, I return an error result of “Use previously extracted page content”. The LLM correctly interpreted the result and knew it should stick with the current HTML content it already had.

Here’s an example run: “Use ‘https://duckduckgo.com’, search for ‘python tutorial’, and return the title of the first result found” (yes, it’s too fast to read so use the scrubber)

What’s Next

If I keep pushing on ToolAgent and WebAgent, I run the risk of starting to build simple frameworks. That might not be the worst outcome, but wasn’t my intention when starting these projects.

Instead, I want to explore some embedding and vector search projects. I’ve been using Simon Willison’s llm library, and it already includes some support for creating embeddings. Both SQLite and DuckDB have support for vector search. Sounds like I’ll have a lot to keep me busy.

I also want to begin using some local models. Again, llm already support this, so I’m unblocked.

Exploring LLMs as Agents: Tools & Benchmarking

Sun, 16 Mar 2025 00:00:00 +0000

I spent some time refactoring the Tool Agent code, added some additional mock tools and even some basic benchmarking. For more context, checkout the previous posts:

Minimalist Approach: I kicked off my exploration by making two agents using the bare-minimum dependencies. I wanted to learn the concepts, before using a do-it-all framework.
Taking Action: I decided to focus on the tool_agent and add some read/write tools for the agent to use. I also discovered the need to send clear and consistent context in prompts.
Planning via Prompting: I looked at ways to improve the agent outcomes by using better approaches to planning. I settled on using ReAct, with some Few-Shot prompts, and had the LLM do some Chain of Thought (CoT) output.

Take a look at the repository to see the code.

Code Refactor

The biggest change to the code is the somewhat large refactor to extract the agent-specific code into a ToolAgent class and the tool-specific code into a set of ToolProvider classes. ToolProvider classes are registered with the ToolAgent:

# Register tool providers
agent = ToolAgent(model_name='gemini-2.0-flash')
agent.register_provider(UtilityToolProvider())
agent.register_provider(AppointmentToolProvider())

ToolAgent can then be used in different ways. tool_agent_test.py is a way to run the agent in a terminal window. I’ve been playing with using the agent in a chat system too, but more on that next time.

python tool_agent_test.py

Tools

I wanted to keep adding more tools, which contributed to the code refactor. I couldn’t keep adding more code into the one Python file. With the new tools, I explored two new areas (for me):

LLM-based Tools: A tool that was implemented using a secondary LLM. For a given set of topics, I wanted to “guess” which topics might be of interest to a user based on their input. LLMs are good at that stuff, so I made a simple tool call that used a separate system prompt. (I just realized that I didn’t add token tracking to that tool!)
Multi-Parameter Tool Calls: Up until now, my tool calls took zero or one parameters. I wanted to start using multiple parameters, so I introduced an object parameter type with a simple schema. In the future, I want the agent to provide feedback when required parameters are not provided.

Benchmarking

I mentioned in the last post that I needed to start doing some benchmarking on the agent. The code factor made it somewhat easy for Copilot (yes, I do some vibe coding too) to whip up a very basic benchmarking harness. Some details (more in the readme):

Test Cases: Run multiple predefined test cases
Success Metrics: Track success rates, execution times, and token usage
Reporting: Get detailed reports on test results
CSV/JSON Export: Export results for further analysis
Token Usage: Track and analyze token consumption

I added simple token tracking because I was running the benchmarking suite quite a bit and wanted to know what kind of money I was spending. Turns out, it was less than a penny a run, but it’s good to know.

Learnings

I was running the benchmark suite trying to figure out why tests kept failing. The benchmarking framework measures success based on two metrics: (1) Did the agent use the minimum set of required tools? (2) Did the agent output a few key words in the final response. Yes, those aren’t the best success checks, but it’s what I have for now.

More Jedi Prompt Tricks

The agent was usually failing at using the required tools. Looking at the debug [thought] output from the agent, I noticed that it would assume knowledge instead of using the tools. I found a blog post from Pinecone that talks about this when using LangChain too. They suggested adding explicit items to the system prompt. I already had something along the lines of “ALWAYS use the tools. NEVER assume information” but that wasn’t enough. I added the following:

IMPORTANT: Do not assume or guess or hallucinate information. Call the tools to get ALL needed information.
    - You DO NOT know the current date or time. Use tools to determine the current date and time.
    - You CANNOT do math. Use tools to calculate math.

Temperature

For the randomness on the output, I saw this note in the same Pinecone blog post:

Here we initialize the LLM with a temperature of 0. A low temperature is useful when using tools as it decreases the amount of “randomness” or “creativity” in the generated text of the LLMs, which is ideal for encouraging it to follow strict instructions — as required for tool usage.

Honestly, I had not played with any LLM options yet, but added the temperature=0 setting and it did help.

More JSON Output Handling

I already had some code to handle the times when the LLM returned JSON using Markdown code fences, but there were times when the LLM would apologize for a mistake and append the JSON. The results were correct, but my code failed to extract it! So I added more defensive measures to safely extract JSON from LLM responses. Yes, I should update to use LLM built-in support for JSON schemas at this point.

With these changes, I regular had 100% success in my benchmarking runs! This means I need to make the tests harder.

You: I want to get a haircut tomorrow morning

Thought: I need to find out the current date to determine what tomorrow is. I will use the get_datetime tool to get the current date and time.
Thought: The current date is 2025-03-15. I need to check for available appointments for a haircut. Since there is no haircut specialty, I should first check the available specialties.
Thought: Okay, 'hair' is a valid specialty. I will now check for available 'hair' appointments.
Thought: I have a list of available hair appointments. The user wants an appointment tomorrow morning, which is March 16th. There are two appointments available on March 16th: 11:00 AM and 2:00 PM. The user wants a morning appointment, so the 11:00 AM appointment is the best fit. I will ask the user if they want to book appointment ID 11.

Agent: I see two appointments available for hair on March 16th: 11:00 AM and 2:00 PM. Would you like to book appointment ID 11 for 11:00 AM?

What’s Next

As I started to explore some non-chat user experiences, I discovered that I will need to agent to provide more structured output. Right now the output is very conversational, which works well for chat, but won’t be easy to use with other types of UX. I found some discussion on taming the output of an LLM in this blog post from Tryolabs.

I’ve also started using ToolAgent in my own chat server. I’m sure this branch of experimentation will continue as well.

Exploring LLMs as Agents: Planning via Prompting

Sun, 09 Mar 2025 00:00:00 +0000

My exploration of how to use LLMs as Agents continues. In this post, I’m looking at ways to improve the outcomes by using better approaches to planning. For more context, checkout the previous posts:

Minimalist Approach: I kicked off my exploration by making two agents using the bare-minimum dependencies. I wanted to learn the concepts, before using a do-it-all framework.
Taking Action: I decided to focus on the tool_agent and add some read/write tools for the agent to use. I also discovered the need to send clear and consistent context in prompts.

Take a look at the repository to see the code.

Planning & Reasoning

I have seen a lot of people talking about “planning” and “reasoning” as crucial parts of an LLM agent system, so I started to read and learn more about different strategies. There is obviously a ton written about LLMs and Agents all over the internet, but I stumbled upon a useful web site that collects a lot of the information and has references to where the ideas originated.

Prompt Engineering Guide, not surprisingly, focuses on many different prompting strategies. It turns out, most of the different planning and reasoning strategies are based on different prompting strategies. Here are some of the strategies I progressed through:

Zero-shot: I think of this strategy as “hoping the LLM was trained well enough to just know the answer”. Hope is not a strategy, but you’d be surprised how many people believe the LLM can just figure everything out on its own.
Few-shot: This strategy enables in-context learning and depends on adding some examples to your prompt to help guide the model to better outcomes. My initial tool_agent used this approach before I even knew what it was called.
ReAct (Reasoning + Acting): This strategy combines reasoning and acting with LLMs into the prompt. ReAct prompts LLMs to generate reasoning and actions for a task. Asking the LLM to “think about the step” is enough to create significant improvements to the outcomes.

Funny thing: I thought I wanted to try using something called Chain of Thought (CoT), which is another prompt strategy to get the LLM to think about the steps. CoT ended up being a pain to implement in a simple way. I ended up wanting to merge the “thought” (reasoning) and “action” parts together into the same response — which turns out to basically be ReAct.

Example Flows

I honestly don’t know if the ReAct strategy is yielding better outcomes than the Few-shot approach I was using. I guess I need to set up benchmarks next? The flows are promising though, and the ReAct approach didn’t add much to the size of the code.

You: What's the weather in Beverly Hills?

Thought: The user wants to know the weather in Beverly Hills. First, I need to get the zipcode for Beverly Hills, then I can use that to get the weather.
Thought: Now that I have the zipcode for Beverly Hills, I can get the weather.
Thought: I have the weather information. Now I need to present it to the user.

Agent: The weather in Beverly Hills is 75 F and Sunny.

Here’s a longer one that uses the appointment scheduling tools:

You: I want to book a haircut

Thought: The user wants to book a haircut. I need to find the appropriate specialty to book an appointment for. I should first check the available specialties.
Thought: The available specialties are hair, dentist, and vision. Since the user wants a haircut, the relevant specialty is 'hair'. Now I need to find available appointments for 'hair'.
Thought: I have found the available appointments for a haircut. Now I need to present them to the user and ask them to choose one.

Agent: Here are the available appointments for a haircut:

* ID: 10, Date: 2025-03-09, Time: 10:30 AM
* ID: 11, Date: 2025-03-10, Time: 2:00 PM
* ID: 12, Date: 2025-03-16, Time: 11:00 AM
* ID: 13, Date: 2025-03-16, Time: 3:00 PM

Please tell me the ID of the appointment you would like to book.

You: 12

Thought: The user has selected appointment ID 12. Now I need to book this appointment.
Thought: The appointment has been successfully booked. I need to inform the user.

Agent: Great! Your appointment has been booked successfully for March 16, 2025, at 11:00 AM. The appointment ID is 12.

Learnings

Reflection and Correction

There are others ways we can improve the outcomes of the LLM agent beyond planning and reasoning. Reflection and correction is a way to help the model learn from its mistakes. I already had some basic checks in place to catch invalid JSON and unknown tools, but I made some additions to the code to help:

Validation of the JSON actions for expected fields
Validation loops to tell the LLM about any validation failures and allow the LLM to try again.

Prompt & Tool Consistency

I also realized, once again, that I was confusing the LLM with some of the ways I framed the prompt and tools.

I referred to both “tools” and “functions” in the initial prompt and subsequent conversational prompts. I decided to only use “tool” and changed all “function” references the LLM saw to “tool”.
I made some of the tool names ambiguous and the LLM would try to use tools that were not appropriate. An easy one is get_specialties which returns the list of specialties used for appointment booking. The LLM would sometimes avoid using the tool to figure out what types of appointments could be booked and would try other tools or guess. I changed the tool name to get_appointment_specialties and the LLM always used it correctly.

What’s Next?

I want to add more tools. I want to add tools that use secondary LLMs and I want to add tools that allow me to use the Agent for my own tasks.

I want to explore different UX around agents. Chat is an obvious choice, and I will likely expose tool_agent into a simple Web Chat system to play with it more and try to use it for actual tasks.

As mentioned above, I may look into benchmarks too.

Exploring LLMs as Agents: Taking Action

Sun, 02 Mar 2025 00:00:00 +0000

I’m still exploring how to use LLMs to build agents. In the first post, I described some of the motivation and approach. I started working on two agents:

web_agent: A basic agent that completes a web-based task using browser automation actions.
tool_agent: A basic agent that completes a task using a set of supplied tools or methods.

I’ve been focused more on tool_agent, trying to expand the types of tools I can provide. I had a suggestion to add some read/write type tools and scenarios, so I decided to try that out.

Taking Action

an agent is characterized by the environment it operates in and the set of actions it can perform. — Chip Huyen Agents

I want to create a more realistic set of actions for the agent to use. Scheduling appointments seemed like a straightforward scenario with some interesting possible flows. I added a few tool methods and a mocked appointment data store.

get_specialties(): Retrieves a list of available specialties for scheduling appointments.
get_available_appointments(specialty): Retrieves available appointments for a given specialty.
book_appointment(booking_data): Books an appointment based on provided booking data. The agent can pass in an appointment ID or a combination of a date and time.
get_my_appointments(): Retrieves the list of appointments booked by the user.
cancel_appointment(appointment_id): Cancels a booked appointment based on the provided appointment ID.

Checkout the tool_agent.py code for more detail on the way the tools work and the way the tools are added to the system prompt.

Example Flows

I was pleased with how well this rudimentary system worked. Without many tweaks (some on that below), I was able to get some useful exchanges.

You: what kind of appts can I book?
Action raw: {"type": "call_function", "tool": "get_specialties"}
Function result: ["dentist", "vision", "hair"]
Action raw: {"type": "output", "value": "You can book appointments for dentist, vision, and hair."}
Agent: You can book appointments for dentist, vision, and hair.

The LLM did a good job of taking some ambiguity (“appts” and “book”) to figure out that it needed to use get_specialties to answer the question.

You: I want to get a haircut
Action raw: {"type": "call_function", "tool": "get_specialties"}
Function result: ["vision", "dentist", "hair"]
Action raw: {"type": "call_function", "tool": "get_available_appointments", "param": "hair"}
Function result: [... list of appointments ...]
Action raw: {"type": "output", "value": "... output ..."}
Agent: Okay, here are the available hair appointments:

* ID: 10, Date: 2025-03-01, Time: 10:30 AM
* ID: 11, Date: 2025-03-02, Time: 2:00 PM
* ID: 12, Date: 2025-03-08, Time: 11:00 AM
* ID: 13, Date: 2025-03-08, Time: 3:00 PM

In this exchange, the agent was able to figure out that it needed to display “hair” appointments. Interesting: the agent came up with the output display format on its own. When I saw the agent using this format, I added the ability to reference appointments by ID in the tool methods.

You: show me my appointments
Action raw: {"type": "call_function", "tool": "get_my_appointments"}
Function result: [{"id": "3", "date": "2025-03-02", "time": "11:00 AM", "specialty": "dentist"}]
Action raw: {"type": "output", "value": "... output ..."}
Agent: Here are your booked appointments:

* ID: 3, Date: 2025-03-02, Time: 11:00 AM, Specialty: dentist

You: cancel the dentist appointment
Action raw: {"type": "call_function", "tool": "cancel_appointment", "param": "3"}
Function result: {"status": "success", "message": "Appointment canceled successfully."}
Action raw: {"type": "output", "value": "... output ..."}
Agent: Your appointment has been canceled successfully.

I thought I could easily confuse the agent by trying to cancel an appointment by the type (specialty) of the appointment, but the agent used the correct ID for the appointment when cancelling via the tool method.

Learning

At one point, the agent started returning invalid JSON. I assumed I need to be more strict in my system prompt, but I discovered an error. Since I was surrounding my Python strings with double quotes, the inner quotes were single quotes. My JSON examples were using single quotes — which is illegal in JSON. The LLM was relying on those examples, so I was causing the LLM to emit invalid JSON.

As I was looking at the code for similar issues, I noticed I was sending Python formatted object output back into the conversation. The LLM was handling it well enough but I decided to change the output to be legal JSON output:

- response.conversation.prompt(f"Function result: {function_result}")
+  function_result_json = json.dumps(function_result)
+  response = conversation.prompt(f"Function result: {function_result_json}")

What’s Next?

Most of Chip Huyen’s post on Agents talks about “planning”, but I have not really adding any planning specific code to tool_agent yet. Right now, I am getting by with whatever amount of planning the LLM can create itself.

I want to learn more about planning, and how to add a little code to help the agent deal with even more complicated scenarios.

Exploring LLMs as Agents: A Minimalist Approach

Sat, 01 Mar 2025 00:00:00 +0000

Large Language Models (LLMs) are powerful tools for generating text, answering questions, and coding. We’ve moved beyond generating content, and LLMs are now being used to take actions as agents — independent entities that can act, use tools, and interact with their environment. You probably already know all of this.

I wanted to explore using LLMs as agents, but I like to get an understanding of the underlying components before using high-level frameworks to hide all of the minutiae and make the process of building production-ready systems. Understanding how the different components work and interact is important to my own learning process.

That’s exactly what my LLM Agents project sets out to do. Instead of relying on frameworks that abstract away the details, the project takes a bare-bones approach to learning how LLMs can function as agents. By minimizing dependencies, I can get a clearer understanding of the challenges, possibilities, and mechanics of building LLM-powered agents.

Why Minimal Dependencies Matter (To Me)

Many existing frameworks promise powerful LLM agent capabilities, but they often come at the cost of hiding the underlying complexities. While these frameworks can be useful, starting with minimal dependencies allows us to:

Understand the fundamentals: How does an LLM process information to take actions? How does the system prompt impact the effectiveness of the agent?
Explore limitations: What challenges arise when an agent tries to perform a multi-step task? How does the shape of the tools (functions or APIs) impact how the agent can process the flow.
Control the design: Without being boxed into a framework’s way of doing things, we can experiment freely. We can then use this knowledge to help pick the right type of framework for more advanced and production use cases.

This project keeps things simple, using only a lightweight LLM library, Simon Willison’s great llm Python library (code & docs), and Playwright for handling web automation.

These agents are not production ready, but they are trimmed down enough to see the mechanisms at work.

Meet the Agents

The repository contains two primary agents:

Web Agent: Navigating the Web with LLMs

The Web Agent is designed to interact with websites using the Playwright Python library. Instead of treating a webpage as structured data, this agent lets an LLM interpret raw HTML and decide what actions to take—whether that means clicking a button, typing into a form, or extracting text. I wanted to see how well an agent could navigate something as confusing as a modern website. If you’ve ever done a “view source” or “inspect” on a modern webpage, you know what I mean.

How It Works

A PageManager class handles the browser automation using Playwright.
The LLM generates the next action based on the current page content and the assigned task.
Two modes are available:
- Non-conversational mode: Every step is processed independently.
- Conversational mode: The agent maintains memory across multiple interactions, reducing the need to repeat context.

Example Task:

web_agent_conversation("gemini-2.0-flash", "Search for 'LLM agents' and return the first result's title.", "https://duckduckgo.com/")

This runs a search query and extracts the first result’s title, all without predefined scraping rules.

How Did It Go

At one point, the agent was not using valid CSS selector syntax and couldn’t “click” the search button. In spite of not getting to the search results page, the agent returned a “successful” answer. I wondered if the LLM was somehow using its trained knowledge to find a valid answer, but I could not find the result anywhere. I searched DuckDuckGo and Google for the title.

I added the “Explain how you solved the task” prompt and the agent replied that since it was not able to get to the search results, it created a hypothetical answer.

I did two things:

I told the agent it was not allowed to make up answers. Just fail gracefully.
I gave the agent examples of valid CSS selectors for id, class, and attribute selectors. This really improved the CSS selector accuracy. I had hoped the LLM’s training would have been good enough.

The conversational mode, unsurprisingly, could finish tasks with fewer steps. Memory and retained context matter.

Tool Agent: Using LLMs to Call Functions

The Tool Agent extends an LLM’s capabilities by allowing it to call external functions. Instead of just answering questions, it can interact with a set of predefined tools—simulating API calls, performing calculations, retrieving weather data, and more.

How It Works:

A registry of tool functions provides capabilities like:
- Web search (search_web)
- Weather lookup (get_weather)
- Date and time retrieval (get_datetime)
The agent follows a conversational loop:
1. Receives a user query.
2. Decides whether a tool is needed.
3. Calls the tool and processes the response.
4. Outputs the final answer.

Example Interaction:

You: What's the weather in Beverly Hills?
Function result: {'zipcode': '90210'}
Function result: {'temperature': '75 F', 'conditions': 'Sunny'}
Agent: The weather in Beverly Hills (zipcode 90210) is 75 F and Sunny.

Here, the LLM autonomously determines that it needs to retrieve a zip code first before getting the weather.

How Did It Go

It’s not easy to get an LLM to only and always respond using structured output, such as JSON. Some models do better than others, and there are lots of ways to use the system prompt to help get the results you want. I found that I still need to check for Markdown code fences in the output, and remove those.

Note: I saw Simon Willison just updated llm to support schemas, to make structured output easier.

Getting the agent to use the tools (Python functions) required not only being specific about the JSON format and the parameters, but also showing examples. The examples seemed to help a lot. I found some discussions about using XML formatted block to describe the set of tools in the system prompt. Something about LLMs being able to handle XML better than JSON. Maybe that is outdated?

I was pretty happy to see the agent use two successive tools (as shown above) to complete a task. I want to play around more to see how that type of chaining can be improved and expanded.

What’s Next?

This has been a fun project and I think there are a few more things I want to try before moving on to the real frameworks:

Expanding the set of tools to include real API integrations.
Use separate agents to implement tools.
Fine-tuning the prompt engineering for better decision-making.
Improving the agent’s ability to recover from errors.

Work-as-Imagined vs Work-as-Done

Mon, 15 Apr 2024 00:00:00 +0000

With engineering focus on reducing incidents and improving operational reliability, I frequently come back to the realization that humans are fallible and we should be learning ways to nudge people toward success rather than failure.

There are whole industries and research machines built around the study of Human Factors, and how to improve safety, reliability, and quality. One topic that struck me as extremely useful to software engineering was the concepts of Work-as-Imagined (WAI) versus Work-as-Done (WAD). Anytime you’ve heard “the system failed because someone executed a process differently than it was documented” could be a WAI vs WAD issue. This comes up a lot in healthcare, manufacturing, and transportation — where accidents can have horrible consequences.

Full disclosure: There are actually many varieties of human work, but WAI and WAD are good enough to make the point. Steven Shorrock covers the subject so well on his blog: Humanistic Systems

Work-as-Imagined

When thinking about a process or set of tasks that make up work, we need to imagine the steps and work others must do to accomplish the tasks. We do this for many good reasons, like scheduling, planning, and forecasting. WAI is usually formed by past experiences of actually doing work. While this is a good starting point, it’s likely the situation, assumptions, and variables are not the same.

To a greater or lesser extent, all of these imaginations – or mental models – will be wrong; our imagination of others’ work is a gross simplification, is incomplete, and is also fundamentally incorrect in various ways, depending partly on the differences in work and context between the imaginer and the imagined. — The Varieties of Human Work

Work-as-Done

Work-as-Done is literally the work people do. It happens in the real world, under a variety of different conditions and variables. It’s hard to document WAD because of the unique situation in which the work was done and the specific adjustments and tradeoffs required to complete the work for a given situation.

In any normal day, a work-as-done day, people:

Adapt and adjust to situations and change their actions accordingly
Deal with unintended consequences and unexpected situations
Interpret policies and procedures and apply them to match the conditions
Detect and correct when something is about to go wrong and intervene to prevent it from happening

Mind the Gap

Monitoring the gap between the WAI and the WAD of a given task has been highlighted as an important practice for organizations aiming to achieve high reliability. The gap between WAI and WAD can result in “human error” conditions. We frequently hear about incidents and accidents that were caused by “human error” in a variety of situations:

Air traffic near misses at airports
Train derailments and accidents
Critical computer systems taken offline
Mistakes made during medical procedures

It’s natural for us to blame the problem on the gap — people didn’t follow the process — and try to improve reliability and reduce errors by focusing on stricter adherence to WAI. Perhaps unsurprisingly, this results in more rules and processes which can certainly slow down overall productivity, and even increase the gap between WAI and WAD.

Safety management must correspond to Work-As-Done and not rely on Work-As-Imagined. — Can We Ever Imagine How Work is Done?

In recent decades, there is more focus on WAD. Examining the reasons why the WAD gap exists and working to align WAI more closely with WAD. Embracing the reality of how the work is done and working to formalize it. Instead of optimizing for the way we imagine work is done, we acknowledge the way work is actually done.

Closing the Gap

In my work, production incidents that occur in software systems are an easy area to find WAI vs WAD happening. Incident management and postmortems have best practices that usually involve blameless reviews of what led to the incident. In many case, the easiest answer to “how can we stop this incident from happening again?” is better documentation and more process.

Modern incident management is focusing more on learning from incidents and less about root-cause analysis. One reason is that incidents rarely happen the exact same way in the future. Focusing on fixing a specific incident yields less value than learning about how the system worked to create the incident in the first place. Learning about how your system work in production is harder, but yields more impact in discovering weak parts of the systems.

This section could be an entire book, or at least several posts, so I’ll leave it to you to read some of the links.

Desire Paths

The whole WAI vs WAD discussion reminds me of desire paths, which visually show the difference between the planned and actual outcomes.

Desire paths typically emerge as convenient shortcuts where more deliberately constructed paths take a longer or more circuitous route, have gaps, or are non-existent

Tying together desire paths with WAI & WAD, some universities and cities have reportedly waited to see which routes people would take regularly before deciding where to pave additional pathways across their campuses and walking paths.

Information Flows in Organizations

Wed, 10 Apr 2024 00:00:00 +0000

I’ve had cause to looked into research and ideas about the ways information flows within organizations. Discussions about transparency, decision making, empowering teams, and trust seem to intersect at organizational communication and information flows.

One of my favorite people to follow in this space is Gene Kim (Phoenix Project, DevOps Handbook, Accelerate, and DORA Reports). He has done a few podcasts that focused on relevant topics and concluded that you can predict whether an organization is a high performer or a low performer, just by looking at the communication paths of an organization, as well as their frequency and intensity. (Episode 16, @54 min)

Some of these ideas might resonate with you. There are generally two forms of information flows:

Slow flows where we need detailed granularity and accuracy of information. Leadership usually needs to be involved in these discussions so communication tends to escalate up and down hierarchies.
Fast flows where frequency and speed tend to be more important. These flows occur in the operational realm, where work is executed, and happen directly between teams using existing interfaces.

In the ideal case, a majority of the communication is happening within and between teams using fast flows. Forcing escalation up and down the hierarchy means getting people involved who probably don’t have a solid grasp of the details. Interactions are slow and likely lead to poor decisions. On the other hand, when teammates talk to each other or where there are sanctioned ways for teams to work with each other with a shared goal, integrated problem solving is very fast.

This doesn’t mean all information flows should be fast. There are two phases where slow flows are critical: Upfront planning and Retrospective assessment. Planning and preparation are the activities where we need leaders to be thoughtful about defining the goals and then defining responsibilities and the structures to support them. Later, slow communications come back when we assess and improve our performance and outcomes.

Thinking, Fast and Slow

I want to be clear that fast and slow information flows are different concepts than the fast and slow modes of thinking explored in Daniel Kahneman’s book Thinking, Fast and Slow. The book explores two systems of thinking that drive the way humans make decisions.

System 1 (Fast Thinking): This system is intuitive, automatic, and operates quickly with little effort or voluntary control. It’s responsible for quick decisions, habits, and reactions.
System 2 (Slow Thinking): This system is deliberate, analytical, and requires effortful mental activity. It’s used for complex computations, learning new information, and solving difficult problems.

Kahneman discusses how these two systems can work together, but sometimes lead to biases and errors in judgment. He talks about how these modes can affect decision-making and offers suggestions into how we can become more aware of these biases to make better decisions.

Obviously another area worth exploring to help understand how organizations can support people to create better outcomes.

Project: Networked LED Pixel Display

Fri, 08 Apr 2022 00:00:00 +0000

I have been wanting to play around with an ESP32-based micro for a while. Once I became comfortable with Adafruit’s microcontrollers and CircuitPython, I thought I’d try out some of their ESP32 offerings. I bought a few Airlift (ESP32) Featherwings to use with the Feather RP2040 boards I was experimenting with.

I’ve also been messing around with some WS2818 / NeoPixel LED 8×8 and 16x16 grids, so I thought it might be interesting to work on a web-based pixel display.

CircuitPython has some very handy libraries for building LED animations on strings or grids of WS / Neopixels. The Airlift ESP32 also has a library to create network clients and access points—surely this wouldn’t be too hard.

Here’s the shapes I used 3D print the parts:

(I really need to improve my enclosure design skills)

Something I’ve picked up from other people making an LED grid displays: Use a lattice grid and diffuser to create an even “pixel” instead of a bright point of light (depending on your tastes). It took a few tries to get the lattice to match nicely with the LED matrix circuit board. While most people use an acrylic diffuser, I just used a piece of card stock paper.

Using the CircuitPython LED animations library, it was relatively easy to try out a variety of different LED matrix animation patterns. Building on the primitives in the library, I created additional functionality that supports text-based and sprite-based animations.

With so many different patterns, sprites, and options to manage—I decided to use a web-based UI to handle the experience.

The UI is served from the device itself. This turned out to be more challenging than you might think. I’ll do a separate post on serving a web-based UI while also running LED animations on the device.

Project: LED Fiber Optic Lamp

Mon, 10 Jan 2022 00:00:00 +0000

Looking back at one of the first real projects I attempted which combined 3D printing and microprocessors. I received an Creality Ender 3 V2 a year ago and after playing around with some test prints, I wanted to try building some more interesting and complex projects. I came across this fiber optic LED lamp project via Instructables. It was just the right amount of 3D printing, microprocessors, and coding I was looking for at the time.

I tend to use components from Adafruit. They have a strong focus on learning. The guides and CircuitPython are great for getting started. So given the great set of instructions, my challenge was basically recreating the 3D models and porting to an Adafruit microprocessor running CircuitPython. The author already provided the 3D models as STLs and in Tinkercad (I also really like the simplicity of Tinkercad), but I wanted to reduce the number of fiber strands and make the lamp slightly smaller.

I figured out the general structure and process of the original model pieces by investigating the Tinkercad project. It didn’t take long to recreate some shapes that I could use to build the lamp structure.

CircuitPython has great support for individually addressable WS2818 / NeoPixel strands, so it was relatively simple to get some code working that would create some simple LED animations. I was using a Trinket M0, which is so tiny. I ran into some space issues where I couldn’t add all the animation support code I wanted onto the board. If I ever revisit this project, I’ll probably switch to a QT Py RP2040 or QT Py ESP32-S2, both of which have plenty of space, way more power, and the ESP32-S2 board would even allow for some network/web configuration UX.

Here are a few photos of the assembly process of the LEDs and optic fiber in the frame.

The fiber optic cable is a “side glow” type used for decorations. It’s designed to create a glow.

Here is the base with the wires and a breadboard for the Trinket M0 (not inserted yet), along with a small button which can be used to change the animation modes.

The CircuitPython code is very simple and is available in a Github repo. I’m pretty happy with the finished project. Some things I’d want to address if I decide to work on a revision:

Using CircuitPython doesn’t leave much room for user code on the Trinket M0, so I’d probably just bump up to one of the newer QT Py models. I’ll be able to add more animation modes too.
Hot gluing the breadboard into the base isn’t sturdy enough. I’ll need to attach the next board with screws/nuts.
Selecting animation modes using the button is not very friendly. If I bump up to a QT Py ESP32-S2, I’ll add a web setup UI.

CircuitPython, LEDs, and Animations

Sun, 05 Dec 2021 00:00:00 +0000

I’ve been playing around with some WS2818 / NeoPixel LED 8x8 grids and CircuitPython. The CircuitPython ecosystem is really rich and Adafruit makes some very handy support libraries. I was using the LED Animation library to create some patterns on the neopixel grid, but wanted to try adding more capabilities. The time-slicing approach made it nice to add other code without blocking the program executing while the animations were happening.

In particular, I wanted to add bitmap sprite animations and text scrolling. There are great libraries and examples in Adafruit’s collection of Learn tutorials, but I didn’t see anything that played well with the time-slicing. I took a crack at building some of my own support.

Animation Extras is a couple of simple code helpers that add bitmap sprite and text scrolling support by building on the LED Animation library.

The LED Animation library has some slick ways of grouping individual pixels to create patterns. I added helpers to create the rectangular animation pattern based on those grouping primitives. Checkout the repo for some example usage.

Tracking Work is Fundamental

Mon, 18 Oct 2021 00:00:00 +0000

“Developers should only need Github Issues and Pull Requests to do their job” — Why should anyone need more than that to track work?

Small companies and startups have small engineering teams. The amount of effort required to understand the ongoing and planned work is low due to sheer lack of ability to take on too much and succeed. Failure weeds out the companies that take on too much, too soon.

Companies succeed and grow, and so do the engineering teams. At some point, multiple engineering teams are created or evolve. Ideally, these teams are self-sufficient and isolated from each other, creating modular and decoupled output. This ideal state rarely lasts and soon cross-team projects start to appear. Teams continue to evolve into product and platform functions, creating more opportunities for cross-team dependencies.

At this point, work can no longer be tracked at the developer-level alone. Success requires collaboration and coordination. Companies without a cohesive work tracking system that can span individual teams start to slow down. Requirements and dependencies become difficult to track and are often not meeting expectations which lead to rework and churn. Deliverables aren’t meeting the guesstimate timelines and drag on.

Making work visible is a core attribute to many different methodologies and processes, even the ad-hoc ones. If you don’t have a bird’s eye view of the engineering work happening at your company, what can you say about your situation? Very little. Try to ascertain the status of a given cross-team project without asking someone. If it takes you longer than 5 minutes, you’re in trouble and the people you would have asked don’t really know either. All of this work required to figure out a project status is wasting people’s time.

Work tracking is something that isn’t hard to introduce and provides value. It doesn’t require adding any extra work for developers, but starts to also provide value to team leads, project managers, and senior leadership.

“Not Jira!” The cry goes out across engineering. It doesn’t need to be Jira, but don’t hate a tool for being successful at what it does. Just because most companies don’t put enough effort into running Jira well doesn’t make work tracking tools bad in general. Pick something else — except fucking spreadsheets!

“You die a hero lightweight tool, or you live long enough to become the villain bloated enterprise-ready system”

Continuously Doing a Thing

Sat, 03 Jul 2021 00:00:00 +0000

Practice makes perfect — Anonymous Parent

A theme that keeps popping up in my world is the idea of how often an action is done being correlated to how well the action is done.

Deploying application and system code
Releasing application distributions
Triaging issues
Testing product behavior
Creating objectives
Running experiments
Executing migrations

A lot has been written about high-performing engineering teams. Accelerate is a great resource for exploring the behaviors of such teams. Frequent deploys is one of the leading indicators and was chosen by the authors as a key metric. With enough practice, deploys become low-risk and low-stress.

Small batches are another trait of successful teams. Performing a deployment more frequently usually means there are fewer changes happening each time. These small batches can actually improve overall quality because fewer changes happen in each cycle.

Rotating a large group of people through activity shifts, like handling issue triage or the application release process, allows the group to share the burden, but there are downsides too. If the activity isn’t part of the group’s primary deliverable, it’s likely not a priority. If there are long stretches of time between any given person taking on the activity there might only be enough time to just do the work, but never think about how to improve the process or tooling. There is no time to become good at the process.

The DevOps Handbook talks a lot about the benefits of shorter feedback loops across many different aspects of engineering organizations. In most situations, shorter feedback loops happen when an activity becomes a more continuous process.

If you have an area that could be improved, maybe you could ask yourself if the process could happen more often.

Being an Effective Engineering Leader

Sat, 25 Jul 2020 00:00:00 +0000

I often wonder if I’m being effective at my job. Might be related to my impostor syndrome, but in engineering management, the signals of effectiveness aren’t always clear. I have some basic, high level criteria I try to think about monthly, or so, to provide some insight.

Providing a clear direction

Lack of clear direction can sometimes be seen when teams are doing medium-term/quarterly planning. If the objectives aren’t aligned with upper management, it’s probably my fault for not creating clear direction and expected outcomes. Try not to be too prescriptive, but make sure the goals are clearly defined.

Try to have a good narrative for each of these levels:

Vision: How the team(s) create impact
Mission: Role the team(s) within the company
Objectives: Think about the next year

Shipping what matters

Do the same problems keep coming up? Make sure we are prioritizing the right work. Make sure we are completing the work. Talk to senior engineers about problems that seem to be holding us back.

Focus on capabilities. Keep improving the operational capabilities of the company. Feature projects are built on capabilities.

Maintain a healthy mix of project sizes. Big projects can stall shipping momentum. Make sure big projects are broken into smaller milestones and iterations. Small projects might feel like low impact, but sometimes are just what people are asking to see.

Helping people grow

Have honest conversations about expectations and performance, and providing actionable feedback.

Make space for other people by getting out of the way. For projects and meetings where I’m getting invited as a point-of-contact, look for other people I can delegate the role to.

Surveys and Feedback Loops

Workplaces typically have company-wide engagement surveys to get feedback on many aspects. Those usually have a management section, and this feedback can be a gift. Interpreting feedback in a positive a way and not a personal attack might be a learned skill, but worth learning.

Thanks to Nick DiStefano for a reminder that manager surveys are also a useful way to get regular feedback on how things are going. Manager surveys can happen more frequently than company-wide engagement surveys and are usually more focused at the team-level.

Stark Raving Finkle

Agentic Engineering: What Agents Can Safely Own

The Right Conditions Aren’t New

What Qualifies Now

Ambiguity Is a State You Can Change

The Number Worth Growing

Agentic Engineering: More Code Isn't Enough

Get Good at the Basics

The Part Agents Can’t Do

What This Sets Up

Agentic Engineering: Where's the Playbook?

The 3X Framework

Reframing 3X for a Methodology

Nobody Agrees What Phase We’re In

Executives Rush to Extract

Practitioners Are Stuck in Explore

Explore Is Not Permission to Be Sloppy

Autonomy Is Earned, Not Granted

Writing a Field Guide to Agentic Engineering Transformation

The Agentic Coding Payoff: Experimentation Velocity

What Experimentation Velocity Looks Like

The Bigger Opportunity: Product and Growth Experimentation

The New Bottleneck

What To Ask Your Teams

Coding Agents Don't Replace Platforms — They Make Them More Important

Agents Already Use Frameworks

Constraints Improve Agent Output

Disposable Code vs Durable Code

Specs, Code, and the Feedback Loop

Using Agents to Extend Platform Reach

The Real Relationship: Agents and Platforms Together

The Takeaway

Agentic Coding: Who Will Review All That Code?

Code Reviews Don’t Catch Many Bugs

What Are Code Reviews Good At?

Agentic Coding Makes the Problem Worse

Using Agents to Review Agent-Written Code

Focus More on Code Planning

Where This Is Heading

The Rise of Hyper Local Software

The “SaaS Is Dead” Take Is Half Right

Why Now

The Opportunity

Glitch Was Almost That Home

Migrating a 20-Year WordPress Blog to Hugo and Cloudflare

Why Move Away from WordPress?

The Export Process: WordPress to Markdown

Fixing Legacy Content

Cross-Post Links

Ancient Blogspot Links

WordPress Shortcodes

Deploying to Cloudflare Workers

Basic Worker Setup

Handling URL Redirects

Deployment

DNS and Domain Migration

Search Functionality

Working with Claude as a Coding Agent

Learnings

Static Sites Scale Down Complexity

Redirects Were Important

AI Agents Change the Learning Curve

Legacy Content Has Character

What’s Next?

Final Thoughts

Further Reading

Agent Coding Update: Teaching Agents Our Primitives

Building a Knowledge Base for Agents

Making Knowledge Discoverable and Actionable

Local Workflows Enable Faster Iteration

From Prototype to Production, Faster

What We’re Learning

Where We’re Headed

Vibe Coded to Shipped Feature

Breakdown and Rebuild

Vibe Code with Our Primitives

Thoughts on AI Assisted Coding

What Kinds of Projects

My Approach

Other People’s Thoughts