Matthew Rocklin

Stepping Back

Matthew Rocklin — Thu, 02 Apr 2026 00:00:00 +0000

Stepping Back

I'm stepping back from my current commitments (Coiled, Dask, Python OSS generally) to make some space for myself. This post details concrete implications of that choice, and then explores some of my thinking.

Coiled

We've transformed Coiled into a pragmatic but less ambitious version of the company.

Coiled is no longer actively marketing itself and we're saying no to long enterprise sales processes. We're focused instead on serving existing customers and simplifying internal operations. The company is now much smaller, consisting of two engineers who keep things humming nicely for customers. Fortunately the Coiled architecture makes this low-effort to maintain long-term.

Financially, while Coiled is unlikely to ever become the next trillion-dollar company, it is now nicely profitable and can continue indefinitely. We had a pile of cash in the bank which has gone towards buying out preferred shareholders. The company is now owned exclusively by common shareholders.

If you're a current customer of Coiled fear not, we're not planning to go anywhere and, unlike before, you're our current focus. If you have concerns I'm happy to chat.

Dask

With Coiled no longer pushing on Dask, and with me stepping back from both, there isn't a strong innovative force behind Dask today.

That's ok. Like lots of well-used open source, Dask today is more focused more on stability than on pushing new features. Dask continues to be maintained, albeit with less vigor. Community members wanting new features for free may be disappointed, but users hoping that Dask continues cranking away should remain satisfied.

Personal Situation

I loved my time building up Dask, especially in the early days when I'd wake up every morning to dozens of emails from excited users building their own things with Dask. Collaborating with those people, incorporating feedback, designing new systems, and together pushing forward transformative change in science and industry has been the highlight of my career.

It feels amazing to work hard and see our work have an increasingly broad impact in the world.

However, work no longer feels that way. At some point turning the crank of productivity turns to grinding, becoming less productive/fun/satisfying. I stopped having fun a while ago, both because I've been at it a while, and because I ran into problems that I didn't know how to solve (Coiled problems were harder than Dask problems, and when issues like marketing became our biggest challenge they started to feel intractable).

Time is finite, and there's a tradeoff between cranking and exploring. I'm in the fortunate financial position to be able to step back and assess what I want from life. I don't know what my next path is yet (start a new company? get a job? get a life?) but I look forward to finding out.

Final Thought

In the meantime, I'm not doing much, and oddly that seems like the most interesting and challenging path for me.

To end, here's a quote I like from T.S. Eliot

I said to my soul, be still and wait without hope,
for hope would be hope for the wrong thing;
wait without love, for love would be love of the wrong thing;
there is yet faith, but the faith and the love are all in the waiting.
Wait without thought, for you are not ready for thought:
So the darkness shall be the light, and the stillness the dancing

Which I interpret as "sometimes you have to do nothing for a while, and that's ok".

For a treatment of this same information, but in a more casual tone, there are some details below:

FAQ

(just kidding, no one asked these questions, but I thought I'd answer them anyway)

How is Coiled Financially?

We make about $1M annually (a little less). We don't seem to be able to grow this substantially, or at least not faster than just waiting around for people to show up. No marketing effort has had much effect. Rather than burn more cycles, it makes more sense to us to just let Coiled do its thing organically. It's also way less effort.

Investors lost confidence that we'd make it big (quite reasonably) and so we came to an agreement where they were able to exit with most of the remaining cash. Not all though; we've a healthy buffer and the service itself is pretty cheap to run thanks to its architecture.

David and Nat work now on Coiled part time, mostly just tracking customer issues and making sure things run smoothly. They don't spend much time on non-paying users or with enterprise procurement teams (sorry, not sorry), and no one spends time evangelizing. I handle the occasional legal issue.

Is Dask Dead?

Usage (as measured by my favorite measure of API docs hits) has been flat for a few years. I don't expect Dask to take over the world, but it doesn't seem to be going anywhere.

Some folks are still doing some mild maintenance (Jacob, James, Tom, Florian, Guido) but it's no one's full time job. It'll be interesting to see how effective community open source is these days at supporting the community.

There are some technical projects I've been wanting to do on Dask for the last few years (query planning for arrays, rewriting in a compiled language, working with high performance hardware). I might get to those at some point?

To be honest though (maybe too honest) I've been feeling conflicted about Dask/OSS work recently. Historically my motivation has come from positive user engagement and community collaboration but over the last few years that relationship hasn't felt as rewarding. Mostly I get the sense that people have started to expect a lot of free work without a sense of collaboration or working together, which doesn't feel good. I suspect that users have been trained by all the for-profit companies throwing free things at them for their attention. While Dask isn't dead, to me it feels like community OSS has died a little.

What will you do?

Yeah, good question. There are a few possibilities. I could ...

Start a new company: I like work, and while making the first company was arduous, there's lots I'd like to experiment in doing this again. I'm especially curious about how lean one can go with AI today. Unfortunately for me though, I don't have a driving project or mission, which puts this plan on hold.
Get an actual job: Maybe someone else has a mission I could believe in, or some lessons I could learn, or piles of money (I'm not above working for piles of money). Fortunately for me though, I'm in a financial situation where I don't need a job, so the bar here is pretty high.
Get a life: I've spent the last two decades immersed in building software, communities, and a company. I've not thought much about the rest of my life.

Right now I'm focused on getting a life. It's hard though. Shifting from "super busy" to "how can I fill up my time today?" is a big lifestyle shift. I've enjoyed speaking to older retirees who have gone through something similar. Right now I'm trying to build a social community, build my legs back up for some running and long-distance walking, and reading in my downtime. Book recommendations are very welcome.

How are you feeling? Is this a big shock?

I'm feeling bored, but generally pretty good, and no, this isn't a big shock.

This has been a gradual transition over many months. We started exploring the small-but-profitable company configuration back in Q4 of 2025, and the company has been operating with current staff since the new year. There's been some work to enact the transition, but overall it's been an easy last few months.

You were doing AI stuff for a bit. What happened to that?

After the decision to transition to small-but-profitable Coiled I spent a couple months playing intensively with AI, making projects like claudechic (along with several others that never saw the light of day beyond personal use). This was super-fun, but ended up not holding my attention, both because of all the noise/hype in the area, and because there wasn't anything that I was really passionate about building. I was just building to build.

I like building things, but I find I need to care deeply about the thing that I'm building in order to stick with it, and I haven't found such a thing.

I'm still very pro-AI, but only in the service of building real things. I'd love to see the world build fewer AI products, and more products with AI.

Saying Goodbye to Winner-take-all Software

Matthew Rocklin — Fri, 13 Feb 2026 00:00:00 +0000

Saying Goodbye to Winner-take-all Software

Today software is a winner-take-all industry dominated by best-of-class products:

GMail for e-mail
Slack or Discord for chat
Stripe for payments
MS Office or Google Docs for documents
...

Generally we see 1-3 products dominate any particular problem space. This makes sense; software is hard to build and free to replicate, so as a product outpaces its competition it gets more resources and invests further (positive feedback) and then distributes that excellent solution all consumers. Everyone gets the same best option.

Other industries differ though; consider clothing/fashion for example. There are hundreds of options for essentially equivalent clothing products. The barrier to entry to make a competitively good pair of pants isn't that high relative to say, building GMail. A small family business can rival the largest incumbents.

This is good. We get lots of different options catering to different tastes.

Software is becoming more idiosyncratic

We can all build our own GMail now

AI makes software easy to build. While GMail took an army of engineers during Google's golden age, I'm pretty sure I could build it today in a couple weeks with Claude. Probably so could you.

In fact, we could each make our own GMail. I wonder how they would differ? How would your taste and mine result in different products and how would those products appeal to different audiences?

I'd like to live in a world where we had as much choice over e-mail as we had over which pair of pants to wear

There's a lot of potential value here. For example my e-mail practices differ greatly from my elderly parents, and yet we both use the same product.

What would e-mail look like if it catered exclusively to developers?
What would e-mail look like if it catered to the elderly?

Given how integral email is to all of our lives, differentiation here seems valuable but absent.

Funding will change

VCs don't fund clothing brands.

Consider Slack as a business. What is Slack's moat? There's an ecosystem of plugins and some network effects, true, but mostly it's that it would be annoying to rebuild Slack. That's no longer true. Heck, I might get excited and rebuild Slack this weekend just to prove a point.

Would I have invested in Slack a decade ago? Absolutely.
Would I invest in Slack today? Absolutely not.

Winner-take-all economies are high-risk/high-reward, which creates a strong partnership between software startups and venture capital. Some of that will surely persist, but I predict that this partnership weakens for the simple reason that the moat for most software products has been the technical difficulty of reimplementation, which no longer exists.

This change isn't bad. Software is no longer expensive, so we don't need millions of dollars of capital to build cool stuff. However, we also shouldn't expect to make trillion-dollar companies. Once a product does well it should expect similar companies to pop up with different tastes.

Today it is probably easier to build a $10M software business, but more difficult to build a $100B software business.

Starting a software company may start to look less like starting Microsoft/Facebook/Google, and may start to look more like starting a clothing brand or design studio. From my perspective that's great.

Open Source will change too

This probably applies to open source software as well.

One reason why Python had such a strong community culture was that building performant software was really hard in Python (it's slow), and so we were forced to organize around a few common libraries (numpy, pandas, cython) that gave us C-level performance for free. This forced organization was critical to the growth and health of the ecosystem. Other ecosystems where performance and abstraction were easy, like C++, didn't have nearly as much cohesion (at least not from my perspective), which stymied their growth.

While I encourage people to collaborate, I don't know that it will be important to share software in the future. I do think that it will be important to share ideas though. Blog posts outlining ideas in sufficient detail that agents can re-implement the idea for each user might be the future of collaboration.

Final thoughts

Making predictions these days is hard; probably everything above will be proven wrong next month. But if what's written above is true, then the main lesson I think is that we need to moderate our expectations:

Old: First movers have an advantage and a strong possibility to dominate the market, generating fame, glory, and wealth
New: There's no rush. Everything is both easier and less important than it used to be. You don't need to reinvent the world today, but instead think more about what you and others would most enjoy.

Claude Chic: Week 2

Matthew Rocklin — Mon, 02 Feb 2026 00:00:00 +0000

Claude Chic: Week 2

Unlike most of my writing, this post was largely written by Claude:

It's been one week since we introduced Claude Chic, and adoption and development since then have been fun.

This week saw 75 commits from 7 contributors. Broadly we saw the following themes:

Smoothing out UX rough edges

When a project goes from single-user to many users there are always various rough edges to smooth out. These included Linux and Windows keybindings, Native Windows support (not just WSL), bugs in /resume and /plan, and much more.
Multi-agent workflows

I've been exploring this space recently and seeing what I like and don't like. I'll describe some things that I like and built into Claude Chic just below.
Performance

Running several long sessions concurrently stressed our use of Textual. Various enhancments were made to smooth things out.
Quality of Life

There are many small things that we added like support for custom themes, small screen / mobile support, and more.

Now, let's dig into some highlights:

Multi-Agent Workflows

The headline this week is new ways to use multiple agents together. The idea: having Claude debate itself produces more thorough analysis.

Plan Swarms

/plan-swarm spawns multiple agents to debate implementation approaches before you commit to one:

/plan-swarm Add caching to the API layer

This spawns several agents, each exploring the problem from a different angle—one might advocate for Redis, another for in-memory caching, a third for a hybrid approach. They present their cases to each other, debate, and often (but not always) come to consensus. You then pick the direction that resonates.

Review Agents

/reviewer

This spawns an agent whose job is to critically examine your recent changes. You mediate as they iterate toward agreement.

Infrastructure

Full plan mode support landed via the SDK's set_permission_mode, with proper tool restrictions during planning and feedback when you reject a plan.

Custom Themes

Claude Chic now supports user-defined themes. The default dark theme works well on dark terminals, but many developers prefer light backgrounds.

/theme chic-light

Or define your own in ~/.claude/chic-theme.json:

{  "name": "my-theme",  "colors": {  "user_border": "#ff6600",  "assistant_border": "#0066ff"  } }

The `--yolo` Flag

For sandboxed environments where you trust the agent completely:

claudechic --yolo

(Or --dangerously-skip-permissions if you prefer verbosity.)

This skips all permission prompts. Use it in CI pipelines, Docker containers, or when you're feeling reckless. The name is intentional—if you're using this flag, you should know what you're doing.

Performance

Multi-agent swarms exposed performance bottlenecks. Running 5-10 agents simultaneously revealed that we were doing far too much UI work for agents you weren't looking at.

Key optimizations this week:

Deferred updates: Hidden ChatViews no longer process UI updates until you switch to them
Collapsed history: Old conversation turns collapse into lightweight placeholders
Lazy loading: Collapsible content only renders when expanded
CSS optimization: Reduced recalculation during agent switching
Spinner fix: The thinking indicator no longer triggers CSS recalc on every frame

The result: swarms of agents run smoothly without the UI becoming a bottleneck.

Quality of Life

Smaller improvements that add up:

Immediate feedback: Creating agents/worktrees shows instant visual response
Compact mode: UI adapts to small terminal heights
Cross-platform shortcuts: Keyboard bindings work consistently across Mac/Linux/Windows
Autocomplete: /worktree and /agent commands now autocomplete existing names
Horizontal scroll: Code blocks scroll horizontally instead of wrapping (#28)

Bug Fixes

The usual parade of fixes, including:

Windows exit warnings (#31)
Session path encoding to match Claude Code (#20)
Crash when pasting long text
Nested tools getting lost after switching agents
Various multi-agent stability improvements

What's Next

This past week was mostly about people other than me trying Claude Chic and reporting issues with their workflow. I expect more of the same.

Try it:

uvx claudechic /welcome

Or update if you already have it:

uv tool upgrade claudechic

GitHub | Docs

Contributors

Code:

Avi Newman - Windows compatibility
Christopher Lamm - plan mode fixes
David Dotson - vim mode
Laura Martin - exit handling
surminus - custom themes
theantichris - session path encoding, model settings

Issues & Feedback:

andrie - WSL timeout
binduwavell - subagents, small screen support
c0ffee0wl - auto-approve mode, collapsible tools
cderv - Windows issues, worktree context, horizontal scroll
chandlervdw - MCP management
dcherian - keyboard shortcuts, agent switching, readline
GaborNyul - multiline input, SSO proxy
jeduden - paste crash
maartenbreddels - branch name handling
quasiben - token doubling
rdubwiley09 - path defaults
sargunv - plan mode

Multi-Agent Workflows

Matthew Rocklin — Thu, 29 Jan 2026 00:00:00 +0000

Multi Agent Swarms: Initial Thoughts

Like many of us, I read articles about systems like Gas Town with a mixture of wonder and fear:

I'm only using a few agents concurrently! Am I falling behind?

This is silly of course. No one is behind because no one knows what they're doing yet.

Still though, I wanted to explore this topic, so I spent the last couple days playing with multi-agent workflows using Claude Chic.

This article is long (sorry!) but here is a helpful TOC. If you don't feel like a long read I recommend skipping to the Lessons Learned.

Obviously Good Simple Multi-Agent Workflows
Platform
First Experiment: Diplomacy
Second Experiment: Session Viewer
Lessons Learned <-- if you read only one section, read this
Experimenting is easy

Obviously Good Simple Multi-Agent Workflows

Many of us use multi-agent systems in simple ways that are clearly a benefit:

Plan then Build: People often have one session make a plan, and then clear out context and start building that plan with a new session.

This has become common enough that I believe it's now the default in Claude Code.
Review: People ask another agent to review the work of an initial agent. This tends to produce better results than having the original agent self-review.

As Wes McKinney writes about RoboRev:

Having a Claude Code session review its own work is so 2025.

There's something powerful about having two different agents think about the same problem and communicate about it, even if those two agents run sequentially, and even if they communicate by having a human copy-paste between them.

Platform

I'm using my personal Claude TUI alternative, Claude Chic. Originally built for nicer style, it also has the feature set needed to support multi-agent swarms.

Start: Agents can start other agents
Send: Agents can send messages to other agents
Branch: Agents can create git worktrees
Merge: There's an automated commit / rebase / resolve-conflicts / merge loop that agents can follow

I built these tools to help me with human-in-the-loop work, but it turns out they're all you need for multi-agent-swarm workflows.

First Experiment: Diplomacy

In the initial demo of Claude Chic, I show Claude setting up a game of Chess against itself.

But then I thought

Q: "what's more intense than Chess?" A: "Diplomacy"

Diplomacy, if you're not familiar, is like "Risk on Steroids without Dice". It's a game of pure discussion and guile where seven players try to dominate 1908 Europe. Claude knows the game (of course) and so I asked it to set up a game among agents.

The gameplay was fascinating to watch. Briefly:

Game starts out tame and typical
Austria thinks it's doing well, but everyone betrays it

Interestingly, the other players joke about Austria behind its back
Russia dominates early on, supported by Turkey
Germany, noticing Russia's rise, convinces everyone else to gang up against Russia

The amount of inter-agent traffic in and out of the Germany agent during this period was intense.
Russia, cowed, makes a deal with Germany to let it survive

Watching the agents play was no less fascinating than watching humans play, and resulted in behaviors I found surprising from AI:

Multiple coordinating agents are capable of more sophisticated behavior than single agents

Silly game, useful lesson. I wanted to try things out in practice.

Second Experiment: Session Viewer

I wanted to look at multi-agent sessions like the one I had just played, and so decided to have Claude(s) build a session viewer for me that I would turn into something with a shared timeline.

I tried this three times with three different prompts, learning each time. Eventually we got to something like this without much human intervention:

It's not great, but it works.

Prompts

They're not beautiful, but here are the prompts I used. You can tell how I learned from one into the next.

First Try

I'm curious about trying an agent swarm technique in development. I was thinking as a project we might try adding a react electron version of this application.

We'd want to have one agent (you) doing a lot of the orchestration, but we'd want other agents for planning, executing each task, reviewing, testing, etc.. You wouldn't do any work directly because you'd be conserving your context to control others.

I want you to think hard about this and how to execute well.

Second Try: I encourage communication

Let's play with multi-agent swarm systems. As a project, I want to build a viewer for sessions. I'm thinking that this is probably a local application, maybe electron that takes a session file and renders it nicely. Eventually I'm going to want to render a few of these side-by-side (to better understand multi-agent systems), but I think that to start we probably just want to get to a milestone where we have an app that runs that can present a session nicely.

For tech we're going to want react/shadcn/tailwind with zustand for state.

For design we're going to want to copy our theme and styling here.

For execution I think we probably want different agents for planning, building, and reviewing. Each building agent should also launch an agent to review its work as it gets close to done (or we should launch it if it forgets). Assuming we use worktrees we're going to want to merge work as it completes rather than all at once. Probably we want a separate integration agent for this. Agents should be encouraged to ask each other questions and collaborate. We shouldn't do much work in this agent. We need to preserve our context.

Think about how best to organize this and then set it going.

Third try. I wanted to limit concurrency.

I'd like to experiment with agent swarms.

As a project, I'd like us to build a session viewer that takes in a session, like the .jsonl file on the Desktop, and renders it similarly to how we do in the app, but with an electron app using react/tailwind/shadcn.

As we work on things I want you to use different agents in different roles.

For our architecture I suspect that we'll want an agent to make an overall plan,
and then we'll want to break apart that plan into a set of tasks with dependencies that agents can work on independently.
We won't want to start everything at once of course. We'll want to pay attention to laying a good foundation first and making sure it's good before moving on.
As we can parallelize we will.
Also, as any agent starts to finish up work they should use the /review skill to create another agent to review their efforts.
When they're done they should use the finalize_worktree mcp command to work on merging their work to this upstream viewer branch, which will be the target of all of our work.

You won't do much work, instead saving your context to help orchestrate the other agents.

Think hard about how to do this well.

If I were to do this again, I think I would sequentialize the entire process. Parallelism didn't buy much.

The overall experience was mediocre

My experience with agentic swarms was so-so. That's not surprising. I'm new at it and bad at it.

As Steve Yegge (famed multi-agent vibe coder) says:

You will turn into Batman, but you can't just put on the suit and say "I'm Batman".

You've got to learn how the tool-belt works

My experience was learning that I'm definitely not Batman yet.

Observations

The head agent delegated really well. It understood that its context was precious and how to create agents to do work. And the plan that was designed was fine.

But parallel execution was generally pretty bad. The agents did fine work, but they mostly ignored each other the first go around. The second time I intentionally prompted the system to chat with peers about the work they were doing, and the third time I encouraged less parallelism. Still though, lack of coordination resulted in lots of strangeness.

On the plus side, the crap it produced was quickly cleaned up by more AI (the solution to AI problems is more AI).

Lessons Learned

As with any philosophy, we should neither accept nor reject it, but should take from it the parts that suit us.

In that spirit, here are my current lessons on multi-agent behavior.

Separate Reviewer 👍

Upon finishing work I used to ask Claude to review its own work:

Let's review our recent changes and see if there is anything we can clean up or simplify

Now I still do that, but I follow up with:

Start another agent and ask it to review our work

They're both quite useful.

Fresh Perspective 👍

Agents get tired. If an agent finishes an especially long session and its context is near 60-70% I'll often ask the reviewing agent to take over and implement the changes.

It's better, of course, to avoid this with well-scoped work, just like a regular human developer.

Mythical Man Month ➡️ Mythical AI Minute

Also like humans, sometimes teams don't perform as well as single developers.

There's a real cost to coordination that needs to be outweighed by the benefits of speed. AI agents are so fast that it's not clear to me when we'd want multiple agents working concurrently on the same code—most tasks finish before coordination overhead would pay off.

My experiment would have been better with many agents, each agent running after the other finished.

Agent Support 👍

Increasingly, the model I have for hard problems is to work with one key agent at a time, and fire off supporting agents to handle tangential work (research, review, performance testing).

It's valuable to keep a key agent on task. (also like humans)

Human Coordination 🧠

I am not (yet) a fan of YOLO Swarms. I want to be involved.

Said differently, all the work I'm doing is sufficiently important to me that I'm happy to pay attention to it. I'm curious about how to best optimize my ability to pay attention to AI work.

I don't think this is universally true. For example during Dask's peak, OSS maintenance was such that I would have welcomed agents handling the first layer of support. Very few of us are in that situation though.

Human-Oriented AI Development

Working on Claude Chic has made me increasingly interested in how I work as a developer, and how to optimize that experience. Rather than ask:

How can I use multi-agent swarms?

I am instead asking:

What is holding me back right now?

How can I best address that?

Today the answer is around ergonomics and context management rather than concurrency (I feel I already have concurrency in abundance). There are great multi-agent workflows I've mentioned above that I use daily. I'm not yet a YOLO Swarm / Gas Town guy.

I haven't really made up my mind though; I should experiment for more than 48 hours.

Experimenting is easy

This is a blatant ad for Claude Chic, but it was pretty trivial to play with (admittedly bad) swarm architectures.

Claude has enough of an understanding of swarms that, if you describe what you want and give it the ability to make agents that talk to each other (like Claude Chic does), then you can just start playing immediately.

Regardless what you think about AI, and regardless how you think about full-auto multi-agent swarms, you should experiment with them. They are, whatever else, quite interesting.

Beginning Again

Matthew Rocklin — Tue, 27 Jan 2026 00:00:00 +0000

We're all Beginners Again

While interviewing early adopters of Claude Chic I heard two common refrains:

"Oh, I'm not very advanced with AI"
"Programming feels fun again!"

These feelings were universal. I thought they'd be good to share out loud.

Beginning again

Everyone was anxious sharing how they worked with AI, afraid of being judged for not being advanced enough. Here's the thing:

No one is very advanced with AI

No one I spoke with was using multi-agent swarms or GasTown or whatever. But everyone was doing something different. The space is wide-open and no one has the answer yet. This should make us fearless.

We're all beginners again, and that's what makes it fun. It's like being twenty years old at university, the world full of possibilities.

Programming feels fun again

Everyone is in good spirits, despite the anxiety.

To me this feels like Python did 15 years ago when there was so much to build and so much new ground to cover. We were in a green field inventing the future against all odds, rather than maintaining legacy software under an increasing weight of success.

Today I see people working on personal side-projects, goofing around, playing.

I'm reminded of these quotes from director Francis Ford Coppola during the making of his recent film/boondoggle, Megalopolis

"You know why I'm doing this? I have money, I already have fame, I already have Oscars."
"What do I get that I want? Tell me."

"Fun! I wanna have Fun."

and

"All the great things in life come from play, never from work, never from toil".

"Toil gives you nothing. Play gives you everything."

We're having fun again, and that's fantastic.

Learning from Python

I had the privilege to experience the growth of the Python data ecosystem from the inside, before it became cool. The way I feel now reminds me of how we felt then.

Not everything is the same though. I think we could improve our culture of communication. What I'd like to see more of:

More casual sharing of how we work (not just the tools or products we've built)
More questioning and critical thinking about popular directions (are multi-agent swarms really the way of the future?)
More chats over beers without an agenda

With COVID and remote work we've gotten too efficient with our communication. We need more un-conferences and hallway tracks—the unstructured spaces where real ideas emerge.

Claude Chic

Matthew Rocklin — Mon, 26 Jan 2026 00:00:00 +0000

Introducing Claude Chic

Claude Chic is an alternative Claude Code UI that provides the following:

Style: visually organizes the message stream for legibility
Git Worktrees: organizes concurrent worktrees
Multiple Agents: runs many sessions from the same window
Quality of Life: loads of other features like diff viewer, shell commands, and more

The official docs are here, but I wanted to share my personal motivation and experience building this.

Or, if you prefer to jump right in, Claude Chic can explain itself:

uvx claudechic /welcome

0. Motivation

I've been deep in AI joy/psychosis the last couple months. The world is super fun right now.

However, as AI speeds up my workflow I find new bottlenecks, and increasingly those bottlenecks are part of my interactions with Claude itself. I wrote about this in AI Zealotry; AI can feel dehumanizing when our primary contribution is administrative, like granting permission.

Beyond permissions, various administrative interactions with Claude became tiresome, and so I worked to improve what I could about the interface for my workflow so that I could feel more human.

Claude Chic is an early result of that endeavor.

1. Style

Let's start with visual design. Claude is functionally great, but stylistically pretty terrible.

Here's a screenshot of Claude Code:

I find this difficult to parse. Everything feels muddled:

No visual distinction between messages
Too wide for comfortable reading (moreso in full-screen)
Lost information - I want to dig into edits before they get summarized away

We don't expect modern design in terminal applications, but maybe we should, especially with Textual.

Textual made it easy to design a skin for Claude code that was, for me, much easier to consume.

Comparison

To me, Claude Code looks muddled.

Claude CodeClaude Chic

To me, Claude Chic organizes the conversation so my eyes can parse it immediately without feeling drowned.

Tool use in particular is interesting. There are more tool use lines in the Chic version (Claude code omits many tool use lines), but they don't get in the way, and each is expandable for full context.

Additionally, Markdown renders properly.

Claude CodeClaude Chic

Key styling choices:

Color: User (orange), Claude (blue), Tools (grey)
Expansion: Tool messages collapse by default, click to expand
Width: Constrained to 100 characters
Markdown: Proper rendering
Streaming: Responses appear as they generate

To be honest, this ended up being a deeper rabbit hole than I anticipated. Totally worth it though. We're still early (definitely beta software), but it feels much better to me than Claude's default interface, or similar tools like Codex, Gemini, and other CLI tools. Everyone should be using Textual for TUIs.

2. Git Worktrees

Git worktrees are underused. They let multiple Claude agents work on the same repository in parallel.

Unfortunately, none of us know how they work. People I've spoken to have all seen an article or Youtube video showing them in action, but in practice they feel foreign. That's ok; AI tooling is great at foreign workflows.

User Experience

Previously, when I had a new idea it went on a TODO list while Claude finished its current task. Ideas would linger on that TODO list for a long time, even simple and easy ones. Coordinating the shared resource of the repository was an annoying bottleneck.

Git worktrees solve this by marrying a git branch with a fresh copy of the repository directory.

~/projects/ ├── myrepo/ # Main worktree (main branch) ├── myrepo-feature-1/ # Some feature you're working on ├── myrepo-feature-2/ # Some feature you're working on ├── ... └── myrepo-feature-n/ # Some feature you're working on

Now you can have n Claude agents working, one in each directory. When I have a new idea, I create a new worktree and have Claude start on it immediately:

/worktree my-idea I want you to explore ...

This creates a new worktree (directory + branch) and a new agent working in that directory in the background. The agent starts working concurrently while I go back to my main task. In practice I have 2-10 of these going at once.

Now I focus on whatever ongoing thread interests me most while Claude pushes the others forward. I no longer wait for Claude; Claude waits for me.

Cleaning up

With concurrent git development comes conflicts. To address this, my git flow looks like the following:

Commit work in a branch
Rebase Rebase on our upstream branch (for linear history)
Resolve any conflicts
Fast-forward merge branch into upstream
Delete worktree

Claude handles rebases and conflicts fine (and asks when it can't), so the workflow above got baked into the /worktree finish command. Concurrent git managemet is, for me, fully managed now.

Now that all of my TODO items can run at once, worktrees shift the decision from "what thread should we start next?" to "which ongoing session most deserves my attention?" This decision, coming later in the workflow, comes with more and richer information.

3. Multiple Agents

Multiple agents run concurrently in one session and can talk to each other. Here they play chess:

This is a cute demo, but in practice I use multiple agents with prompts like the following:

Reviewing: "Start an agent to review our changes critically"
Research: "Start an agent to explore this topic further"
Dogfooding: Recently when adding a profiler, I spawned an agent to use it and suggest improvements back to the agent building it

There are intense multi-agent systems out there like Gas Town and OhMyOpenCode. I haven't needed that level of automation yet, but I find this space interesting.

4. Quality of Life

Claude Chic comes with myriad other niceties like a /diff viewer, vim keybindings (/vim), shell commands (!ls, !nvim README.md), and more.

Diff Review

Review uncommitted changes before asking Claude to commit:

I used to switch back and forth to a terminal to run git diff. Now I get something that I like more than git diff and without context switching.

Shell Access

Run shell commands without leaving the UI:

I typically use this for git log and vim, but it's nice generally.

There are loads more small features in Claude Chic, and more coming daily. It's exciting to work on.

Relationship to other projects

Related (and more mature) work:

Claude Code: More robust. Claude Chic uses the same claude-agent-sdk logic as Claude Code.
OpenCode: Beautiful wrapper supporting any major LLM with its own agent logic.
Toad: LLM CLI in Textual by Will McGugan, Textual's creator.
Conductor: Mac App that provides many of the same features (design, worktrees, multi-agent)

Alpha Status

This software is early, and full of bugs. It crashes regularly when we encounter new situations. Even-so, I encourage adventurous claude users to put it through its paces and raise issues. Development on this project is (as of 2026-01-25) extremely fast.

Final thoughts

For users: AI is fast; we need to avoid becoming Amdahl's bottleneck. Our developer tooling warrants a lot of reimagining today.

For LLM providers: Please check out Textual. It's a great technology. (I have no financial stake in Textual)

# If you use uv uvx claudechic /welcome # If you use pip pip install claudechic claudechic /welcome

Video

If you prefer video, here's one of me explaining this project:

AI Zealotry

Matthew Rocklin — Wed, 31 Dec 2025 00:00:00 +0000

AI Zealotry

I develop with AI today. It's great.

There are many articles you can read on why AI is great (or terrible) or how to use it. This is mine. I focus on the experience of a senior engineer (and why we in particular should use AI), on my experience operating within the OSS Python Data world, and on practical suggestions that I've found myself repeating to colleagues.

This article contains learned lessons of two types:

Big Ideas: Grand(iose) philosophy on why AI is great for experienced programmers
Tips: Taken from my workflow using Claude Code

We'll interleave these two. I'm hopeful that this approach will make this more fun.

Why AI

AI development is more fun. I do more of what I like (think, experiment, write) and less of what I don't like (wrestle with computers).

I feel both that I can move faster and operate in areas that were previously inaccessible to me (like frontend). Experienced developers should all be doing this. We're good enough to avoid AI Slop, and there's so much we can accomplish today.

I like this quote from this blog

I get it, you’re too good to vibe code. You’re a senior developer who has been doing this for 20 years and knows the system like the back of your hand.

[...]

No, you’re not too good to vibe code. In fact, you’re the only person who should be vibe coding.

I think that really good engineers, the kind that think hard before writing, can have a tremendous impact and fun while developing with AI. I wouldn't ever go back.

Why Not AI

That being said, there are some serious costs and reasonable reservations to AI development. Let's start by listing those concerns:

LLMs generate junk
LLMs generate a lot of junk
Writing code ourselves builds understanding
Reviewing code for correctness is the slow part, not writing it
AI workflows can be dehumanizing when you just press "yes, allow" over and over again

These are super-valid concerns. They're also concerns that I suspect came around when we developed compilers and people stopped writing assembly by hand, instead trusting programs like gcc to pump out instruction after instruction of shitty machine code.

We lost a deeper understanding as developers when we stopped writing assembly but we gained a ton too. As in any transition, we need to navigate the situation to capture the advantages while losing only a little, balancing the costs and benefits of a new technology.

This article is how I've been navigating this transition personally.

Big Idea: Minimize Interruptions / Climb Abstraction Hierarchy

Early in using Claude Code (or Cursor) many of my interactions were saying "Yes, it's ok to run that". This was frustrating and dehumanizing. Mostly my job was to enable AI, rather than the other way around.

There are many tricks to resolve this (see below), but more broadly "stop doing simple shit" has been a mantra that I've found myself constantly coming back to. The more I identify and reject simple tasks and add automation to my workflow, the higher an abstraction I'm able to climb to and the more effectively I'm able to work. Our goal in programming is to climb an abstraction ladder and gain more intellectual leverage. This requires thought and consistent attention.

Fortunately AI can help with this. If you complain and say "I'm always doing X" it'll suggest solutions like what I'll talk about below, but more tailored to your situation.

Tip: Hooks

AI developers, like human developers, benefit from structure.

Most people start with an AGENTS.md or CLAUDE.md file. This is a great start, but I find that the AI agent often forgets what's in there. The real solution for me here (at least for Claude Code) is Hooks.

First, let's outline a couple of annoyingly common problems.

Example Problem: Ignoring instructions in CLAUDE.md

Let's say you tell AI that you want to run tests with uv:

when running tests, use uv run pytest tests

While this works sometimes, AI often decides to run

$ pytest tests/ command not found: pytest

While the agents read CLAUDE.md, they don't always follow the instructions. And so you're stuck saying "no, use uv" over and over again. Gah.

Solution: Hooks

Here's a hook that catches pytest commands missing uv run. You could put something like this in ~/.claude/settings.json:

{  "hooks": {  "PreToolUse": [  {  "matcher": "Bash",  "hooks": [  {  "type": "command",  "command": "python ~/.claude/hooks/check-uv-pytest.py"  }  ]  }  ]  } }

#!/usr/bin/env python3 import json import sys data = json.load(sys.stdin) cmd = data.get("tool_input", {}).get("command", "") if "pytest" in cmd and "uv run" not in cmd: print("Use 'uv run pytest' instead of bare 'pytest'", file=sys.stderr) sys.exit(2)

There, we've just automated that annoying task for you forever.

I don't actually do this though (I allow Claude to fail and then it finds the right approach.) Mostly this works because I've gotten good at giving Claude fairly broad-yet-safe permissions, which is coming up next.

Example Problem: Incomplete Permissions

Even worse, Claude often asks for permission to do things that are just slightly different from what you've already granted. You allow uv run pytest *, but Claude keeps finding variants:

timeout 60 uv run pytest ... timeout 40 uv run pytest ... uv run pytest ... | head .venv/bin/pytest ...

Claude Code's permission language sucks. It only supports prefixes, while I wish it could handle regexes, or maybe even just arbitrary Python code.

Solution: Hooks for permissions

I have a complex Python script as a hook which overrides the permission system. It uses regexes, but also arbitrary Python code as logic. This allows me to encode arbitrary combinations of rules. It's great.

On the rare occasion when Claude asks me for permission for something new, I have a running Claude agent that thinks about this file and considers if it should update the permission script.

Solution: Hooks for sounds

My personal favorite hooks though are these:

"Stop": [  {  "hooks": [  {  "type": "command",  "command": "afplay -v 0.40 /System/Library/Sounds/Morse.aiff"  }  ]  } ], "Notification": [  {  "hooks": [  {  "type": "command",  "command": "afplay -v 0.35 /System/Library/Sounds/Ping.aiff"  }  ]  } ]

They play subtle little sounds whenever Claude is either done, or needs input from me. This lets me ignore Claude when it's busy. Previously I found that I was constantly checking back in with Claude to see if it was done, and that action was dehumanizing, so I automated it by asking Claude to play a sound.

Hooks are great. There are more ways to provide structure (Skills, Commands) but I've found that Hooks are the most dependable, a great starting place, and often augment any other structure that I put in place (like Skills).

Big Idea: Build Confidence Without Looking at Code

In a recent large AI-assisted PR a frustrated reviewer said the following:

To me, this [size of PR] implies that either

reviewers should blindly trust Claude, or

reviewers should spend the months worth of effort going through Claude's changes, without the developer bothering to do the same first.

It's a valid problem, even in single-person projects. We're able to generate code far more quickly than we're able to read it. How should we handle review? Everyone needs to figure this out for themselves, but my answer is "find other ways to build confidence".

We already do this today with human-written code. I review some code very closely, and other code less-so. Sometimes I rely on a combination of tests, familiarity of a well-known author, and a quick glance at the code to before saying "sure, seems fine" and pressing the green button. I might also ask "Have you thought of X" and see what they say.

Trusting code without reading all of it isn't new, we're just now in a state where we need to review 10x more code, and so we need to get much better at establishing confidence that something works without paying human attention all the time.

We can augment our ability to write code with AI. We can augment our ability to review code with AI too.

Tip: Self-review

Testing

Mostly I establish confidence on AI-generated work by investing heavily in tests and benchmarks, the same as I would with humans, just moreso. TDD is baked into most of the prompting structure I have with agents.

Remember that this is way cheaper than it used to be. Now rather than write a benchmark I can type

How does this compare in performance to the old version? I'm particularly interested in memory use.

And that's it. If it's bad, the agent will say so (and then diligently work to make it good).

Grilling

Additionally, if I'm nervous about something subtle like "Is it possible this change might unexpectedly affect performance in this other feature?" then I'll ask the AI exactly that question:

Is it possible this change might unexpectedly affect performance in this other feature?

And it'll just go and investigate exactly that question. Unlike human authors, the AI has no ego at stake in its work, and isn't in the least bit lazy. It's our job to ask "Have you thought of X" and its job to go learn if that might be an issue. Don't trust its answer? Ask it to prove it to you.

AI has flaws, but it is diligent, and it lacks ego. If you question it, it'll investigate thoroughly and critique its own work honestly.

Simplifying

Also, my favorite command:

Let's review our work and see if there is anything we can simplify or clean up

Before Opus 4.5 came out this was essential. Now it's merely nice. I've turned this into a /cleanup command and integrated it into most of my Skills as a final phase in development.

Tech debt

From time to time I also ask a fresh agent to do a full review of the project, with an eye to cleaning up technical debt. I tell it to review everything and think hard. It takes a while, but it often comes back with a nice list of work for itself, which it then of course diligently performs.

AI creates technical debt, but it can clean some of it up too. (at least at a certain granularity)

Feedback

In general we want to give our agents good automated feedback. Tests do this, benchmarks do this, prompting them to assess themselves does this, asking them to explain things to us and have us weigh in on high level topics does this.

LLMs are smart enough today that if they're given enough of the right feedback they converge to a good solution as-well-or-better-than a senior human engineer (that's my experience at least).

Our job is to construct a system that gives them the right feedback at the right time, hopefully without our intervention. This is the same job we have when we build human teams; now it's just more impactful to do well.

Cursor vs Terminal Tools

I started AI development with Cursor. It was great having the AI experience inside a VSCode-like editor, where I could see everything that was going on. When I saw terminal-based tools like Claude Code I thought "whoa, that doesn't seem sensible, I need to see what's going on".

Today I code with Claude Code, git diff, and occasionally vim. I don't feel a need to OK every change in the diff. I've got more important things to do. I suspect that you do too.

Big Idea: Drop Python. Use Rust and TypeScript.

I deeply respect the philosophical position of Python, which I'll state as follows:

Prioritize human performance over compute performance.

By optimizing for ease and iteration speed we're able to search solution space more broadly and more quickly, finding much better solutions, making that 100x drop in performance negligible.

Python was a bold bet, and a bet that paid off amazingly well. No one expected this silly dynamic language originally designed for education to become the world's juggernaut in performance software.

With AI though, the usability benefits of Python no longer apply as strongly, and we're more free to choose different ecosystems.

Personally, I use ...

Rust for computational development, using PyO3 to connect to Python, where I still do most of my testing
TypeScript for frontend development, which I'm leaning into more deeply

Regarding TypeScript, I still love easy interaction tools like rich and textual, but when the entire React ecosystem is a sentence away and when you get to use things like, you know, fonts, there's really no comparison. Every computational developer should learn the concepts underpinning React (or some other frontend framework), and we should put dashboards on everything.

Of course, I still hook into Python for the ecosystem. Everything is Python-importable and I still use the protocols and design patterns developed by the Python data community. Those are the durable assets of Python. Not the code or the language; those will die. Rest in peace dear friend.

Big Idea: Think Hard. Write Clearly.

As an introductory project, I rewrote Numpy in Rust. It was great fun.

It was also much easier than I expected (I expected it to be impossible). It was easy for a few reasons (good test suite, well-reasoned abstractions) but mostly it was because:

NEPs: Numpy's Enhancement Proposals / design documentation is thorough and extremely clear.

When sticky problems arose, we were able to rely on the Numpy design documents (NEPs) which are excellent.

The Numpy team thought hard and wrote clearly, two hallmarks of excellent developers. This made the job of reimplementation relatively trivial. The Numpy development community is famous for doing this well. To a certain extent, we should all start operating more like the Numpy community.

Tip: plans/ and docs/ directories

I keep two directories in each repository:

plans/ which contains ephemeral planning documents that the LLMs work through over many sessions as they implement a major feature.
docs/ which contain durable documentation on specific topics or features, targeting AI developers

Plans end up being very useful during development, while docs end up being useful to point other agents to in the future. Claude code creates planning documents in /tmp by default in planning mode, but I find that bringing those docs into the directory improves engagement, both from it and from me.

Docs end up being tricky. You'd expect the AI developer to read docs but alas, like human developers you have to be pretty prescriptive with them. Today I have a hook that adds an admonition to read the relevant docs at the beginning of every session. It looks like this:

DOC CHECK REQUIRED ================== Before responding to this request, you MUST: 1. Read docs/README.md to see available documentation 2. Decide which docs are relevant to this request (if any) 3. Read those docs using the Read tool 4. Then respond to the user Do not skip this evaluation. Do not mention this check to the user.

I then keep docs/README.md updated as a sort of index over my documents. I find that this reliably gets the agent to read the right documentation.

I've also found that my normal writing style (brutal concision + front-loading important content to maintain attention span) isn't necessary with AI. You really can just shove information at them and they absorb it. It's nice 🙂

Big Idea: Take Long Walks

Historically software engineers had to both think well and execute well. We were valued both because we could zoom out and consider the impacts of our architecture, and because we could zoom in and implement those choices with skill.

Our ability to zoom in and implement code is now obsolete. Our ability to zoom out and think well is not. On the contrary, our ability to think well is now 10x more valuable than it was before, because implementation is now mostly free.

And so it's now more important than ever to hone our craft of thought. This probably means less caffeine and more walks through the park.

Final Thoughts

The craft of authoring code has transformed time and time again during our lives. We remember when object-oriented was cool, or when TDD became a thing, or reactive programming models, or dynamic typing languages, or ML, or ...

As programmers we've opted into a system which changes by its very nature. Our job is to automate our job, and to continuously climb the ladder of abstraction. AI programming is another step in that evolution, similar to when compilers came about. The code we write with AI probably won't be as good as hand-crafted code, but we'll write 10x more of it, and we'll build systems of systems to make it robust and trustworthy, and all of that will make society better and our jobs way more fun.

I'm looking forward to having way more fun.

Appendix: Permissions file

After writing this a couple friends asked me for a copy of my regex/Python code that replaces Claude's permission system. I'll include it below, but really, you don't need it. Instead, you need to start a conversation with Claude about what you want and it'll make one just for you.

Code is free these days. Extending the "AI is like Compilers" analogy, asking for someone else's script is kind of like asking for someone else's compiled binary. There's no need; just make it yourself. It's trivial.

Here was my original prompt to Claude Code:

I recently wrote this reddit post

https://www.reddit.com/r/ClaudeAI/comments/1puqrvc/claude_code_annoyingly_asking_for_permissions/

I'm wondering if you have any suggestions on how to resolve this? Adding stuff to CLAUDE.md or permissions to settings.json doesn't seem to be working well enough.

That, along with subsequent conversation as I've been working, resulted in this Python script

But really, you're better off working with Claude to make one just for you. Code is free now.

Twitter, LinkedIn, and Failing in Public

Matthew Rocklin — Wed, 23 Jul 2025 00:00:00 +0000

Twitter, LinkedIn, and Failing in Public

Twitter is dead. Long live LinkedIn. Well, kinda.

During Twitter's golden age, I loved seeing people work in public. We shared half-formed ideas, debugged projects in real-time, and celebrated small wins alongside inevitable setbacks. Twitter's casual tone made it easier to be vulnerable about the messy nature of work. Working in this way felt to me like playing with friends.

Today on LinkedIn, I mostly see polished announcements: product launches, job promotions, completed milestones. This isn't inherently bad (I've done plenty of self-promotion myself) but LinkedIn feels emotionally hollow because we share only the final act.

We know from every compelling story that we connect to heroes during their struggles, not just their victories. The uncertainty, the failed experiments, and the moments of doubt followed by picking yourself up; that's what makes us emotionally invested in someone's eventual success. LinkedIn is like watching only the last 20 minutes of the movie.

I suspect that this shift reflects broader cultural changes. The stakes today seem higher, and mistakes more costly, both economically and reputationally, so people naturally hedge and default to sharing only their wins. The safer it feels to only show success, the more isolated everyone becomes in their struggles. Our sense of community and authentic connection depends on our shared vulnerability. So too does our sense of play.

One solution is something we can all do more of (myself included), fail heroically in public. We can share more messy ideas, experiments, and failures as they happen. This requires courage, but I suspect it'll also transform a scary activity into one more resembling play, scraped knees and all.

I Walked across Spain

Matthew Rocklin — Mon, 14 Jul 2025 00:00:00 +0000

I Walked Across Spain

Last month I walked across Spain, east to west, following the historic Camino de Santiago, a historic pilgrimage route travelled by thousands of people a year. It was an experience.

Since returning lots of people have been asking me a seemingly simple question:

So how was the Camino?

And it's been difficult for me to give a pithy answer that fits neatly into a conversation. This post is my attempt to solve that problem. Here's a written version of some reflections from my walk.

Overlapping Experiences

I think that the Camino is best understood as several overlapping experiences. It is a ...

Physical experience
Social experience
Cultural experience
Religious experience
Cloistered experience
Humanist experience

Overlapping these experiences at the same time weaves a meaningfully different way of life than I've found elsewhere, neither in normal quotidian life, nor in vacations or other physical adventures.

I'll briefly share my thoughts and personal experiences on each layer:

Physical

I walked the typical route of the Camino Frances, starting from the French border in Saint Jean Pied de Port. This crosses the Pyrenees on the first day (imagine being a downed WWII RAF pilot escaping occupied France into neutral Spain in some Hemmingway-era novel) and then continues 500 miles / 800 kilometers to Northwestern Spain, and the town of Santiago de Compostella. That's roughly 20 marathons, back to back.

This is both easier than it sounds, and harder than it sounds.

Walking a long distance is actually pretty easy when it's all you have to do for the day. You pick up a slow and natural pace and just keep at it. You're carrying gear, but not much (the walk is supported with hostels and cafes along the way). The median walker is probably a retiree in their sixties who does about 20km a day. I'm a pretty fit 40-year-old and did anywhere from 20km-50km a day. I finished in 27 days.

However, walking this distance is also a harder than it sounds because of accumulated stress. In normal life you're active one day and play hard, and then the next you rest and recuperate. On the Camino you just keep going day after day. Your body accumulates stress and injuries but you adapt and walk through that stress, eventually reaching an equilibrium where you heal about as quickly as you injure yourself. People manage blisters, joint pain, swelling, plantar fascitis, diabetes, and more.

Personally, I fell and bruised my knee pretty badly the second day and then promptly did a lot of descent, which caused a highly inflamed knee that I walked on for about a week before it healed. Then after a couple weeks, my metatarsals / forefeet became inflamed which took some getting used to and slowed me down. I walked in minimalist footwear (Vibram-five-fingers) which is totally normal for me at home (I've worn them for close to 20 years and run barefoot regularly) but I had never experienced this much repetitive stress for this long, which made it all the more interesting to accomplish.

This level of endurance was a new kind of challenge for me, and discovering this level of resilience was a new kind of satisfaction.

Physicality aside, socially the Camino is simultaneously a solitary and communal experience.

First day

I arrived in France alone, and travelled to St Jean alone, and set off from my hotel in the pre-dawn morning alone (thanks for the early start Jetlag!). But you cross paths with other walkers pretty quickly and stop and chat for a while. That first day included people like ...

An American teacher just starting her Summer break
A Brazilian guy my age who spoke no English and about as bad of Spanish as mine, which made for jovial if halting conversation
A Swiss gentleman who had walked from Switzerland
A Dutch woman taking a break from her cyber-security consultancy
An American couple, one of whom was deeply struggling with diabetes and blood sugar regulation
An American family from Texas (parents, three daughters aged 16-19) that I met throughout the day (the daughters were faster than the parents)
An Australian priest who had just finished graduate school
An American religious studies professor, recently retired
A French retiree who had walked from a town much more deeply in France, and was finishing at the Spanish border (my first day on the Camino was his last)

You have easy chats and then walk on. At the end of the day you all arrive at a big hostel where there are shared meals. You eat together, drink together, and chat about the walk. It was lovely.

Then you sleep (poorly), wake up, and do it all again.

All the other days

The Camino is a river of people. Some people are fast, some are slow. Some start early, some late. Some are chatty, some mostly silent. Everyone walks a little differently.

Emotionally you bond quickly with people (easy when you all struggle towards the same goal, suffering the same hardships) and you form groups. Over time, for various reasons those groups grow and shift and break. People get injured and take rest days; people feel energy and burst forward; people start or stop their journey at different cities to get back to normal life.

There's a tension between staying with people you've bonded with and walking in a way that suits you. It's like life generally, just compressed. There's a strong pull to stay with a familiar group, but ultimately everyone walks their own Camino.

Cultural

Spain is a wonderful host country for this kind of activity. The vibe is chill; the people are warm; the food is cheap. It's super accessible. That being said, the Camino is not an especially Spanish experience.

There are a few Spaniards walking the Camino, but not many, and most interactions with townspeople are practical in nature. And while the Camino stops is a few delightful cities (Burgos, Leon, Santiago, ...) the majority of the time you're out in countryside or in very small villages of 100 people or less.

To make this concrete, people often ask me "how was the food!?" to which my answer is "pretty terrible actually". This isn't a reflection on Spanish cuisine, but rather on the fact that we're walking through tiny towns, and that Spanish restaurants open at 8pm and pilgrims are usually bedded down by 9pm. As a result most of my calories came in the form of bananas/nuts/sausages picked up at small grocery stores, Spanish tortilla (a potato based quiche dish) and non-alcoholic beer found in roadside bar/cafes (NA beer is delightfully ubiquitous in Spain, and the perfect mid-walk beverage). Dinner was typically prepared in a hostel by hostel staff who, while delightful, were not especially accomplished chefs.

Culturally the Camino is more of a globalist/international experience than a Spanish one. Linguistically there were a few groupings:

English: typically North American, Australian, and New Zealanders, as well as Germans and Dutch. Maybe 40%?
French: on the Camino Frances there are a number of French speakers, maybe 20% at the beginning dropping to 10% by the end?
Spanish: consisting of some Spaniards, but more Latin Americans, as well as Portuguese and Italian speakers, all of whom get by in an interesting (and incredibly easy to understand) Spanish patois
Korean: a surprising though delightful group, South Koreans made up maybe 5-10% of the pilgrims, and are by far the most represented East Asian country due to their stronger Christian heritage there.
Other: some Japanese and Chinese (more-often Taiwanese).

While quite different and sometimes unable to communicate, these groups got along amazingly well. There's a culture of understanding and charity on the Camino that's universal. I received help and support (like blister care) from people I had just met and couldn't speak to, and happily gave what I had to others. My claim to fame in terms of generosity was organizing an Amazon shipment of Lacrosse balls to give out along the trail (the one I brought from home was in incredibly high demand in hostels for foot massage).

Despite most pilgrims not being religious, the Camino's culture was one of the better expressions of some Christian values I've experienced.

Religious

The Camino de Santiago was and is an important religious pilgrimage. We're walking to the site of the remains of Saint-James (or in Latin Sant-Iago), one of the twelve apostles. Christians have walked from their homes to the site of his remains for a thousand years (and then amazingly, also walked back).

I am not religious, and I'd say that the minority of pilgrims are today. But walking with the more devout pilgrims was a treat. Empathy is so natural on the Camino that you get to feel a little bit of their experience second-hand.

For those who were interested there was nightly mass to attend, historically relevant churches/cathedrals/hermitages/etc. to visit, monasteries to stay in, and so on.

I didn't engage this layer much, but I appreciated its presence. Being among the religious pilgrims helped to make the walk more than just a big hike.

Humanist

The people walking the Camino come at interesting times of their lives. I met people who ...

just retired
just quit their job
recently separated from a partner
lost a loved one
were trying to lose weight
just given up a drug habit
just sent a loved one to rehab, and for the first time had time to themselves

Generally the pilgrims are at interesting and reflective phases of life, and that brings some interesting energy to the crowd. Everyone is thoughtful and generally open to change in life. This cultivates both a level of empathy and emotional honesty that I found pervaded the experience.

Cloistered

You're so focused as a group on walking that you naturally detach from the broader world outside the Camino. This was surprisingly meaningful.

Like most people, I check my e-mail / the news / my phone actively throughout the day. This behavior goes well beyond "staying on top of work"; it has become a pretty strong compulsion for me (and for many of us I think). I don't think it serves me well, but I've historically struggled to change the behavior in a lasting way.

Fortunately, on the Camino that desire to check in with the world disappeared entirely. The novelty and buzz of the world felt boring and unwelcome. I didn't check my e-mail or the news for days at a time, not through intention and discipline, but through lack of interest.

Although when I finally did check the news I found that it had been a pretty intense few weeks for others. This period included both the LA immigration riots and the attacks on Iran.

This made me think more about the right level of active engagement one should do. I think that one should care about the world, but perhaps not at the cadence that is typical today. There's a frequency and mania of staying on top of things that I don't think meaningfully helps; and does considerable harm.

Lessons

The Camino was hard and like many hard things, it changes one's perspective.

Upon returning people ask me:

"What did you learn?"

"What did you take away?"

I'm not sure I learned a specific lesson, but I do think that stepping outside of my life and walking the Camino instilled a few behaviors or qualities that I appreciate:

Introspective: I'm more open to questioning larger parts of my life
- "do I like living in my city?"
- "is this particular relationship serving me?"
- "would I prefer to stop attending this regular event?"
rather than going with the flow that past-me set up. I'm more active in assessing what I do and do not appreciate, and more open to making more decisive changes.
Calm: I think I'm more thoughtful and less frantic with how I engage with the wider world. Not meaningfully less engaged I don't think; just a bit less manic about it.
Satisfaction: I seem a little more satisfied with life moment-by-moment. The persistent nagging voice in my head saying things aren't good enough seems a lot quieter today. I take more naps and sit outside watching people more often.

It's been a few weeks since I stopped walking and these traits so far have stuck around; which I find encouraging.

Call to Action

My age group (30s-50s) was generally absent on the Camino.

That makes sense; we're in our most productive years. We're raising kids and at the peak of our career and so reasonably find it difficult to take a month away.

However I encourage people to try. This might be an especially important time to step outside our lives and reflect on what we want from it.

Thoughtful Professional Writing

Matthew Rocklin — Wed, 07 May 2025 00:00:00 +0000

Thoughtful Professional Writing

The hard part about writing is thinking.

When we speak with each other, our language just flows as our thoughts come. It's easy and doesn't require much forethought. Lips move at the speed of thought. While easy, this also means that we're not forced to organize our thoughts ahead of time, they just flow. This might be hard on our listener who now has to organize our thoughts as they listen to us talk.

Writing, especially writing in a constrained space, forces us to think more about the concepts we want to communicate, how best to organize them, and how to distill them into the simplest possible form. This is hard, but it makes it easier on our reader, who is now presented with a more organized sequence of thoughts.

Writing at Work

At work we have a culture of written communication. Almost all group meetings are preceded with a Google Doc, and many 1-1 meetings have notes sent ahead of time. Personally I find this very useful:

It gives me space to think about the topic ahead of time

I like to think before making decisions, and I think much better outside of a time-pressured meeting.
Google Doc comments enable concurrent questioning and discussion

It's much easier to collect and address a high volume of comments and questions if they can happen at the same time. We're all able to read and comment on a doc simultaneously, while we can't all share the speaking floor simultaneously. This handles the easier / simpler questions.
It makes it easy to broadcast context and decisions to others

Often people not in the meeting are curious about the decision made and the context and arguments that led to that decision. It's much easier to share that context to others if most of it is written down.

But most of all, the document shared ahead of time organizes the discussion. A good document collects relevant context that everyone should know. It synthesizes the core tensions involved. It lays out a few of the best options we could take. It also includes some opinions and recommendations, and an analysis on why those recommendations might be good. This organization quickly puts everyone in a good frame of mind, so that they can meaningfully contribute to effective decision-making.

Organized Thoughts

Of course it's not writing that's important here. It's how effectively we're able to serve up context and organize thoughts for our colleagues.

There are other appraoches to this that can be just as effective, like a well-crafted slide deck. There are also written appraoches that can be ineffective, like a very long and unorganized document that people are expected to slog thorugh.

For me, the value is in the following:

Concisely define the problem to be solved
Remind the readers of relevant context
Lay out some possible options
All in about a page
Ahead of time so people have time to reflect

This is hard. It takes time and mental effort. It's also something that I really appreciate when my colleagues take on that burden.

Sports Analogy

In Ultimate Frisbee there are several possible throws to move the disk up the field. They differ in how easy they are to get past the defender guarding the thrower, and how easy they are to catch:

Forehand/backhand throws are typical. They're difficult to get past the defender guarding the thrower, but if the thrower manages it then they're pretty easy to catch.
Hammer throws are atypical. They're an overhand throw sent above the defender (and so easy to get past) but they fly upside down and are more difficult to catch.

Good throwers typically restrict themselves to forehands and backhands. They take on the burden of beating the defense and so make their throws easy for their teammates to catch. More novice throwers often get flummoxed by their defender, and as the shot-clock winds down often resort to throwing hammers. This shifts the burden of the pass completion from the thrower to the receiever.

We expect more from the thrower because they have ten seconds (Ultimate's shot-clock) to find a good passing opportunity, while receivers have only one chance to make a catch.

Final Thoughts

In any communication or decision there's a burden to organize thoughts. This burden will be either on the speaker's side or on the listener's side. Often, it's the speaker who has more context of the situation. In this case it's considerate and often more efficient for the speaker to take on the burden of organizing their thoughts, rather than depend on the listeners.

I find that the best way to do this is through a concisely written document. I find that this is hard and takes time, but most of that time is time spent thinking which should be done anyway.

Transactionalism vs Respect

Matthew Rocklin — Mon, 05 May 2025 00:00:00 +0000

Transactionalism vs Respect

Consider these two statements:

Can I ask you for your help?
If you help me I'll pay you back.

Which would you rather hear?

I like the first. It makes me feel like a person who helps people, rather than a person who's trading favors.

Context

This has come up recently in sales and marketing conversations. Often, we want something from people:

We want them to visit our booth at a conference
We want them to participate in a survey or customer interview
We want them to tell their friends about us

A typical marketing approach here is to incentivize the behavior we want with some object of value in return. This might be a bit of swag, or an entry to win a prize in a raffle, or an Amazon gift card.

It makes sense. People like getting stuff.

Reaction

But these incentives never feel right to me. They feel transactional.

As a developer this transactional relationship isn't what I want with users, like I'm trying to buy respect. As a user it make me feel cheap, like my time is being bought.

Instead, I'm often quite happy to help out with a project under the right conditions:

Is the project interesting?
Is it something I care about?
Is it important to me?
Am I important to it?

Yes? Then of course I'd be delighted to help. Helping to build something great is all the incentive I need.

When to Incentivize

You should incentivize if you're building something that people don't care much about. Conversely you should instead foster a spirit of collaboration when you're building something great that the people you need care about a great deal.

If you're building something great that people care about, then don't act like you need to buy their time. What's more, trying to buy their time cheapens the relationship.

More than swag, people like respect.

And Give when Possible

It's still great to give stuff away though!

For example, at conferences we've been giving out this awesome Dask shirt:

{figure} /images/dask-punk-shirts.jpeg :width: "50%" :alt: Dask Punk T-Shirts

I love this shirt. Good design. Excellent pun, at least for those who were coming of age during Daft Punk. I want everyone to have one! So we made thousands and gave them away at conferences. We didn't ask people to give us their e-mail or have a conversation with us to get one or anything (indeed, some people were confused by this). We just did it because we thought it was an awesome shirt and relatively cheap to do.

People loved these shirts. I'd reliably see people the second day walking around in them proudly. Seeing that made me so happy.

This is the world I want to live in. A world in which we avoid extraction and instead collaboratively build great things.

Hammers and Nailguns

Matthew Rocklin — Mon, 06 Jan 2025 00:00:00 +0000

Hammers and Nailguns

People often ask me, "What is your tool for?"

They want an answer like "AI" or "ETL" or "Training Models".
They want a clean and specific application for easy comparison.

I, of course, do not give them that satisfaction.
Instead I ask "What is Python for?" or "What is the cloud for?"

Them: Well, all sorts of different things.
Me: Exactly!

Unfortunately, this answer sucks.

Simple Tools, Humble Opinions

The analogy I use here is of hammers and nailguns.

Nominally these tools do the same thing (nail things)
In reality they do very different things:

The nail gun puts nails into wood. The nailgun is 10x better than a hammer at doing this. If we have to nail 1000 nails then we should definitely get a nail gun.
The hammer whacks things. Those things can be nails, but they can also be wrenches, or rusty bolts, or drywall, or heck, even a zombie’s head.

I own a hammer, not a nailgun. I use it for all sorts of things (most recently to tap in drywall anchors). I can’t recall the last time I actually nailed something with it. Nailguns are better at nailing things, but it’s rare that that’s exactly what I want. Nailguns make enough assumptions about what I want to do that they’re rarely the right tool for me. The hammer's design is more humble; it doesn't assume much about how I want to use it.

Home improvement, like Python, is a place where all sorts of strange problems arise, and it’s great to have simple and flexible tools that extend our natural abilities (like whacking things) in any situation.

Nailguns are easier to market

Writing copy for nailguns is easy

Nailguns: "Sink nails 10x faster, save time and energy!"

Hammers are harder

Hammers: "Whack things! So useful!"

Hammers have more applications and are far more commonly used. However the lack of a single "killer application" (just thousands of mundane ones) makes them awkward to pitch successfully.

I Like Hammers

As a craftsman (in both wood and code) I like hammers. Hammers extend our natural abilities just a little (naturally our bodies know how to hit things, just not as hard). Hammers fit in our hands, are simple, and are intuitive to use in any situation.

We especially need hammers in unpredictable situations, which is exactly where Python data devs live. Python devs are the tip of an organizations problem-solving spear. They always get asked to tackle the wacky new dataset or computational method. Their problems are often novel and untested. Of course for any problem we would prefer a purpose built tool, but the market rarely responds to our needs that quickly, and by the time it does it's usually someone else's problem.

Simple tools with humble opinions like hammers are what I reach for when I face a new problem. They're difficult to justify in words, but they're always easy to use and effective at getting the job done.

Python Support is Socially Complex

Matthew Rocklin — Mon, 09 Dec 2024 00:00:00 +0000

Python Support is Socially Complex

Supporting a Python product is difficult because it’s hard to draw a line between your product and the rest of the ecosystem. This creates confusion, especially when it comes to triage and attributing failure to one product or another.

When a complex product like a Python-based data workflow fails, most of the work in fixing it is in diagnosing the problem. When the product is self-contained and supported by one company, like a car for example, it’s clear whose responsibility it is to diagnose and fix the problem.

However, when a product is complex and ownership is jointly distributed among many different responsible entities, diagnostics become fraught. No individual entity has clear responsibility to solve any particular problem, and no entity has responsibility to triage the problem as a whole.

Easy Case

When you buy a self-contained product, like a car, it’s obvious when a part becomes defective whose fault it is. The entire car was assembled by the same manufacturer which takes full responsibility over all parts, so it’s clearly something to do with their work.

Problem: Have an issue with your car under warranty?
Solution: Take it to the dealer.

Difficult case

But in an ecosystem like Python, you use hundreds of tools simultaneously, each of which has complex interactions with each other, and each of which is managed by a different team. It becomes difficult to attribute a particular failure to any particular tool. The burden of diagnosis is on you.

This is like if you didn’t buy your car from a manufacturer like Honda or Ford, but instead assembled it piece by piece, having purchased parts from individual factories. No individual factory is going to do the work to figure out why your car died until you prove that it was their part that was at fault.

Problem: Have an issue with the complex jumble of software systems you’ve glued together?
Solution: 😩

As an example, here is a recent support request and response from our support team:

Example: Support Request

I'm trying to get started with your product, however after pip installing I get the following error:

raise ClientConnectorCertificateError(req.connection_key, exc) from exc aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host cloud.coiled.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')]

Could you let me know if you have a suggested solution to try? Thanks!

Example: Support Response

… after some other back-and-forth with the user diagnosing their situation …

It looks to me like HTTPS is broken for your local install of Python - I don't think it's related to our product. For example, I expect this would fail in the same way (you can check):

from urllib import request request.urlopen("https://google.com").read()

We generally suggest that folks not use the "built-in" MacOS install of Python for this kind of work. Installing Python with miniconda is my favorite option, but I'd probably need to know more about how folks on your team usually work with Python to make a good suggestion.

Example: Analysis

The user’s failure has absolutely nothing to do with our product, and yet we spent non-trivial time tracking down what was going on. From the perspective of the people providing support, this felt like an abuse of the support relationship.

However, at the same time, the user is unlikely to be able to diagnose an issue like this on their own (SSL errors are not often taught in data-scientist school) and on the surface it totally appears to be related to our product. It’s both totally reasonable for them to want help from us, and totally reasonable for us to refuse to help. This causes tension on both sides. This raises the question …

Question: Whose responsibility is it to triage complex issues?
Answer: Unfortunately, no one. For users, this sucks.

What can you do?

Really big companies buy “Data Science” from large integrators and those large integrators take on responsibility. However, in my experience, the support they offer is simplistic and rarely helpful.

More often, teams depend on the good will of support teams of the individual products they buy, along with lots of effort from within their own team. This isn’t great, but can be ok if you’re both hardworking and nice about asking for help.

Main lesson: Be hardworking and ask nicely.

Support teams are made of people, and people everywhere enjoy helping other people (we’re genetically programmed to do this) especially people who have put in the work and are nice.

See Craft Minimal Bug Reports, which gives tips on how to nicely ask for help in an OSS context.

Our approach

We try to be nice and help when we have time (as happened in the SSL example above) but we assume no formal responsibility to help with problems that aren’t clearly the sole fault of our software.

Sometimes people get upset when we don’t spend the time to diagnose ambiguous issues and we all have to live with that outcome. In those cases we typically back off from that client, or point them to this post to better clarify our understanding of support expectations. Sometimes that leads to the client backing away (they need more consultative help) and sometimes it leads to a deeper paid support relationship.

Programming is cool because you can glue together lots of different systems and get a totally new and wonderful system out. That’s also what makes programming really hard though. Gluing together systems that were never designed to work together results in all sorts of unexpected interactions, and Interactions are the hard part of systems development.

We’re here to help navigate, but ultimately you’re responsible for the system you construct.

Cloud Computing is Broken

Matthew Rocklin — Tue, 03 Dec 2024 00:00:00 +0000

Cloud Computing is Broken

Famous Quote

This AWS API is so intuitive it's awesome. My whole team uses it.

-– no one ever

Introduction

Recently I was chatting with an investor about the market and they asked:

What do you see coming for the world of Data/Cloud/Compute Infrastructure?
Is there some new dataframe library?
LLM applications?
What hunger are you seeing on the ground?

My answer, after thinking about it for a while, was this:

Boring shit.
The people I talk to struggle most with the basics of cloud computing.**

This article explores this answer in more depth, drawing on undelivered promises, and the launch of the iPod.

Promise of Cloud Computing, Undelivered

The promise of cloud computing for data developers goes like this:

Infinite scalability
Any place any time
Turned on in about a minute
Pay for only what you use
No need to maintain hardware

This promise is valuable to companies today. Developers want this. Companies need this. Especially those people for whom SQL doesn't suffice and who have to roll their own computations.

Yet most developers we see either don’t use the cloud, or don’t use it well. We see two common architectures today:

::::{grid} 1 1 2 2 :::{grid-item} Overused Big VM

They get one Big VM
They leave it on all the time
Maintenance needs are surprisingly demanding
It’s really expensive

::: :::{grid-item} Underused Kubernetes Cluster

They deploy Kubernetes
Experts use it for 1-2 applications
Normal users never really get on board
Maintenance costs and non-use eventually kill it

::: ::::

What people want is simple. They want to turn on a bunch of computers, run arbitrary code on those computers, have those computers turn off reliably, and eventually get logs easily. They want user and cost management without having to set much up.

Don’t AWS/Databricks/Sagemaker already provide these features?

But this stuff already exists, right?

On paper, yes. AWS Batch/Lambda, Databricks jobs, Snowflake’s Snowpark all provide arbitrary code execution on lots of ephemeral machines with exactly those features. In theory, they deliver on this promise.

However in practice we don’t see this. People are still rolling their own with the EC2 and EKS APIs. I’ll bet that if you surveyed data professionals in organizations with access to modern compute platforms you’d find that the majority who would want to use such a tool don’t use them regularly. I’d guess penetration is in the 10-20% range.

Providing functionality is easy. Making functionality useful is hard.

Cloud computing today is like MP3 players before the iPod.

Huge potential demand (mp3s were everywhere)
Theoretical delivery of that product (many products on the market)
Complete market failure to actually deliver on the promise.

MP3 Players and the iPod

MP3 players existed, but only kinda.

Imagine you’re Steve Jobs and you want to propose the iPod in year 2000. Your marketing team tells you that the iPod isn't innovative. Loads of MP3 players are already on the market and doing poorly.

Anyone above the age of 35 knows this history. Recall the RIO 500 and Archos Jukebox.

Rio 500	Archos Jukebox

They store songs, let you play them, fast forward, rewind, and show you what’s going on. On paper they do everything the iPod will do, and yet market penetration was tiny, less than 1% of what it would be. Looking at the lack of success of these products you’d conclude that humans don’t like listening to music.

The majority of people who would have wanted an MP3 player didn't use any of these products.

Eventually, the iPod was released, and it changed our listening habits forever

iPod Generation One delivered the same functionality, but designed well enough for everyone to want.

Same exact functionality, just better in an indescribable and yet obvious way. The combination of ease of use, pocketability, storage capacity, iTunes integration, and straight-up style made the iPod a must-have item years after equivalent products launched and failed to create and claim the market.

Compute Technology has Outpaced UX and Adoption

We as a technology movement have gotten ahead of ourselves.

I believe in the innovation of the Startup + VC process. Yet I think that in striving to climb the technology sophistication ladder (cloud computing -> ML -> Model management -> LLMs -> Agentic systems) we’ve outpaced our users and their ability to follow us.

We’ve chased the few truly sophisticated users out there, the geeky early adopters, rather than the majority of ignorant humans we should be serving. We’ve been building for Steve Wozniak.

Watch 50s of the video above about Steve's cool watch.
Then see ChatGPT explain how to run "Hello, world" on AWS Batch below.

No compassionate human can watch that and think "Yes, the potential of the cloud has been realized".
We're not done innovating yet. There's more to do.

Where does this lead us?

Let’s strive towards excellence on boring problems

I think that there’s huge untapped potential in solving some of these boring problems well. This work is hard and finicky, but I think the data infrastructure space has pushed sufficiently far ahead that we've built up substantial potential behind our frontier.

I’d love to see UX become as strong a focus as AI for the industry. I’d encourage technology leaders everywhere to ask the following question:

What important and common problems today lack products that inspire rabid loyalty?

Those problems may be ripe for iPod-style solutions.
Cloud and compute platforms are certainly among them.

Venting and Constructive Communication

Matthew Rocklin — Fri, 25 Oct 2024 00:00:00 +0000

Venting and Constructive Communication

Constructive conversation and venting are mostly distinguished by asking ourselves

"Will these words help me achieve my goal?"

If the answer is "no" then we're probably venting. This post explores and tries to name this universal experience.

Have a Mental Model of Your Partner

When I'm at my best I have a mental model of the person I'm speaking with.

If they like sports, I'll use sports metaphors
If they're in a rush, I'll be to the point
If they don't speak my language well, I'll use simple language and speak clearly
If they're struggling emotionally, I'll avoid anything critical, nod my head, and listen actively

Effective language targets the listener. It's not the words that we want to say. It's the words that we think will be most effective at influencing our partner. Having a good mental model of our partner helps us craft effective language.

Ideally when I want to influence someone (say, get them to add a feature to a product) I first check my words against my mental model of that person and come up with the right words to engage them in particular. I then hope my model is accurate.

To be clear, I'm pretty bad at this; but this is what I strive towards, and this is my standard for "constructive communication".

Venting

Conversely, when I intercede with colleagues having a dispute (or when I look back at my own disputes) I'll look at e-mails or listen to people talk and I'll see unconstructive language like the following:

"Do you really think that was a good idea?"
"This is a really badly done"
"This direction is obviously going nowhere"

These things might even be true but they're usually not constructive.

If we ask the person who said this a few questions (or ask ourselves, if we are that person), it quickly becomes clear that we're not serving our own interests:

Q: What is your goal here?
A: To get them to change their behavior
Q: How do you think what you've just said/written will land with them?
A: ... I guess it'll make them upset ... but they're wrong!
Q: Do you think that that's more likely to make them change their behavior?
A: Yeah ok, probably not

Obviously no one ever has this conversation, but it's a good check for us to do on ourselves.

Q: Do we think that the thing we just said serves our interest?

If yes: it's constructive, and in service of our goals
If no: it's probably just venting, and we're sacrificing our relationship (and thus any future ability to affect change) for immediate gratification of our frustration

We all vent. Everyone. Me. You. The Pope. Even Barack Obama vents.

Venting isn't professional

Venting is not good, but it is normal. Everyone is expected to vent sometimes.

However, as you climb an organizational chart, you become less able to vent. If you manage people for example, you're not really allowed to vent to them all that much. You can probably vent up the chain of command, but I find it's good to start with "I just really need to vent right now, do you mind?". The larger the crowd you're in (like a big meeting) you're less able to vent; you have to drive the conversation in a constructive direction.

Also, venting damages relationships. Success in organizations depends on one's ability to build and maintain relationships. Without relationships you're unable to collaborate effectively, and you'll only ever be as strong as yourself or your own group. While some ICs might not be expected to maintain relationships across the company, managers and company leaders certainly are.

Relationship building is a requirement for climbing an organizational chart, and venting runs counter to that goal.

There's Gold Behind the Vent

But venting does have real value. There's some idea behind the vent that's causing you stress. Your gut is telling you something important and valuable.

The mark of a real leader is someone who can feel those raw frustrations acutely, vent internally, ask "What am I feeling here?", and then identify the constructive idea behind the vent. Finally, effective communicators think how to constructively communicate that idea to one's colleagues in a way that they will connect with.

This is super hard. If it feels impossible then that's probably a signal that you need to do two things first:

Build a better relationship with your partner

Grab a beer. Talk about their kids. Anything just to jointly establish that you're both reasonable people. Don't push work topics too early.
Build a better mental model of your partner

This way you can shape your message not as you most want to say it, but as you think they're most likely able to receive it

This takes time. It's an investment. It absolutely pays off.

SatCamp Retrospective

Matthew Rocklin — Fri, 04 Oct 2024 00:00:00 +0000

SatCamp Retrospective

This week I attended SatCamp, a small unconference focused on the Satellite / Earth Observation / Remote sensing industry held in Boulder Colorado.

SatCamp was a wonderful collection of imagery nerds talking about technology, public good, cloud performance, and business models, all while hiking/biking/trail running in the Boulder foothills. (This is the most fit collection of humans I've met at a business conference).

I was a bit of an outsider at this conference, so I thought I'd write up an outsider perspective in hopes of giving other people like me a sense for this industry and community.

Disclaimer: *I am not an expert and don't know this space well. The thoughts below are only a first impression."

Big Opportunity / Immature Market

Potential

Satellite imagery's potential value is incredible. We have these technological marvels watching the earth and observing change day to day. Satellites can help us answer questions like …

How is this field of potatoes doing on my farm?
Which of these houses would gain the most from solar panels?
Is this road clear of water so that disaster relief can finally reach this flooded region?
Can we send insurance payments immediately after a disaster instead of waiting months for an assessor to arrive?
… and of course … How many people are at this military training camp?

Struggle

However despite the huge potential, companies in this space seem to be struggling. Giants like Maxar were recently taken private by private equity at a distressed valuation of $6bn (for context Shell's annual revenue is $300bn / year, a rate of roughly one Maxar valuation per week). High profile tech companies like Planet Labs make about $200m per year, and recently underwent personnel contraction. Those are the success stories who were present. Not present included ESRI, which people here refer to with the same gravity as Lord Voldemort. They're a bit more valuable and profitable than Maxar, but not much.

There are hundreds of small climate tech companies who do things like track carbon credits or help companies understand climate risk, all of whom seem to struggle and many of whom are expected to close up shop in the next couple years.

From Pixels to Business Value

My sense is that it's not clear how to convert pixels in a satellite image to business value. There are many possible applications, but relatively few obvious scalable ones. Any institution wanting to get business value has to go all the way from raw satellite imagery to business value on their own, and that's hard.

In most industries there's a collection of companies ranging from the broad and horizontal infrastructure products (think DataBricks), to the more vertically specific products (think the thousands of MLOps companies), and finally to the last-mile consultants, all of whom convert data into insight and value by working together. Satellite imagery / remote sensing / earth observation doesn't have that ecosystem yet. We have consultants who read data directly from AWS without any real tooling support.

It's actually kinda bonkers how much data is available today for how immature this market is.

There are some horizontal product companies, but they seem to struggle to demonstrate real value on their own. Product revenue seems small in this vertical.

Consulting Companies do OK

There are a few consulting agencies in this space (Development Seed and SparkGeo were well represented at the conference, along with a few independent consultants), and they all seemed to do fine. They have to be technically capable, ranging experience from cloud infrastructure, to data science, to understanding verticals like insurance or agriculture. These groups seemed to be the most collected and respected at the event.

The consultants seemed to have "figured it out" while everyone else was struggling.

Exception: Government and Defense

The one exception to "No one knows how to make money here" seems to be those companies targeting Defense and civilian government (like NASA). The larger enterprises (Maxar, Planet) mostly target defense today. The US-based consultancies (DevSeed/Element84) largely target civilian government.

The US Government seems to highly value observing the earth and its people.

Future?

My sense is that there is significant value here, but that it'll take a while for an symbiotic ecosystem of product and consulting companies to grow up and learn to better deliver business value.

This feels like the Internet before web search or Oil before the automobile. Amazing potential, but it's not clear what to actually do with it today.

There's a ton of potential here. I don't know if there's a first mover advantage, or if several companies need to try-and-die here, fertilizing the soil before someone can finally thrive.

Tight Community

This community feels the way Scientific Python felt before we got popular. Everyone is super passionate. Everyone knows each other. Everyone is really kind and supportive. Everyone wants to change the world. It was a great vibe.

Mostly it made me miss the days before Python was flooded with money and success (not that I'm complaining about the money and success).

There was some impractical idealism, but there were also strong and respected voices speaking out for practicality. It was refreshing hearing people speak so much about business models and what they as a community could do to change things, rather than only wishful thinking about how larger third parties should act.

Small Conference Format was Great

This event had a great form factor.

It was capped at 100 people, had only lightning talks and panels, and had a lot of unstructured and outdoor time. It was also a pretty specific community. Ticket prices were modest and my sense is that it wasn't as hard as a proper conference to pull off (not to diminish the effort of the organizers, who were great).

I think the world could probably use more low-stakes and hyper-specific workshops to complement the broad and large events that dominate most work travel today. We ran a Dask developers invite-only workshop a few years ago with about 50 people and it felt a little like that. Great energy.

Technology Needs

From my perspective as someone who cares about Python+Parallelism+Cloud, mostly this community just needs embarrassingly parallel cloud computing with relatively simple processing. Even the sophistication of distributed Xarray often feels like overkill here. I mostly heard people talk about AWS tools like Lambda and Batch, and workflow managers like Prefect and Flyte.

They're definitely inefficient. When asked my favorite question "How much do you think it costs to process 1TB of cloud data" answers ranged in the $20-50 range (real answer is $0.10) which matches my experience with other semi-technical groups.

While many people have access to large scale cloud computing, they mostly have over-engineered systems around Kubernetes or cobbled together AWS architectures, typically that reduce to just AWS Batch-or-Lambda.

When showed Coiled demos people were generally blown away, mostly by ease of use, simplicity of architecture, and cost efficiency. The sophisticated task graph stuff or ML libraries didn't matter much.

Final Thoughts

I had a good time (people were great) and learned a lot (people should attend events adjacent to their space more often). I'm oddly both more and less enthusiastic about engaging this community than I was before.

Photos

Thanks to the folks who took these.

How I Almost Quit My Job, and How Being Selfish Saved Me

Matthew Rocklin — Fri, 20 Sep 2024 00:00:00 +0000

How I Almost Quit My Job, and How Being Selfish Saved Me

Every human being has a basic instinct to help each other out
-- Mark Watney, The Martian

Leverage and the loss of ego

Positions of leverage feel great, but result in self-sacrifice and burnout.

When in a position of leverage, a small amount of your work helps others a great deal. Sacrificing minutes of your time saves days of someone else's, so of course you sacrifice. You'd be a monster not to. Sacrificing for others is part of being a human; we're social creatures and helping others is deeply rooted in our DNA.

And helping other people feels great! Especially when it's so easy. This feeling makes you work long hours, and squeeze extra time from the odd corners of your day, helping unblock as many people as you can. Unfortunately, your personal life and ego can fade in service of your project.

Sometimes this fade results in repressive or self-destructive behaviors

"I've put on a few pounds sure, but I don't really have time to go running with this new project starting"
"Yes, this person's behavior makes lose sleep, but if I show how I feel it'll cause chaos because of my position"
"My partner doesn't like it when I work after they come home, but there's a really important project going on, and all they're doing is watching TV."

There are natural release mechanisms like venting anger or taking time off that stop making sense day-to-day when you're in a position of leverage. Your time and position are too important to let loose. When these release mechanisms no longer make sense day-to-day, your leverage swallows part of your identity and, eventually, you break (or at least I do).

This breaking point is an interesting moment because it's at this point where the entire project is at risk unless you fix a very important problem, yourself.

You can't fix yourself through more sacrifice. You can only fix things by selfishly prioritizing your needs. This is a rare moment in life where egotism and altruism are perfectly aligned. However, prioritizing your needs doesn't mean "do less work". Often you can achieve far more effective interventions than just "work less". You can reshape your relationship to work in more creative ways than by just stopping.

This is the moment when you say "Fuck it" to your colleagues and start working from the beach or while motorcycling across America (or whatever). It's either that or you and the whole system break down.

Personal Examples

I've almost quit my job (running an early-stage tech company) several times in the last five years. Rather than actually quit, I've instead decided to make changes like the following:

I have no recurring meetings
I don't log into Slack
I've checked out of work for a month, focusing more on running and yoga than customers or team morale
I've bailed on conference talks and company travel for six months

These actions are selfish, but they've also improved my life tremendously and saved my relationship with work, giving me the space to continue high-leverage work. If I didn't figure out these creative ways to be selfish, I would have broken and everything else would have broken for others around me.

Every time I made a change, I came back with more energy and more impact. These changes improved everyones' outcomes, not just my own.

Optimize for Yourself, Creatively

No one has full freedom over their worklife, but everyone has a lot of freedom, far more than they exercise.

Most founders and other leaders I run into are stressed out. Most question their relationship to work. My advice to them (and to you) is to be selfish, creatively. Don't just stop doing work or helping others; figure out very specifically what you want and design work around what you want. The higher the leverage of your work, the more power you have to tailor your work life to suit your needs.

What Does Pangeo 2.0 Look Like?

Matthew Rocklin — Wed, 18 Sep 2024 00:00:00 +0000

What does Pangeo 2.0 Look Like?

Past

In January 2018 I published a blogpost titled Pangeo: JupyterHub, Dask, and XArray on the Cloud, which introduced a new architecture for running scalable Python computations on the cloud using ...

GCP
Kubernetes
Dask + Xarray
Dask-Kubernetes (actually named Daskernetes at the time)
Zarr (a brand new file format at the time)

This architecture was unveiled during an AMS workshop keynote, the 3-minute demo video of which is still available if you want a blast from the past:

This coordinated work from many people.

What was great about the Pangeo stack

This effort was impressive, and showed that it was possible to …

Do large scale multi-dimensional distributed computing
… from Python
… on the cloud
… interactively
… for anyone with access to the internet

This was quite different from the standard at the time in the scientific/geo space, which was MPI codes or batch jobs run on supercomputers. The Pangeo effort upended the status-quo at many scientific and government agencies trying to enable access and analytics on larger scale data.

What was terrible about the Pangeo stack

Unfortunately, there were challenges with this stack:

Kubernetes: It required expertise of Kubernetes, and that expertise was rare

"What’s a pod? Is it a Docker? How do I upload a pod?"

Costs: Controlling access and costs and usage was impossible

"I want my graduate students to use this, but I’m worried they’ll go beyond my budget"

Maintenance: It was difficult to deploy and maintain

"How do I tune my cloud autoscaler? Actually, what is a cloud auto-scaler?"

Regular Jobs: It failed to address non-interactive use cases

"My notebook is great, but how do I run this job every day?"

Rigidity: It assumed a uniform development environment

"I want to use the newest version of scikit-learn, but the Docker image has an old version"

Multi-cloud: It didn’t work across regions or clouds

"How do I access my collaborator’s data in Europe?"

"Does this work on Azure?"

Storage: File formats were still wonky

"Zarr seems cool, but we’re still using NetCDF.

Also, my data keeps updating, can Zarr expand like this?"

Scale: Scale was still an issue

"This works great on 200 GiB, but when I move to the full dataset my workers start running out of memory"

Personnel: It required people to keep it alive, and no one was paid to do so

There were incremental improvements in the components of the stack. Dask-gateway replaced dask-kubernetes. Zarr became more common. Dask and Xarray both improved in scale. There were also many extensions of it, with new libraries built on top of Xarray and a wonderful growth in community engagement.

However overall this deployment architecture was a technological dead end (at least in my opinion as original architect). Personally I’m proud that we were able to generate so much enthusiasm (most of the credit here due to Ryan Abernathey, Rich Signell, and others who pushed the Pangeo stack within scientific circles) but mildly ashamed of the architecture itself. It’s easy to build a cool demo with the Pangeo stack, but difficult to actually solve the computational and business needs of larger organizations. My experience has been that the Pangeo stack was fantastic demo-ware.

Present

Since 2018, there have been several re-imaginings of this stack. There are full-platform options (everything included, but not particularly powerful) and some tool-specific options (innovative and powerful, but require assembly). Let’s list a few in each category:

Full Platform Options

2I2C is a classic Kubernetes + JupyterHub architecture similar to what we built in 2018 (pure OSS), but sold with a service to set up and maintain it for organizations that don’t happen to keep a Kuberentes and cloud expert on staff.
Nebari (née QHub) is a better architected execution of the same components, and with more components (workflow managers, editors, …) Nebari is pure OSS and deployed using git-ops, but recognizing that people need help (OSS isn’t free to maintain) Nebari is often sold with services from Quansight who maintains it.
Saturn took the same UX with Jupyter and Dask (and others tools), but dropped dask-gateway and JupyterHub, and instead added in a proper database-backed web application. Saturn is a commercial SaaS product with one professionally managed central control plane that can be used anywhere.

Innovative and Tool-Specific Options

Coiled (compute) is what I and people around me built.

Coiled focuses exlusively on computation, while also including all of the user management and cost controls that were missing in Pangeo. Coiled doesn’t serve Jupyter or store data. Instead, it works with other tools users already have, running from anywhere people run Python (most notably, their existing laptop).

I’ll claim (perhaps with hubris) that Coiled is among most high-end Python cloud compute platforms on the market today. This product was built with creativity and soul, and delivers a developer experience that I’m proud of (and user praise seems to back this perspective).

Arraylake (storage) is what Ryan Abernathey and Joe Hamman (the other two primary Pangeo architects) built at Earthmover.

Arraylake is a data lake platform built around the Zarr data model which incorporates many database-style features, such as ACID transactions, data version control, fine-grained access controls, and a built-in data catalog. A bit like Zarr + Git, or Delta Lake if you’re familiar with the dataframe storage space, Arraylake solves many of the data management challenges we experienced in the first iteration of Pangeo.

Other tooling like workflow managers (Prefect, Dagster), code editors (VS Code), ML/AI tooling (PyTorch, LLMs) and more have come online. The data infrastructure world today is a vibrant and mercurial environment.

Status

Today I see a few things:

Coiled and Earthmover deliver an innovative top-shelf experience to some geospatial users, but not many. Those users tend to be early adopters and low-bureaucracy users like startups rather than older and larger organizations like NASA, weather agencies, and the energy majors.
Home Grown Pangeo 1.0 deployments still get proposed within large agencies by IT groups who want to solve the data accessibility problem, but don’t land well on completion.
2I2C and Nabari deployments are used in more on-prem-ish environments where SaaS isn’t welcome, but end up delivering mostly a JupyterHub+ experience. While users report enjoying the cloud notebook capability, I haven’t seen an organization succeed with scalable/distributed computing with these platforms in practice.
Single large VMs tend to be standard. While simple, this often means that users don’t do full-dataset computations and, ironically, have crazy high cloud bills (single large VMs tend to be set up in a way that they stay on forever, which can be startlingly expensive).

Community Developments

There has also been a lot of wonderful work in building community around Pangeo, with a thriving discourse forum, numerous third-party packages like xbatcher, xradar, rechunker, etc.., educational groups like Project Pythia, and more.

While core technological innovation has struggled, there’s been a lot of innovation on the human side.

We’re at a strange place where we have a compelling POC (thanks Pangeo 1.0!) that’s motivated a lot of people to act, but no great place for them to go just yet. We also have an encouraging set of next-generation products (Coiled, Arraylake, Prefect, …) pointing the way to a bright future, but nothing that a mature organization can easily pull off the shelf. Procuring and integrating services from multiple different SaaS vendors comes with its own set of headaches.

Future

So what should we do here? What does the Pangeo 2.0 platform look like? Ryan Abernathey and I were discussing this topic last week over drinks and started shooting lots of ideas at each other.

Is it just Coiled with notebooks and regular jobs?
Arraylake but with online update algorithms?
Does Saturn do everything people need?
Maybe it has to be on-prem like Nebari for government folks, but with a proper user database? Can we get government on Cloud SaaS?
Maybe there are many different architectures for different kinds of applications and so organizations are fated to DIY?

I’ll give some of my opinionated constraints below. I think that we got some things right with Coiled:

Raw-cloud architecture rather than Kubernetes
Environment synchronization rather than Docker images
Cost controls and user management in a database, rather than JupyterHub/Kubernetes
Centrally managed SaaS product with one set of operators, rather than several instances of the same stack
Trivial integrations with other tools (our integration is just import coiled in a Python prompt)

But maybe we got some things wrong:

Notebooks: while I still believe that notebooks and computation should be separable (it’s important to drive computations from things-other-than-notebooks) I now realize that single-node cloud notebooks are sometimes valuable, especially for organizations who want to deliver a more curated and homogenous experience to a population of non-power-users. (I tend to overweight the experience of power-users)
Regular Batch Jobs: while I still believe in integrating with best-in-class workflow orchestration products like Prefect/Dagster, some organizations want something trivial baked-in, particularly when migrating from an on-prem HPC-style environment
SaaS: I’m actually still pretty convinced that this is the right way forward. SaaS within government agencies has proven a major source of friction, but the development velocity is just so much faster.

Additionally, I’ll follow with some perhaps unpopular organizational perspectives:

Deployment Infrastructure should be SaaS rather than OSS
Faster iteration (we can release daily rather than monthly)
Better visibility (we can see every failure, often before users do)
Maintenance costs are a tiny fraction when shared among different user groups (we service hundreds of companies with a tiny cloud dev staff)
For-profit companies move faster than the community
Most organizations mostly care about if it works, rather than who builds it

However, I still have lots of questions:

What key activities and pain points are people looking to solve today? What features do you need?
Dask / Xarray features?
User / cost / administration features?
Notebooks in the cloud?
Better Scalability?
Batch jobs / workflow orchestration?
Databases and data catalogs?
How do geo-specific organizations buy software?
Do you buy software or do you build everything yourself?
Do you deploy SaaS or do things have to be installed on-prem or in fully-isolated cloud VPCs?
Who makes these decisions? What do they care about?
How do science users impact this decision-making process?

If you’ve made it to the end of this post then I could use your help answering these questions. If you or someone you know works at an institution that buys this kind of software and are open to a 30 minute conversation, then I’d like to probe your brain with a few questions like these and gather your perspective.

Send me an e-mail at pangeo-2.0-discovery@coiled.io and I’ll schedule a chat this week.

Usage Based Pricing

Matthew Rocklin — Wed, 04 Sep 2024 00:00:00 +0000

Pivoting away from Usage Based Pricing

We're moving away from usage based pricing.

This post talks about the pros and cons of usage based pricing in our situation, and then talks through alternatives.

Where we are today

People who follow my blog may recall that ...

We have a product people love
But we weren't selling it well
So we hired AEs (new!)
They're working out fantastically well (new!)

Hooray 🎉! The bottom of our funnel now flows decently and our revenue is dramatically increasing (easy to increase dramatically when it starts low 🙂)

However, now we have a different problem. Pricing! Today our biggest problem is figuring out how to capture more of the value we generate for customers while also making them feel good about paying us. This problem is tricky.

Today, we do usage-based pricing. This was good for a while, but ended up being a bit of a dead end for us. Let's talk through why.

Why Usage Based Pricing was Great

Previously, we charged $0.05 per CPU-Hour, the same as major clouds or Databricks. We liked this when getting started for a few reasons.

It’s simple and it’s easy to understand quickly
It’s standard in our space and so familiar to buyers
It starts small so on-boarding isn't focused on financials
It’s on-par with other things the customer is already paying for (the cloud in this case) so we know that they have budget for bills like this
It’s unbounded so if a company does really really well, we can make a lot of money

Mostly it was a simple default choice, and a good basis for a pricing scheme. It’s minimally scary, and so good at getting people in the door.

But eventually it broke down. Let’s talk about how, and then we’ll talk about what we do today.

Why Usage Based Pricing was Terrible (for us)

There were a few fail cases:

Customers for whom our product was critical infrastructure (high value) but whose usage was really small (low price)
Customers where we optimized their workloads 10-100x, and so reduced our revenue 10-100x (misaligned priorities)
Customers who didn't like percentage increases on their usage (like how paying $10 for a bottle of water feels bad at an event)

Consider the following conversation as an example:

Customer: You charge a 100% markup. That’s too high.

I was expecting closer to 30-40%. I’m paying $2,000 monthly to AWS and $2,000 to you
Us: Yes, but before using us you ...
- were paying $20,000 to AWS because you were so inefficient
- were spending half an FTE maintaining that terrible infrastructure
  
  (costing 50% * $200k == $100,000/year) - had a data science team of five was spending 20% of their time fighting that terrible infrastructure
  
  (costing 5 * 20% * $200k == $200,000 / year) - couldn't finish that project that was critical to your business and which you were just able to finish in a couple weeks since using us
  
  (the value of which only you can determine, but is probably higher than the rest)
Customer: Oh yeah, I guess that's right. I guess this is worth the $2,000 you're asking for
Us: actually, now that I think about it, we should probably be charging you much more

(I never actually said this last part, but it was a frequent thought).

There were two problems here:

We captured only a small fraction of the benefit we provided to users
Users felt bad about the small fraction that we did capture

Something was clearly wrong here. Users value our product beyond what they pay (they’re getting a great deal!) but they feel like they’re getting screwed, and that's not good for anyone.

I've learned that pricing is as much about psychology as about aligning cost and value.

Why did this not work out for us?

Usage-based pricing is the norm for computational cloud products. Why does it work for others (like AWS or Databricks) but not for us?

I have a few theories:

Efficiency we're just much better at making users efficient.

Users often come in with $100,000 annual cloud bills, but after using the product realize they only need to spend $5,000-10,000.

The vast majority of cloud usage today is misuse and our product stops misuse. Stopping misuse is great for the user (🎉) and bad for the cloud (🎉). However, it's also bad for us (😞) under usage based pricing.
Python is more prone to these 100x efficiency swings than other technologies, like SQL, where 2-5x is more common.

Python can be either incredibly efficient or inefficient based on expertise. With our product novices are less cost-prone, both because of the product itself, and because of our incredible support team (more on this later)

We could make the product worse or stop giving good support and we'd probably earn more but this doesn't feel great. There's clearly a lot of value that we can generate and would like to capture.
SMBs for velocity reasons we mostly focus on Small and Medium Businesses (SMBs) rather than large enterprises.

SMBs tend to value human performance more than cloud spend (humans and speed are more expensive than cloud at this scale). Maybe our focus here puts us in a strange place where the main value we provide isn't aliged with cloud cost, but rather human acceleration.

After seeing the $100,000 -> $5,000 transition a few times and cursing the efficiency of our product we started rethinking how we price the product.

So let's look at some possible new solutions. Before that though, let's talk briefly about what we sell.

What do we sell?

In general people like using Coiled because they get to operate at larger scale more smoothly than before. This is due to a couple of things:

Product: the Coiled product is powerful and has a UX that feels good to Python data devs. It's a powerful tool that's comfortable to wield.
Support: the product also tells us a lot about what users do, and our support team reaches out proactively as we see users struggle. One Coiled support engineer easily unblocks a dozen users daily.

Users love our support team, and describe it as unlike anything else they've experienced. It's like we've constructed a platform that enables us to do lots of high impact micro-consulting. Unfortunately, the market isn't used to this.

The product is hard to price because of the efficiency issues with Python (we often make people 10x more efficient and so shoot ourselves in the foot).

The support is hard to price because we're trapped in between two things people are used to:

Typical (bad) product support most SaaS products have some support tier that costs 20% on top of usage and offers access to a mediocre support experience. I've personally never found value in SaaS support teams.

This is like what we do, but cheaper and worse.
Professional Services consulting firms charge by the hour to unblock customers. This is much more valuable, but a way of working that's also pretty distracting and that we're not set up to do (I'm done counting hours).

This is more like the value we provide, but operationally quite different (we dont bill hourly and we only work on what we want to).

Our support is similar enough that we get pattern matched to existing offerings, but different enough that those approaches don't make sense for us.

Objectives for a good pricing scheme

So let's say we want to invent something new? What are some objectives?

Easy to start using the product without worrying

(we want the customer to see the value of the product before thinking about money)
Total price matches the value the customer sees and can pay, so if a customer ...
- is unable to pay anything (like a student) it should be free
- is getting a lot of value and is able to pay, it should cost a lot (but less than the value they experience)
Mentally easy for the customer to connect price to value

We want the customer to be able to easily understand the price and connect it to the value they're getting. So for example if most of our value is accelerating their people, then we should charge per person.

What we're trying today

Today we're rolling out a change to our pricing.

For basic/individual users not much changes. It's still usage-based which makes it easy to test things out without thinking too hard. We are putting some seat-caps on this tier though.
For larger teams we switch to usage + seat based pricing. We charge some very small percentage of an FTE cost per seat. This is big enough to be meaningful to us, while small enough that when the customer asks themselves "How much more efficient is my team with this tool?" they come up with an answer that is far larger than the percentage we charge.

Future work

This feels better but still not great.

It feels good that we're shifting to human-based pricing (to match the human-based value we think we generate)
But human-based value feels squishy and hard to quantify, which requires more thinking on the user's part, and more conversation on ours.

We have our work cut out for us. I look forward to seeing how this lands.

How GitHub taught me to Micromanage

Matthew Rocklin — Sat, 11 May 2024 00:00:00 +0000

How GitHub taught me to Micromanage

Feedback is critical to performing good work as a team. Good feedback cultivates quality work and professional growth. Bad feedback degrades quality and erodes relationships.

This article explains my experience of feedback in open source community code review, and the culture that emerged from intense detailed feedback. Then we shift to corporate settings and see and how these lessons fail to apply cleanly.

Open Source Code Reviews

I first experienced transformative feedback when working in old-school open source communities. These weren't corporate environments. These were a bunch of volunteers working together to build something awesome used by rest of the world. There were no managers, no performance reviews, no video meetings, no meetings at all in fact. There was just the work, and the work was the focus of all communication.

The work took three forms:

Issues: where we discussed concerns and considered technical designs as a group
Coding: where one person would write a solution to the proposed issue
Code Review: where anyone could drop in, review the code, and provide feedback.

Most of the work was in Code Review, not Coding (writing code is a vanishingly small fraction software development) and so I spent several hours every day, for years, giving and receiving highly detailed feedback from fellow collaborators. This is where we taught each other. This is where we learned to align on mission and style. This is where we expressed humor and learned about each others' character and about our own.

Let's see an example

Example

In this example we see some simple proposed code below, and then consider bad/ok/good feedback on that code.

Proposed code

x = 0 for element in data: x = x + f(element)

Bad Review

Please don't use for loops. They're hard to understand. Also please use informative variable names.

OK Review

Please don't use for loops. They're hard to understand. Also, please use informative variable names.

How about the following?

total = sum([f(record) for record in housing_records])

Good Review

I suggest that we avoid explicit for loops here. I think that a list comprehension would be a better fit.

This is a bit subjective, I find that list comprehensions make it more clear to the reader what's going on at a glance because they're a more constrained construct. (The same could be said for using map, but I rarely see that pattern in this codebase).

How about this?

total = sum([f(record) for record in housing_records])

Now, this practice can definitely get out of hand (I've seen some crazy nested list comprehensions) so there's a tradeoff between using common explicit technologies like for loops (easy to reason about, but you have to read them) and using patterns like list comprehensions (tricky when large, but immediately recognizable if simple). I think that for simple cases like this the tradeoff is strongly in favor of the list comprehension.

Anecdotally, this is also the norm in this codebase.

Analysis

Let's assess each piece of feedback and why I personally think that they're bad/ok/good.

Bad feedback critiques code, but doesn't offer a better direction.

Critique without constructive direction is just complaining, and adds constraint to the author without also helping with momentum.
OK feedback provides direction.

The reviewer works with the author to produce a better result.
Good feedback educates the author.

The reviewer works with the author to produce a better result.

The reviewer works to also improve the author, an investment in their joint future.

The good feedback is thorough, especially given how trivial the code is, and especially given that the new solution and the old solution do the same exact thing. It's not like there was some important bug here.

The main reason for good feedback is to invest in the author.

Role of Code Review

We see then that code reviews serve a few different roles:

Ensure quality work
Teach fellow developers
Align on conventions and process

Personally, this is how I learned to write code. This community and practice is the reason for my success in career, which has transformed my life. I am grateful. This community and practice is also the primary way for me to give back to fellow devs around the world. I can see the impact I and others have had daily in the lives and success of colleagues that I've known for decades.

OSS Idiosyncrasies

After thinking about this life experience and comparing it to corporate work (which I'll get to next), I think that there are some interesting idiosyncrasies about this practice and people that are worth mentioning:

Craftsmanship: These people care deeply about their craft
Volunteerism: Everyone is a volunteer, spending their personal time on this activity
Work Focus: All feedback and conversation is focused on the work, rather than on the person
Public: All feedback is fully public
Respect: Respect is assumed broadly (we rarely volunteer with people that we don't enjoy)

Corporate contexts

Why the heck is the CEO correcting my grammar!?

This same approach of highly detailed and thoughtful review does not transfer well to corporate contexts in my experience. As an example, let's consider the following exchange over a google doc:

When I first transitioned from OSS dev to CEO at a startup I did lots of review like this. What I found were questions/concerns/problems like the following (all of which are valid in hindsight)

Why is the CEO correcting my grammar?
If you want these changes, why not just edit the doc directly?
This doc isn't high priority. Why do you want to talk about this so much?
Why do you keep telling me how bad I am at writing?

These questions confused me. I realized that it was because that the implicit message received behind my feedback was very different.

Implicit messages

There was a major difference in the implicit message that is being received in the first open source case than in the second corporate case.

Open Source: I believe in you and your work. I see potential for you to get even better at your craft and would be proud to invest my time to see that happen.
Corporate: You're doing a bad job. I'm going to explain, in minute detail, exactly what you did wrong, over and over again because I hate you and want you to suffer.

What before was a supportive behavior is now an abusive one. Everything depends on context.

Note: to be clear, this miscommunication isn't the receiever's fault. We as reviewers need to better model our context so we can anticipate how feedback will be received.

Corporate Mindset

This negative reception feels more reasonable when you consider the mindset and context in which this work is done. Referring back to the OSS Mindset above, let's compare with corporate.

Honing a craft: in old-school community OSS people cared a lot about getting better at development. Micro-adjustments to our form were really valuable to us. Development skill was a big part of our (or at least my) self-identity.

Conversely in corporate few people care as deeply about the craft of writing, meeting facilitation, graphic design, etc.. Skillbuilding is nice, but it's secondary to getting work done. The assumption is that feedback is primarily focused on the job at hand, rather than on building our teammates.

Assumed Respect: in volunteer OSS if people don't like you they just ignore you. Anyone reviewing your work is doing so because they care and want to help you.

This doesn't hold in a corporate environment, where people are paid to help whether or not they like you.

This does beg the question though, what would a corporate environment look like where people cared deeply about things like meeting facilitation and slide design? Probably the company would go quickly out of business, but it's a fun thought to consider, and maybe apply a little bit of in our everyday life.

Feedback Reception Budget

And so, I've learned that a person can absorb only about one piece of feedback per interaction, and then only if it's labeled very specifically as feedback with the best intentions, and wrapped delicately in a feedback sandwich.

Because of this volume constraint, infrequent feedback must be more general and higher level, ideally accompanied by impersonal assets like a blogpost that the recipient of the feedback can read separately (this is partially why I write blogposts).

But please ...

For the love of god ...

Remove filler words like "really", "very" or anything else that doesn't change the meaning of a sentence
Do not start meetings by asking everyone to introduce themselves
Attach agendas to meetings ahead of time
When making slides think about text contrast and use alignment guide-lines

Thanks for listening, hopefully this article ...

Helps people around me understand the implicit message behind my feedback
Gives everyone desiring a spirit of craftsmanship more interest in developing a strong culture of constant constructive feedback

Matthew Rocklin

Stepping Back

Stepping Back

Coiled

Dask

Personal Situation

Final Thought

FAQ

Saying Goodbye to Winner-take-all Software

Saying Goodbye to Winner-take-all Software

Software is becoming more idiosyncratic

Funding will change

Open Source will change too

Final thoughts

Claude Chic: Week 2

Claude Chic: Week 2

Multi-Agent Workflows

Plan Swarms

Review Agents

Infrastructure

Custom Themes

The --yolo Flag

Performance

Quality of Life

Bug Fixes

What's Next

Contributors

Multi-Agent Workflows

Multi Agent Swarms: Initial Thoughts

Obviously Good Simple Multi-Agent Workflows

Platform

First Experiment: Diplomacy

Second Experiment: Session Viewer

Prompts

The overall experience was mediocre

Observations

Lessons Learned

Separate Reviewer 👍

Fresh Perspective 👍

Mythical Man Month ➡️ Mythical AI Minute

Agent Support 👍

Human Coordination 🧠

Human-Oriented AI Development

Experimenting is easy

Beginning Again

We're all Beginners Again

Beginning again

Programming feels fun again

Learning from Python

Claude Chic

Introducing Claude Chic

0. Motivation

1. Style

Comparison

2. Git Worktrees

User Experience

Cleaning up

3. Multiple Agents

4. Quality of Life

Diff Review

Shell Access

Relationship to other projects

Alpha Status

Final thoughts

Video

AI Zealotry

AI Zealotry

Why AI

Why Not AI

Big Idea: Minimize Interruptions / Climb Abstraction Hierarchy

Tip: Hooks

Example Problem: Ignoring instructions in CLAUDE.md

Solution: Hooks

Example Problem: Incomplete Permissions

Solution: Hooks for permissions

Solution: Hooks for sounds

Big Idea: Build Confidence Without Looking at Code

Tip: Self-review

Testing

Grilling

The `--yolo` Flag