Anthropic published a study that puts numbers to something I’ve been writing about in the design context for a while now. They ran a randomized controlled trial with 52 junior software engineers learning a new Python library. Half used AI assistance. Half coded by hand.

Judy Hanwen Shen and Alex Tamkin, writing for Anthropic Research:

Participants in the AI group scored 17% lower than those who coded by hand, or the equivalent of nearly two letter grades. Using AI sped up the task slightly, but this didn’t reach the threshold of statistical significance.

So the AI group didn’t finish meaningfully faster, but they understood meaningfully less. And the biggest gap was in debugging—the ability to recognize when code is wrong and figure out why. That’s the exact skill you need most when your job is to oversee AI-generated output.

The largest gap in scores between the two groups was on debugging questions, suggesting that the ability to understand when code is incorrect and why it fails may be a particular area of concern if AI impedes coding development.

This is the same dynamic I fear in design. When I wrote about the design talent crisis, educators like Eric Heiman told me “we internalize so much by doing things slower… learning through tinkering with our process, and making mistakes.” Bradford Prairie put it more bluntly: “If there’s one thing that AI can’t replace, it’s your sense of discernment for what is good and what is not good.” But discernment comes from reps, and AI is eating the reps.

The honest framing from Anthropic’s own researchers:

It is possible that AI both accelerates productivity on well-developed skills and hinders the acquisition of new ones.

Credit to Anthropic for publishing research that complicates the case for their own product. And the study’s footnote is worth noting: they used a chat-based AI assistant, not an agentic tool like Claude Code. Their expectation is that “the impacts of such programs on skill development are likely to be more pronounced.”

I can certainly attest that when I use Claude Code, I have no idea what’s going on!

The one bright spot: not all AI use was equal. Participants who asked conceptual questions and used AI to check their understanding scored well. The ones who delegated code generation wholesale scored worst. The difference was whether you were thinking alongside the tool or letting it think for you.

Cognitive effort—and even getting painfully stuck—is likely important for fostering mastery.

Getting painfully stuck. That’s the apprenticeship. That’s the grunt work. And it’s exactly what we’re optimizing away.

Karo Zieminski spent nine days breaking Claude Cowork before writing this guide:

I’ve seen enough of shallow tutorials that simply rephrase the official docs to know I wanted to do something different. So I rebuilt some of my workflows from scratch, tracked what failed, measured what saved time, and mapped 56 practical tips into the resource I wish existed when I started.

I appreciate her methodical breakdown of the app, especially when to use which flavor of Claude, which for me TBH, has been an issue.

Zieminski’s nice breakdown of the differences between Claude Chat, Cowork, and Code.

The guide barely talks about prompting. It’s almost entirely about the pre-work: dedicated folder structures, global instructions via CLAUDE.md, chunked skills, delegation patterns that define end-states instead of steps. The distinction Karo draws between Chat skills and Cowork skills:

Skills in Chat were useful. Skills in Cowork are operational. They shape autonomous work. Your brand guidelines skill doesn’t just influence a reply. It governs every file Claude creates. Your writing guidelines skill doesn’t just shape a draft. It governs every article Claude writes autonomously.

Zieminski on skill architecture:

Chunk your skills instead of building one giant skill that tries to handle everything. I’ve tested both approaches and the results from one giant skill were much worse. For example, I use three separate writing skills instead of one: an overall voice skill, a corporate writing skill, and a newsletter writing skill. Each handles its own context. Claude never confuses who I’m writing for.

If you’re already using Claude Cowork or just Cowork curious, bookmark this one.

Claire Vo on her How I AI podcast gives a plain-language walkthrough of agent loops in Claude Code and Codex. The episode is technical, but the useful translation for designers is familiar: define the job, give it tools, decide what good output looks like, and make the system prove it. Once an agent can run without you, the interface problem moves into instructions, permissions, and checkpoints.

And then a couple other foundational things that I think are helpful when you’re running loops. And why are these things helpful before we get into what they are? They just keep the work clean.

If you are going to be yolo-ing loops all over the place, you’re going to want some consistency in execution, you’re going to want clean workspaces, you’re going to want conflicts resolved and avoided. And so, all these things are really to make those loops effective.

And so, what are the things? They are work trees. I feel like this entire podcast could be Git 101. But work trees are just basically a way to isolate the work, especially the coding work of an agent away from other agents’ work in a sandbox. There are skills, repeated ways to do common tasks. We have a full episode on what skills are from earlier last year when they came out. Plugins and connectors, these are just the tools that your agent has access to. […] Sub agents, both Codex and Claude Code allow you to kick off sub agents. This is just a way to federate out work from the main thread, so that sub agents can do specific tasks, especially validation. And then there’s some way to track state. And essentially just think of this is like a to-do list.

The practical translation for agent orchestration: write the operating conditions around the work before asking for the artifact.

Calling it a loop makes it sound like automation plumbing. The design work is the brief: what the agent can touch, what tools it can use, and how it knows it is finished.

The employee comparison makes the loop feel less exotic:

Now, people are going to ask, “What should I use a loop for?” And when you’re designing loops or designing agents, I say this is the time for the manager. You are designing a job. And so, just imagine that you’re onboarding an employee. That employee could be an executive assistant. That employee could be a customer service agent. That employee could be a software engineer.

Every Friday, EA, I would like you to review my calendar, see who canceled on me, where I could have used my time more effectively, if there are any follow-ups, and send me a Slack to get this done. Um and I want you to do that every Friday. Guess what? You’ve just designed a loop for your executive assistant. […]

So, I really like to think about loops as designing workflows and designing jobs to be done for people. It just happens to be that you can put this intelligent agent against the loop, and then it’s ready to go.

For designers, that last sentence is the translation layer. A designer already knows this move from creative direction: clear constraints produce better work than a blank request. The only difference is that the worker now happens to be software.

A loop without success criteria is just a faster way to make someone review the same ambiguity at a higher token cost.

We all want our agents to work for us on a schedule whenever we want, doing work that we don’t want to do. It’s great. What are some of the problems? One, loops can get expensive. So, I just kicked off an automation that happens on a regular basis. It does wide-ranging work. It decides when to spin off sub agents.

And it does loop-based validation, which means it’s burning tokens until it hits a threshold that it decides is successful. If you do not write that loop well, or your validation criteria is too thin, guess what? Your agent is going to burn tokens. I think we’ve seen this with open claw in particular or some of these agent harnesses is they’re really good at loops. They’re very diligent. They get interesting work done, but man, do they love to burn tokens.

[…] Loop-based prompting is just its own thing. Goal-based prompting in specific is just its own thing because you have to be very precise about evaluation and success criteria. If you are not, you will be very disappointed and use a lot of tokens for not a lot of output.

Software developer raf, who learned the craft by building and breaking web applications, describes a new apprenticeship problem that starts with an apparent success: a beginner can generate competent work on day one.

I’ll tell you this much: it’s not from consuming good work. You cannot read a hundred excellent programs and absorb the judgement by osmosis, any more than you can become a chef by eating in good restaurants. Taste is built the slow, stupid, humiliating way: you make something bad, you are forced to live with it, it fails in front of you, and some part of you files the failure away. Then you do it again. The palate is an accretion of your own mistakes, sat with long enough to sting.

The friction was not an obstacle to developing taste. The friction was the curriculum. Every wall I cursed while climbing it was, without my noticing, teaching me which walls were worth climbing. The cost that rationed my output also educated my judgement, because paying the cost over and over is how you learn what is worth paying for.

For designers, the old bottleneck did two jobs at once. Producing a weak concept, defending it in critique, watching it fail with users, and revising it made bad choices costly enough to remember. AI can compress that cycle into a polished mockup before a beginner has learned what to reject.

raf on the cost of taste:

This is the quiet cruelty of the situation and I do not have a comforting way to phrase it. Taste is slow. It says “no, again.” It sends the plausible thing back because plausible is not the same as right, and while it is doing that, the person without it has already shipped, closed the ticket, and moved on. The market timed you both with the same stopwatch and it did not see the difference. It cannot see the difference. Taste does not show up in the diff.

That makes critique more important, not less. Design teams need to value frank feedback from the skeptics in the room, not only the ones who ship fast.

Adam Waxman built five personal tools using his household’s rules and data from several apps. Most work the same way:

Aggregate data, add an LLM. The cost drop doesn’t just mean more apps, it means apps can be far more personal. Most of mine follow the same shape: pull data from multiple sources, combine it in one place, and use an LLM to generate insights from the full picture. My fitness app combines nutrition, sleep, weight, and training data that lives in four separate apps, then uses that context to make recommendations none of them could alone. I think this pattern will spread to professional tools too.

Two more lessons:

Ephemeral is fine. We used the sleep app for about four months. Our son sleeps through the night now, so we retired it. If the app had taken me months to build, that might sting. Because it took a week, I’m just glad it worked when we needed it.

AI floods big markets and unlocks small ones. Yes, AI produces endless derivative apps. But the same tools also let me build apps for my household that no company would bother making. Beyond my household, I built nycjazz.guide for NYC jazz fans and claudecodedaily.com for Claude Code developers, audiences too small for a business but worth serving.

These fleeting micro-apps change what is worth designing. An app no longer needs a large market or a long life to be useful. They also avoid the customer adoption problem Andy Budd describes: Waxman starts with one household and one problem, not a market he has to win over.

These personal apps depend on other products letting people use their data:

Good APIs matter more than ever. I switched from Cronometer to FatSecret because FatSecret had a better API. My fitness app pulls from Strava, Oura, Withings, and FatSecret, and the quality of each integration depends on how well the API is designed. As more people build personal software, users will expect their apps to have APIs and MCPs worth connecting to.

Designers now need to ask whether people can get their data out. If a service keeps that data locked inside, people can’t use it to build tools that fit their own lives.