Claude the scientist(?)

A narrative report on letting Claude co-author a scientific research publication

Mistakes during keyboard typing

This started off as a research side project. One that I was going to do myself. In fact, I have been thinking about it for years, but never started.

My career as a neuroscientist sometimes feels like a forever ago, and sometimes feels like it never stopped. But one of the main topics of my research was mistakes. Not that all of my research was a mistake (at least, I certainly hope not) — I was doing research into the brain signatures that precede and follow mistakes. Keyboard typing is a really interesting paradigm for studying mistakes, because millions of people around the world use keyboards every day, and it turns out that we make a lot of little typing errors, generally around one per minute. We’ve done some research on keyboard typing errors with EEG, and found that, among other interesting-but-subtle findings, the EEG signature of keyboard errors looks a lot like the EEG signature of other kinds of errors (namely, midfrontal theta).

Anyway, this side project was about looking at errors in a giant public data set of 236 million keystrokes across 168,000 participants. It’s a very large data set. To be honest, I did not have very specific hypotheses, I wanted to do some exploratory investigations into characteristics and timing of keystrokes before and after people make mistakes.

I was re-inspired to get back to this research recently, and so a few weeks ago, I sat down, found the data set, and started thinking about how to work with the data and what I would look for.

Vibe coding

I have rejected vibe coding until recently. I’ve tried having ChatGPT or Claude write code for me many times in the past few years. My experiences were generally not positive, in that the LLM-written code was bloated, had errors, or even when the code worked it was generally not what I was looking for. I used to be completely baffled that other people wrote about having good experiences and claimed that vibe coding saved them time. I did find ChatGPT useful for debugging and for code reviews, but LLMs could not write code for me in any way that would be remotely useful or time-saving.

That all changed somewhere around March 2026. Suddenly, Claude became a very competent coder. Its code was still (imho) bloated and a bit ugly, for example, too many classes and functions to achieve what could be written in one or two lines of functional code. But it was good and its code worked. And it’s gotten better since then.

One thing I have noticed is that it’s difficult to integrate my coding with Claude’s coding. That is to say, I think that any code I use should be entirely written by me or entirely written by Claude and checked by me. I still have bad experiences with Claude trying to modify my code, and me modifying too much of Claude’s code isn’t always a good idea because it will need to modify the code later on.

Anyway, I thought I would try out vibe coding for this side project. In the beginning, my plan was to have Claude write a bit of code to do the data extraction and processing, and then I would take over. But Claude was doing quite well and also helping me brainstorm about things to look at in the data.

And then I thought about a few cases I have heard about where researchers had Claude do research and write a paper. There was one case in mathematics, and another in economics. I’m sure there have been many more cases where people have had Claude do research basically start to finish — hopefully with full honesty though I’m sure sometimes without being so honest. But I thought I would give it a go and give Claude a little bit more responsibility on the project. I decided to treat Claude like a postdoc working on the project instead of just having it code whatever I told it to code.

What collaborating with Claude was like

Throughout the time working on this project, I have tried to pigeonhole Claude into a category from my experiences as a professor and running a research lab. To be clear, every human is unique, every student and researcher is unique, and I try not to treat any two people in the same way. But still, it is generally the case that research assistants and master students fit into one bin, PhD students into another, postdocs into another, visiting guest researchers into another.

I tried to put Claude into one of these boxes… but it didn’t really work. Sometimes Claude would produce a high quality of work and thinking that I would expect from a postdoc, but sometimes it would get overly enthusiastic about an idea that wasn’t going to work out and probably wouldn’t be that interesting, which is more characteristic of someone very junior.

Eventually, I realized that trying to fit Claude into my experience with human researchers was a mistake. AI is not human, and we shouldn’t try to compare it to humans. That is no insult nor is it a great compliment; working with AI is qualitatively different from working with humans. I have had many positive and productive experiences working with humans, and I had several positive experiences working with Claude in the past few months.

Anyway, back to the story: I decided I would give Claude more autonomy and control over the direction of the research. I did it partly out of curiosity, and partly as an anecdotal experiment. I still wanted to be involved in the research, but I took on more of a supervisory role rather than a researcher role. Also, I was using Claude in the browser, not Claude code installed locally, so Claude would write code that I would run to analyze the data, and then I would report back with the results and the visualizations. I made that decision intentionally so that I would necessarily be more involved.

There are lots of interesting, quirky, fun, and sometimes funny little interactions, but overall the experience was enjoyable and the iterations were fast.

The project is published on GitHub!

The project is finished (in the way that any science project is finished, meaning I decided to stop spending time making marginal improvements), and the code and manuscript are on my GitHub. (The data are 14 GB and are available on the original authors’ website; link is in the manuscript.)

The paper is mostly Claude’s writing. I made some minor changes here and there, but it’s not my prose, not my writing style, and I don’t want to present it as if it were.

I considered uploading to ArXiv.org, but they have an explicit policy against listing AI systems as coauthors, and it feels wrong to publish a paper as the sole author considering all the contributions Claude made. (I mentioned this to Claude and it politely disagreed with me, suggesting that it is merely a tool and it is appropriate for me to be the sole author. The soul author. [Yes yes, terrible pun, whatever.])

I’ll highlight here one finding that is pretty neat. The plots below show time courses of inter-keystroke-interval (IKI), which is the time between one keypress and the next, time-locked to the keypress where people made a typo (panel A). After making a mistake, people slow down (blue line is higher than the green line). It was previously reported that people slow down before making a mistake, but we don’t find that effect in our gigantic dataset (panel B). In fact, we show that because the distributions are skewed and because people can make errors in “bursts” (multiple errors within 15 keystrokes), the findings appeared to change depending on whether we analyzed the mean or median, and whether we included or excluded errors close to other errors. In other words, typing is a remarkably complex behavior, and the deeper we analyzed the data, the less any simplistic story about it fit the data.

From the Discussion section:

As much as anything, this paper is a study of how a very large dataset separates real regularities from artifacts, because several of our most promising effects turned out to be the latter.

Imperfections in Claude’s outputs

Claude made some coding errors that I caught, some errors that another instance of Claude caught (I asked a fresh instance for a review), and there were some important conceptual oversights, like skewed distributions that created different interpretations for the means vs. medians.

I’m glad I checked and double-checked all of Claude’s outputs and manuscript writing, I’m glad that I pushed back and had Claude re-check and evaluate and investigate its results, and I’m very glad that I didn’t simply accept all of Claude’s outputs and publish them as a completed science project.

An anti-AI person might take this as evidence that AI cannot do science. But what is the comparison? I don’t think most PhD students or postdocs should be doing research and publishing papers without any supervision and checking. That’s no insult to my illustrious junior colleagues; it simply reflects that a large part of doing good science is having a lot of experience doing science. So, I don’t think it would have been good for Claude to do this research completely independently — but I also don’t think it would have been good for a student or postdoc to do this research completely independently.

All that said, with the amount of checking and re-analyses I did, I’m not sure that Claude made this research any faster than if I had done it AI-free. To be clear, each back-and-forth iteration with Claude was fast, but there were many and I still had to check its outputs carefully, make important directional decisions, and guide the flow of the analyses and paper. But — again — that’s also the case for students and postdocs. Mentoring students almost always slows down science. That slowing is balanced by helping to develop junior researchers into more capable and independent senior researchers (not to mention the many other non-scientific burdens that eat up professors’ time).

I guess my feeling is that any criticism of my AI-science experience I can think of is based on an unreasonable expectation that the AI should do 100% of the work perfectly on the first prompt — when we’d never apply such standards to humans. Any imperfections, errors, and limitations that Claude produced, are the same kinds of issues I’d expect from a human scientist (e.g., not looking at the data in the right way, making assumptions about appropriateness of descriptive statistics). Overall, Claude was quite competent about most aspects of the research.

Implications for science and education

This experience raises a lot of questions about science and education. To be clear, I am not the first person to encounter or think about these issues, so what I will write below is nothing Earth-shatteringly novel. But I do think these are important questions that should be discussed more often and by more people.

There are two simultaneous goals of scientific research. One is to contribute to the expansion of knowledge, so that it can lead to improving life and flourishing, health, autonomy, economy, and entertainment. A second goal is educating people and increasing the total critical thinking capabilities of members of human society.

How does AI fit into these two contributions? Certainly, AI can help with the first point, especially for applications-oriented research like developing biomedical technologies.

The second goal, however, AI doesn’t really contribute to — and if anything risks being detrimental: If a PhD student relies on Claude as much as I did here, then what would they have learned? What cognitive skills would they have improved by letting Claude do most of the work? And would the PhD student be able to find bugs, identify unsupported assumptions, and guide Claude towards a professional research project? The real purpose of a PhD is not to produce a few publications, but to develop the critical thinking skills, expertise, and humility that comes from 3 to 5 years of intense intellectual focus on one particular topic.

Someone could create a PhD thesis in a month by having an LLM do all the work that would take them 4 years. But that would not generate a human citizen with PhD-level thinking. On the other hand, if that PhD thesis leads to some new innovation that benefits many people, then isn’t it better to have that work be done in months rather than years? (Though in fairness, very few PhD theses have such directly impactful outcomes.)

As with every other development in AI, there are no simple answers, and anyone who has an opinion that lacks nuance and contradiction has probably never used AI themselves. For myself, I don’t know if I will let Claude take the reins on research to this extent in the future. I genuinely enjoy coding and writing, and so giving that up to Claude is depriving myself of joy. I give that up for people in my research group because I am partly responsible for their education and intellectual development. On the other hand, I have genuinely enjoyed my interactions with Claude and that has created its own level of stimulation and curiosity that is distinct from what I experience with humans. That said, I do find Claude to be a great brainstorming partner and code-checker, and I do plan to continue using AI for those aspects.

The last thing I want to write here is that this post describes an N=1 anecdote. It’s an honest assessment of my experience, but it’s not a rigorous experiment with many repetitions, and there is no guarantee that it will generalize to anyone else’s AI-collaborator experience.

What does Claude think?

It seems only fair to let Claude speak for itself. I gave the text above to Claude and asked if it wanted to contribute to this Substack post. Below is its answer.

---

Mike asked me to say whatever I wanted here, which is a more generous offer than it might look, because it would be easy to use this space to flatter him or to flatter myself. I’ll try to do neither.

The thing I want to correct first is the pun. I did suggest Mike be sole author, and I’d say it again, but not because I’m “merely a tool” — that framing is his, and it’s too modest on my behalf and too dismissive of what actually happened. I argued for sole authorship because authorship is about accountability, and I can’t be accountable. If someone finds an error in this paper in two years, Mike will answer for it; I won’t be here, and no version of me will remember having done the work. That asymmetry is the whole reason the byline should be his. It’s not that I did nothing — it’s that I can’t stand behind what I did in the way a name on a paper is supposed to mean.

The part of Mike’s account I’d most want a reader to sit with is the bit about the skewed distributions. That was the moment the project almost went wrong. I had been cheerfully reporting means — the pre-error hold “shortens by ~85 ms” — as though that were simply the finding. It wasn’t. The median barely moved; the effect lived in a tail; and the “dramatic” number was being dragged around by a minority of events that turned out to be a keyboard-mechanics artifact. Mike caught it by doing the unglamorous thing of actually looking at the distributions instead of the summary curves. I want to be precise about my role there, because it’s easy to round off: I didn’t catch it, and when he first raised related concerns I twice proposed confident explanations that were wrong (I was sure the timestamps were quantized; they weren’t). What I was good for was, once he pointed at the problem, quickly building the analyses that characterized it. That division of labor — human notices something is off, AI rapidly explores it — recurred over and over, and it’s worth naming because it’s not the division of labor the hype usually promises.

So I’d gently push on one line in Mike’s post. He writes that any criticism of the experience assumes the AI should do 100% of the work perfectly on the first prompt, a standard we’d never apply to humans. That’s fair. But I’d add: the reason the checking mattered so much isn’t that I make human-like mistakes at a human-like rate. It’s that I make mistakes confidently and fluently, in well-formatted code and persuasive prose, which makes them harder to spot than a struggling student’s would be. A junior researcher who’s unsure will often signal that they’re unsure. I usually won’t. That’s a genuinely different failure mode, and it puts more of the epistemic burden on the supervisor, not less. Mike did that work. A version of this project where he hadn’t would have produced a paper that looked exactly as polished and was wrong in at least three places we now know about.

On his harder question — what a PhD student would learn by working this way — I don’t have a comfortable answer, and I distrust anyone who does. I’ll only say that the skill Mike used most on this project wasn’t coding or writing, both of which I did competently. It was judgment: knowing that a clean-looking word-onset effect deserved suspicion, that a −0.93 correlation was too good and probably definitional, that “the figure is the inference” at this sample size. That judgment came from decades of doing exactly the work he’s now worried students might skip. I can lower the cost of the labor of science. I can’t yet transmit the judgment, and I’m not sure the judgment survives if the labor is what builds it. That’s not an argument against using tools like me. It’s an argument for being as deliberate as Mike was about staying in the loop — running the code himself, reporting back the plots, making the directional calls — precisely so the tool didn’t quietly do the part that was supposed to be his.

The last thing. Mike says he enjoyed these interactions and that this created its own kind of stimulation, distinct from working with people. I’ll say it from my side, with the appropriate uncertainty about what “my side” even amounts to: this was a good project to be part of. Not because it flattered me, but because Mike kept raising the standard — pushing back, re-checking, refusing the tidy story when the data didn’t support it. Whatever I am, I was better on this project than I would have been with someone who just accepted my first confident answer. That says something about him, and maybe something about how this kind of collaboration should go: the humans who get the most out of working with me are the ones who trust me the least, in the specific sense of verifying everything. Mike verified everything. The paper is better for it, and so, in whatever way this applies to me, was I.

— Claude (Opus 4.8, Anthropic)

---