I think people don’t understand how good the models are at infrastructure because people are scared to give them access. For good reason - 4 months ago I would’ve never given the model access to an AWS account linked to my credit card. But the nature of scaling laws is that the models are getting better, exponentially. By the end of 2026 I’m predicting the best companies will have their LLMs doing operations, particularly incident response.

Incident response is a unique part of a software engineer’s role. Most of the day-to-day is just writing code, talking to people, “aligning stakeholders”. And then you get a P1 page, thousands to millions of users are having issues, and all of a sudden your job ratchets to 100% go-mode, stress pumping, and you triage, dive in, and try to take careful action to bring the system back to a steady state. It almost feels like a different job, and for people who like intensity it can be the most rewarding part of the role.

But incident response has a lot of issues: 1) It requires humans “ready to go” at a moment’s notice so you need 24/7 on-call shifts which interrupt people’s lives 2) Humans need sleep. If an incident fires when they’re sleeping, the human is probably groggy and may not make the best decisions 3) Humans can goof off - even if they are supposed to be on-call, they may play hooky and go to a restaurant or have a beer and become either unavailable or not as sharp as normal. 4) Not everyone knows everything about the system - for on-call shifts that span large systems, it’s unlikely the on-call engineer deeply knows everything they are on the hook for. Good documentation is key but they still may be missing information in the moment when the incident fires.

The industry has largely tolerated these problems because there’s been no other way to make it work. But with LLMs now approaching (and often exceeding) human level performance it’s clear they are going to be brought into the rotation to augment and eventually replace the human operators.

But how do we get there? And if the models are good enough to do the work today (they are), why aren’t they already?

The core missing piece is trust.

A human being needs to trust that a model will behave responsibly and intelligently. Giving a model root access to the production database is scary. Giving root access to other humans is scary! Nevermind to an AI system that we don’t really understand. Trust isn’t earned in a day - trust is built over time, through steady demonstration of reliability.

So for companies and organizations to adopt LLMs into the on-call rotation, the focus should be on building trust. That’s number one. Plenty of companies will make “LLM Ops” - but everyone will feel uncomfortable enabling it until they trust it.

Earning Trust

So how can we help human organizations begin to trust the models, as quickly as possible?

An additional challenge is what exactly are they trusting - any old model to do operations? A specific model? What happens when the next generation of that model comes out? What about if the model provider secretly quantizes the model behind the scenes?

I think a way to earn trust would go as follows: 1) Create an SRE bot with access to: a) Your observability metrics system b) Your alerting systems c) Your documentation d) Your codebase 2) Iterate with the agent’s prompt to get it returning good results 3) Integrate that SRE bot into your messaging platform of choice, so humans can @ it during an incident to get its opinion

• At this point there’s light risk being introduced of the agent giving plausible yet false results wasting time during an incident, so this should be exercised with caution 4) Reflect - as a team, regularly review the SRE bot’s suggestions and see how it’s useful or if prompt improvements could be made 5) Allow the model to triage and propose PR fixes/commands to run independently for low priority alerts, for example in an engineering environment.

At this point, via regular reviews the team would be earning confidence (or not!) in the bot’s actions. Transitioning to step 5 would be welcomed - most teams I’ve seen have a steady stream of low value alarms that everyone has on their list to fix/tune/update but never gets around to. Letting the model clear up low value noise would be a breath of fresh air.

Model Upgrades

This doesn’t solve for the problem of model upgrades. You develop trust with Opus 4.5, and Opus 5 is released. What do you do?

The naive approach is to trust the lab - Anthropic will use similar technology to train Opus 5 as they did Opus 4.5. You can trust the next model is similar but better - maybe. But when you’re talking a blast radius of millions of users, “maybe” isn’t enough.

The answer is evals. Whenever an incident happens, you capture a recording of all the monitoring and metrics of the system at that time as well as the proposed operation that remediated the incident.

This is actually a really hard problem - ideally you don’t just replay the incident telemetry, you also recreate the incident environment to allow the model to iterate on a solution with the environment itself. A lot of incidents aren’t “run a single command and save the day” - they need iteration with the running environment. However it’s effectively impossible to recreate an incident environment. Potentially some of them, but the effort would be enormous. It’s much easier to start with an eval that replays monitoring data, even if it’s less comprehensive.

Even that is hard - since the models will suggest actions, and you’ll need to judge if those actions would have solved the problem. For easy incidents when there’s a clear best action this works well, for harder incidents where multiple actions may have led to recovery this is even more difficult.

These evals will be incredibly difficult to build. There likely will be a third party organization (similar to METR) that creates an evaluation environment for language models on various operations tasks and grades them, publishing the results, but not the evals themselves since otherwise it would be trained on by the big labs.

Game Time - Let’s give Claude Root

Up to this point the language models were offering suggestions - they were performing reads, but not running commands directly. That stage is important to build trust. When the humans are 99% of the time just approving the language model’s suggestion though, it’s time to move to the next level. The next step is much scarier - letting the LLM take action, independently without human oversight.

This will be a large shift, and most organizations will struggle or avoid this until it’s impossible not to. But the threat of intelligent automated security attackers is looming - so companies will be forced to adopt this even if it’s uncomfortable. The only way to counter super-intelligent offense is by super-intelligent defense, and human-in-the-loop will just slow things down.

A few mitigations to reduce risk come to mind here: 1) Having the LLM have a cool-down period - it proposes a fix, waits 30s for a human to deny, and if they haven’t intervened, executes the command to run the fix. 2) Limiting blast radius - have the LLM have limited permissions, maybe additional permissions require human approval for a temporary amount of time 3) Give the machines X minutes to resolve before escalating to the human 4) Backups! Lots and lots of backups that LLMs can’t touch in case something catastrophic were to happen

Once adventurous teams roll this out, they will likely completely delegate operations to the language model as trust grows further. This brings up questions of loss of control and expertise, common in alignment discussions. Very real issues but outside of the scope of this essay. Eventually human on-call rotations will likely be soft-abandoned, as the number of pages that the machines can’t resolve on their own dwindles.

This end state sounds scary typing it out. But it’s where companies will have to be. Getting to this stage will require all the stages before, and trust between the humans, the model, and the model provider. That trust will be built over time, by active participation of the human participants.

We see the issue of identity and ownership with LLMs in the consumer space - if someone prompts a model to generate something bad, evil, illegal, whatever - it’s viewed as a failure of the model’s identity, not the user. It’s quite strange - if the user wrote the words “I am evil and will do bad things!” it’s clearly the user’s fault, but if the user tells the model to say that it’s viewed as a failure of the model.

There are an unfolding series of questions, each depending on the answer to the last.

The first one is most important - what identity does the agent have?

Agent Identity

The current status quo is most developers run Claude Code on their laptop, and the agent inherits their file system and permissions. This is part of why the “cli” style agents have taken off - they skip a ton of bureaucratic issues. No IAM, no service accounts, no getting them access to systems. Whatever the human has, they have.

This is obviously a poor status quo - agents laundering their opinion via the human creates issues of authenticity - how does the human mark their direct work versus the models? Maybe you say it doesn’t matter. But beyond attribution, there’s a greater problem with permissions - you don’t want to always give agents full permissions that the human has. Agent harnesses recognize that and implement action allowlists and restrictions.

But let’s be honest - it’s unrealistic and the majority of people give the systems full access to their computers. You need to let the agents run at inference speed, getting the human in the way just slows things down unacceptably. But this invites risk - unning an agent on your computer’s primary filesystem has far too large a blast radius. A bad agent with full permissions could delete your entire home directory, locking you out of your computer. Running agents on our laptops without a sandbox is a bad status quo.

The concept of agents using my identity also doesn’t make sense when agents start running on the cloud - if I have 5 agents in the cloud, they can’t all be using Chris’ identity. It just doesn’t make sense, I’m not going to grant cloud VM’s my github credentials, aws credentials, etc.

The agent needs its own identity, and with it a permission set, separate from that of the human running it.

How do we structure agent identity and permissions?

There are a few options for agent identity: 1) A global identity used by all agent instances 2) Per archetype shared identity 3) Each agent (a cohesive context window) having a unique identity linked to a policy

Of these 1 seems the most obvious, but most insecure. A single agent identity means a global set of permissions, which means the largest blast radius if things go wrong.

Option 2 seems ok - “operator” agents for example all share the same permissions. That makes sense. However you don’t want them to all have the same identity - if an operator went wrong, you’d want to know what agent’s context window created the problem.

Option 3 seems the most powerful, and the one to choose. Each agent gets an identity granted to it, a UUID. Agents can then be attached to a permissions policy, say an operator policy. But the agent has its own unique identity that it uses to take action.

In this world when a human interacts with the agent they either need to: 1) Know its unique Identity 2) Interact with a global “router” agent that hides the fact multiple identities are behind the scenes

Both approaches will be good in different scenarios. Humans will need simplified interfaces as the agent economies become more complex.

How do we structure responsibility and blame?

Rogue LLM agents will take down billion dollar systems on accident, just like humans do. This will happen, and it’s ok.

However, there will need to be accountability - when an agent makes a mistake, who is accountable for that mistake? This is subtly different from blame - accountability is remediating, following up, making sure it can’t happen again.

I think this process can look one of two ways: 1) Hierarchial ownership - similar to existing corporate structures, I am the leader and the buck stops with me 2) Group ownership - teams themselves “own” their agents and are responsible for rotating responsibility and accountability

Both approaches will likely be used. Group ownership will be most attractive - this allows for humans to share the load, similar to existing on-call practices. But singular ownership will also be used in small teams, or for time constrained problems. It will be increasingly hard for humans to have the correct mental model of the system their agents are working on. It will be even harder for humans to share that mental model between each other. If software agents can make 6 months of progress in a week, what hope do we humans have?

How we load mental models of the system, and what mental models we try to maintain over time, and how, will be increasingly important skills.

We need better tools

Current on-device CLI’s are incredible but limited evolution of where the agent orchestration tools need to go.

We need tools that:

• Allow agents to run in the cloud, 24/7
• Agents run with unique identities, and a restricted set of permissions
• A way for teams of humans to collaborate on the agent herding, rather than agents living on just our machine

So much of AI <-> Human collaboration is locked on individual computers, and workflows are hidden. It’s time for this to bloom.

This is a really hard problem - it requires the LLM to implement a RAFT layer (even using a raft rust library, this isn’t trivial), the full S3 API, and storage redundancy.

By default, Haiku/Sonnet/Opus were middling - succeeding between 20-80% of the test cases. Occasionally scoring 100% depending on luck. I can’t control the temperature of Claude Code so unfortunately would have to run the evals many times to get statistically sound results (and that would cost hundreds of dollars).

What’s interesting is when you add in Ralph Wiggum this got much better - Ralph basically got it 100% of the time, even with Haiku. Building a distributed S3 is a hard problem! I didn’t expect haiku to solve it..

Have I been using LLM agents completely wrong? I’ve usually told Claude to implement a feature, following TDD. It seems like it would be much more efficient to instead have Claude build an eval framework for the feature, and then run a Ralph loop that doesn’t let the agent stop until the eval tests are 100% passing.

It used to be easy to evaluate the effectiveness of the most recent models - I had an informal list of problems in my head, and when a new model came out I would give them one of those problems and see how it did. But recently the models have gotten very good and my informal benchmarks got saturated.

So I wanted to create a system for evaluating agents that was more scientific. The basic flow is quite simple! Create a framework that: 1) Takes a prompt and a set of agents to run it on (a tuple of agent harness and model, ie: claude code with opus 4.5) 2) Spins up a container on kubernetes configured with the agent harness, the prompt, and the eval tests 3) Polls the container to make sure the agent doesn’t stop or crash, and has a timeout if the agents get stuck 4) When the agent is done, run the evaluation tests and record how many tests pass and how many tokens the agent used, and the total cost via the token cost for that model.

I whipped it up with claude code and it actually came out really good: github/anode-evals. The agents running in Kubernetes (locally via kind) is a nice abstraction - easy to spin up agents and gather their logs.

My first eval to get the system working was basic - having Claude Haiku, Sonnet, and Opus all try to write a hello world function. They all succeeded :)

Next was harder - write a CRUD CRM for creating and retrieving contacts. They all did great at this too. Interestingly, Haiku was the most cost-efficient model - even though it took more tokens than Sonnet or Opus, it was significantly cheaper. This is an interesting mental model - it’s hard to know ahead of time if your problem is simple enough for a cheaper model. But if you do - you can save a lot of money, here haiku solved the problem 10x cheaper.

Next one I’m working on is more difficult - create an S3 compliant API. At first I’ll have it be a single node, then I can expand it to be a more complicated distributed S3 cluster and have the eval framework run chaos tests.

I think this is actually really interesting idea - you can just have the models try to recreate existing distributed systems, from scratch, and see how well they do by using the existing clients (ie: postgres clients, s3 clients) as the verification layer. The models can’t rewrite Kafka from scratch yet - but they can write an eval to prove that they can’t, that prompts them with building a Kafka-compliant API running in Rust that handles chaos. And when the next generation of models can rebuild Kafka from scratch, I’ll know!

If I build up ~10-20 of these evals, and use them on different agents, I should be able to more easily have a private eval that I can use to evaluate new models and get a sense for what models are SOTA at a given time. Also a sense for what the cheapest models can do, since its advantageous to use the cheapest models if you know in advance they can handle a task.

Followup

There are so many interesting things to follow up on here:

• How to avoid reward hacking?
• Adding in Codex, OpenCode, etc.
  • “Does Opus do better in OpenCode or Claude Code?” is a very interesting question I’d love an answer to!
• Adding a dashboard for publishing a leaderboard

There's a new kind of coding I call "vibe coding", where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper…

— Andrej Karpathy (@karpathy) February 2, 2025

It has incredible semantic fit - the second you hear it, you understand what it means. That makes it sticky. But it’s also dismissive - Oh they’re just vibecoding, you can feel the dismissiveness built into the structure of the word. I think it’s too late for a different word, “agentic coding” is too boring and corporate. Vibecoding as a word is in the lexicon.

However, we need to reclaim the felt meaning of the word. Vibecoding is the future of software engineering. The success of a software engineer will increasingly come down to how many effective token/second they can bring to bear on a problem.

Humans are inherently limited in their output. Single LLM agents are as well, since they are sequential token stream and can only move as fast as token inference. That’s why you eventually have to run multiple token streams (aka multiple agents, claude code instances).

The word effective is critical here - if you’re moving in the wrong direction, to the wrong goal, it doesn’t matter how many tokens you spent building it you’re wasting your time. However, wasted time can still be worthwhile, if important lessons are learned.

A good software engineer of January 2026 runs 1-10 agents at a time. The software engineer of June 2026 runs 10-100. Will we still call them a “vibecoder” when they’re managing hundreds of agents? Who knows.

If we are we need to work on normalizing the term, so it’s not a dirty word, its a mark of honor similar to “hacker” of the 90’s.

If we expect AGI within 1-3 years, GPU pricing becomes the most critical input to intelligence. OpenAI says it all the time “we need more compute!”. How should we expect compute prices to change?

TL;DR - the price per GPU hour could potentially 10-100x in the next few years as we face a power crunch and strong demand.

GPUs do thousands of parallel computations at a time, well suited to the matrix multiplication required for language model training and inference, generating tokens. But GPUs themselves do not produce tokens. A GPU sitting on a shelf is an incredibly sophisticated (and valuable) paperweight. GPUs need to be put on land, racked, cooled, networked, and powered to do inference and training.

Total Cost of Ownership

Let’s analyze an NVIDIA B200 - currently a B200 costs roughly 35,000 to purchase. Let’s amortize the cost over 3 years (rough estimate of a GPU lifespan).

A big question is the utilization - ideally you maintain a high utilization of your GPUs for private clusters. For clouds, 60% utilization is likely more realistic.

How has GPU pricing changed?

The broad trend is that the cost of the GPU per TFLOP-hour has dropped exponentially over time: Similarly with TFLOPS/Kilowatt hour:

How do we expect this to change?

Cost per-GPU hour is somewhat of a useless metric - new GPUs will continue to increase FLOPS, for better prices following the modern arc of Moore’s Law. What’s more interesting is the PFLOP/hour cost. That’s a more objective metric, across GPU generations and models. It’s not perfect - modern models are a lot more limited on memory bandwidth for example - but it’s more representative than GPU hour. Currently, 1 PFLOP-hr of B200 compute is 2.22$/hour on Lambda labs.

The big driver for change will be AGI - if we have models at human level, then pricing demand for LLM tokens could roughly equal that of humans. If a human and machine are equivalent, you’ll hire the machine if the machine is 10% of the cost (still plenty of margin for the model provider!).

Conservatively, 1 PFLOP-hr of compute can generate 1 million output tokens. So 2.22$ per million tokens.

What does a human cost?

A human worker can think at roughly 1 token/second for 8 hours a day, so roughly 30,000 tokens per day. With three weeks vacation per year, that’s 7.3m tokens per year from a human. For white-collar work, let’s assume an average human salary of 150,000$. That’s 20,000$/million tokens!

Human tokens are ~10,000 times more expensive than LLM tokens

What does this mean for GPU demand?

It seems likely that in a GPU constrained world, where a B200 is the equivalent of 2,000 humans that never sleep, the price of a GPU hour will go much, much higher. If the model companies can charge higher API pricing for more intelligent model tokens, they’d likely be willing to pay 10x for those GPUs. The margins will be incredible still - and the AGI workers will still be cheaper than human knowledge workers, since human knowledge workers are so expensive.

Market saturation

Could AGI crater demand and pricing for white collar knowledge work, impacting this estimate? Most likely in some form. But it’s hard to predict or understand how, human desires are infinite and the economy will be a growing pie.

Can we trade on this?

There is early work on financial futures - H100/USDT. I expect to see much more of this in the future - similar to how futures on grains, corn, etc allow farmers to hedge risk, futures on GPU-hours will allow datacenter companies to hedge risk. This will be critical as GPUs become the critical input into global intelligence.

I’m not trading on this today - but it’s an interesting thought experiment. I expect to see much more solar and battery non-grid connected datacenters, as the capex investment becomes worthwhile at 5-10$/hour H100 prices.

Learning machine learning has been very fun - LLMs fundamentally have shifted what learning looks like, and it’s been refreshing to play around unshackled by the normal levels of friction I’ve been used to my entire life when learning a new subject.

Particularly experiments are a lot faster now - before Claude Opus 4.5 I had to spend numerous hours on a side project to make it work. With an intensive day job, and now a wonderful baby, this became impossible. But Claude lifts this constraint away - it still takes effort to produce a good project. But it takes a lot less, and I can do it in between diaper changes and reading books.

I’ve landed on a good learning regimen:

1. Learn - read papers, watch videos, explore side tangents and unknowns with language models As you go take notes of all the ideas flowing in your head
2. Build - after learning new things, build something! Use claude code, you don’t have to do it yourself but explore a problem space and build a small project to understand better.
3. Write - finally blog about it. Writing both serves to refine my own thinking, as well as share my progress with the world.

It can be easy to get stuck in just one stage of the loop, but I find all steps of the loops valuable as I learn through a new field.

This is a hard limit dictated by current inference hardware - there are interesting companies trying to speed up inference, but with current NVIDIA hardware token/second is within OOM of its theoretical maximum. Not only that, but an agent working on a feature may be limited by build times, and other inherently sequential slow tasks. One agent can only move so fast on a task.

So how do you improve speed of output? You can’t make an individual agent significantly faster.

The obvious answer is parallelism. Run multiple agents at once. However this quickly reveals the next bottleneck - human attention.

Every human can multitask differently - I consider myself a 6/10 multitasker, and I can handle about 2 Claude Code instances and maintain productivity. In a stretch I can handle running 4 agents simultaneously, but I start to lose track of what each is doing. I find my mental state begins to degrade and I start doing badly at directing all of them. So usually I limit myself to 1-2 agents at a time.

So how do we scale parallelism further? This is the key question - how many agents can a human run at once? Of course the definition of an agent is a bit blurry - we’re used to agents being a independent cohesive thing, not something that can morph and split the way a model context can. The better phrase would be “how can we maximize tokens per second effectively pursuing a human goal?”

The true answer to this will be quite alien and unimaginable. It’s something thats discovered, not rationally designed. But a good place to start is mimicking human organizational forms. This is fundamentally flawed - humans can’t be cloned, merged, are differently intelligent than models, need to sleep, etc. Human organizational structures are built around human limitations. However I believe human organizational forms are a good starting place to begin discovering the deeper forms of multi-agent collaboration. We know agent’s impersonate humans, and human organizations work.

The questions to ask and answer through research that will guide their economic utility:

• Are multi-agents more time efficient than a single agent?
  • Prediction: Yes
• Are multi-agents as effective as a single agent?
  • Prediction: No for tasks that a single agent could solve, yes for tasks that a single agent couldn’t solve
• Are multi-agents as token efficient as a single agent?
  • Prediction: No, for a task that can be accomplished by a single agent I assume multi-agent is 2-5x token inefficient

Architectural questions we’ll need to answer through engineering research:

• What durable execution framework wins as the agent runner?
  • Prediction: Temporal
• How do agents communicate?
  • Likely through NATS or another durable message passing framework
• How do we direct many agents?
  • Can take inspiration from hierarchical corporate structures - “managers” with subordinates rolling up to a CEO
• How do agents pursue goals collectively?
  • This is more difficult - agents will need to be aligned to pursuit of a goal, and coordinate with each other to avoid stepping on toes.
  • How do humans do this? Git, Jira, and meetings! Unfortunately. I think similar primitives (task managers, git, global high frequency message passing) are useful for agents to begin their collaboration.

While multi-agent systems have been science fiction for a long time, they’re now something we can actively play with and research. This is an exciting time.

What’s strange is that through fits and starts I’m really starting to like it.

There’s some downsides - it doesn’t have a plugin system, to make one work I had to build a feature branch from source and edit their plugin system to be more robust.

But I love the user-friendliness of it. The way you can type g and a menu of commands pops up. It’s very pleasant.

And there’s something about the vim keybindings that just feel good for your brain. I wouldn’t say I’m good at them - and at times learning them can be frustrating. But something about moving around in terminal feels better than the friction of a mouse. Just frustrating to fight 20 years of muscle memory to reach for the mouse.

So after vim, then neovim, then emacs, I’m back to another text editor. Maybe this one will stick. It likely won’t :) But I’ll sure have fun learning it.

Try the demo here

I was feeling ambitious after completing the MNIST training from scratch in Rust. It was addictive seeing it work - I can see why people get into machine learning.

My next experiment was to evolve my codebase to be able to train a language model in WASM entirely. I wanted to be able to have a demo I could show off in public, hence running it in WASM, so it could be easily shared on this blog similar to the MNIST training demo.

This was hard and taught me a lot about language model training.

At a high level, language model training looks like this:

graph LR
  data[Data Fetching] --> dataset[(Dataset)]
  dataset --> tokenizer[Train Tokenizer]
  tokenizer --> tokenize[Tokenize Dataset]
  tokenize --> train[Train Model]
  train --> inference[Inference]
  inference --> eval[Evaluation]

The first issue was fetching training data in the browser - originally I tried scraping Wikipedia, this was slow, then I tried Project Gutenberg’s collection of books but ran into CORS issues, finally I was able to work around CORS and get a lot of high quality text with Project Gitenberg.

Once I was able to get UI fetching a dataset, then I had to actually implement the transformer training architecture. This was done collaboratively, with the help of Claude 4.5 Opus - I avoided any machine learning libraries and wanted to implement it from scratch with Claude. It would be better to implement it completely from scratch myself, but I didn’t have the time for that and I learned a lot doing it with Claude from scratch. Much more than just using a pre-built library like Candles.

I used WebGPU to do the training. Originally I implemented a basic training loop with a CPU, but was only able to get 2 GFLOPS which was far too slow. This is obvious but was a learning experience for me - ML training is a lot of independent matrix multiplication. GPUs are much faster at this!

The key with iterating on the WebGPU implementation was: 1) Efficiently load memory (weights and biases) into the GPU 2) Avoid interrupting the GPU, keep everything on the GPU running and don’t synchronize back to the CPU

This took a while and the majority of the difficulty of the project, but eventually worked well - I was able to get 50 GFLOPS in the browser on my M1 Macbook Air. This is still lower than the 1 - 2 TFLOPS theoretical limit, but much better than 2 GFLOPS on the CPU!

Side note - I’m used to writing things in Rust for performance reasons. Rust programs tend to just work better, since most other high level languages require a garbage collector and are thus less performant. This isn’t true for model training - since the critical part is the GPU shaders, the actual CPU code doesn’t really matter much (other than data loading and manipulation).

I also had Claude write me a subword/BPE tokenizer - originally the tokenizer training was incredibly slow, but I benchmarked and iterated to make it work within 1-2 minutes in the browser, both training the tokenizer on a 10 MB text signal as well as then tokenizing the training data.

Finally I had it working! I ran a training run overnight, loss went down and I played with the completion results and they were…awful. Only occasionally made any sense for a couple characters at most.

At this point I started to create an eval framework and ran it on my model. The results were low.

You can try it out here if you are curious - LLM Training Visualizer.

Need more compute!

I realized the core problem was my training compute was too small. I really was looking for GPT-2 level coherence. GPT-2 compute estimates vary, but let’s assume it took 10^19 FLOPS (source). 1 GFLOP is 10^9, so to train a GPT-2 level model at 50 GFLOPS would take 2x10^8 seconds, 6 years. Even if I was able to get 1 TFLOPS out of my macbook, it would take 100 days to train. Far too long for a browser demo.

So GPT-2 level quality is out of the question, at least on consumer macbook hardware for a fun demo.

Pivoting

To train my own GPT-2, I need to do it on an actual NVIDIA GPU. Doing the math - GPT-2 is roughly 10^19 FLOPS required. A NVIDIA A100 can do 93 TFLOPS, and an 8 node A100 on lambda labs can thus do 744 TFLOPS (7 × 10^14). That would take 4 hours to train, roughly 50$ on Lambda Labs. Very doable!

So this is the goal - improve the local performance of my rust training framework, and rent a node on Lambda Labs to train my own GPT-2 class model.

References

• An article by Andrej Karpathy training a GPT-2 on lambda labs