It works on my Mac

How hard could it be? A story about running your own AI models. Episode 4.

Fran wants Pawel’s model in the daily report as soon as possible. Right now the report runs on my laptop. If I’m sick or on holiday, there’s no report.

How do I deploy this code to a Linux server? I haven’t dug much into the code, since Pawel was handling that, but now I need to understand what’s going on at a deeper level. In the old days, I would have started searching and reading blogs, tutorials, or even Stack Overflow. That’s gone. Now I just start a session with an agent and figure it out.

After researching some options, I decide: I’ll start simple, understand what I’m doing, then automate it all. Fran wants it soon, but if I skip the learning now, I’ll pay for it every time something breaks. I’ll launch a g4dn.xlarge, the smallest EC2 instance with an NVIDIA GPU I could find, 16 GB. For the models we have, it should be enough. I clone the repo on the server and try to get the evals running.

The first surprise comes before I install anything. Pawel built everything on his Mac, with what he had there: MLX, Apple’s framework for machine learning. On a Mac, Metal does what CUDA does on NVIDIA cards, and MLX does what PyTorch does. It works great on his laptop. It worked on mine, but it was just never meant for a server.

Everything I read about running models on NVIDIA uses PyTorch, and ModernBERT, the model Pawel started from, is already supported by the transformers library, phew! With the agent, I rewrite the eval code in PyTorch. It’s not much code. Pawel kept it simple.

I try to install all the Python dependencies and get some cryptic errors. It seems I need to install the NVIDIA drivers first. The server doesn’t even see the GPU yet. OK, I probably should have started with one of the starter AMIs, but I didn’t, and besides, usually those come with so much stuff you probably don’t even need. Once the drivers are installed, there are a ton more errors, but with the agent and some back and forth, passing errors, we manage to get it all installed. It’s a bit of a mess. We need a specific version of PyTorch that is compatible with our specific NVIDIA driver, and the project is using the latest version of everything. So there’s a lot of trial and error until we get the right combination.

Then the code runs, and the results make no sense. Above them, there’s a table I almost scroll past:

ModernBertForSequenceClassification LOAD REPORT from: ./model
Key                                    | Status     |
---------------------------------------+------------+
bert.layers.{0...21}.attn.Wqkv.weight  | UNEXPECTED |
bert.embeddings.tok_embeddings.weight  | UNEXPECTED |
cls_head.weight                        | UNEXPECTED |
...
model.layers.{0...21}.attn.Wqkv.weight | MISSING    |
model.embeddings.tok_embeddings.weight | MISSING    |
classifier.weight                      | MISSING    |
...
- MISSING:  those params were newly initialized because missing from the checkpoint.

Everything in the file is unexpected, and everything the model needs is missing. This is where I finally understand what the weights are. A model is code plus weights. The weights are millions of numbers saved in a file, with a name for each group of numbers. Pawel’s file has the right numbers, but with the names his MLX code gave them. The PyTorch code looks for its own names, doesn’t find them, and fills those parts with random numbers. It doesn’t fail, it warns you and keeps going, which then turns into you pulling your hair out not understanding what’s going on. Luckily I saw it. I guess some old experience reading a bazillion logs is worth something.

So I find a script that transforms the model weights and I run the evals again on the 500 old tickets: 71 caught, 11 missed, 30 false alarms. Not exactly what we had, but very close. Good enough for now, but we need to start doing the training on Linux too, so we’re sure.

I ask the agent to make an install script with everything we did. I delete the instance and start a new one from scratch. There are a few minor issues, which get ironed out, and now we have a reproducible build. I shut down the instance. These GPU servers are expensive, and having one idle most of the time makes no sense for a cron job. I’ll have to think about it.

I start making a sort of mind map of what I’ve got so far in my notebook. I like the feeling of paper, there’s something gratifying about it, like paper books. I’ve tried e-books and they’re convenient, but the feeling for me is just not there.

My notebook: the Mac and Linux stacks side by side

Then I figure out how to run the cron job cheaply. AWS starts the server at 3am, it runs the report and shuts itself down when it’s done, or after two hours if something gets stuck. One hour of GPU a day instead of twenty-four.

I spend the rest of the week juggling some of the team’s work for the k8s migration and turning my POC into proper code: Terraform for the server and the schedule. On Thursday I manage to deploy everything so I can check the full run results on Friday morning.

It works. Fran receives the email. I was expecting something to break on the first run, but I guess we’re lucky. I make a plan for the upcoming week. Automate the model training on our infrastructure, and include some automatic performance evaluation. I’ll need to discuss this with Pawel, so he can keep running his experiments.

I attend a few meetings later that day. We have this no-meeting Fridays policy, but people seem to think it only applies to recurring meetings, so you end up getting invited to all sorts of one-off meetings, since people actually have some free time in the calendar. Before I realize it, it’s 17:00 and the office is half empty. I close my laptop and put it in the bag. I’m going out for beers tonight with some old friends. They’re probably already at the bar.


All the characters and companies are fictional. Inspired by real experiences.