What does good look like?
How hard could it be? A story about running your own AI models. Episode 2.
After the project got halted on Monday, I spoke with Juan the next day and asked if we could test with some of the open source models hosted on AWS, and he agreed that if we kept the data within our own accounts we could make use of it. He only asked me to tell him which models we use, to be safe. I made some changes to my initial code and had a running report by end of day. It passed Juan’s check. I sent the new report to Fran and kept working on some of my other team duties.
That was two days ago. This morning I have an early meeting with Fran and Pawel, who’s starting today. I’m excited to see what sort of fresh ideas he can bring in. I don’t know if I’ll make it in time though. I’m stuck in traffic. It’s bad today but not totally horrible, just normal bad. I rarely take the car to work, but I have some errands to do later and I need it.
Music is the only thing that makes it bearable for me, the Altın Gün radio is playing on Spotify, I like this band. I think I’d go crazy without music in this traffic. I can’t even imagine listening to the news on the radio and getting even more nervous. No way, music is fine for me, thank you.
The company I work for is near Las Rozas, in an industrial zone with many big office buildings in the outskirts of Madrid, what we call a “polígono”. I find a parking spot not far from the office building and I arrive 5 minutes before time. I was sweating it for a while thinking I’d be late but not today. Excellent! I think. You have to take the wins whenever you can.
In the meeting, we make a round of introductions. Pawel is on Erasmus and he’ll be with us for a semester. He’ll be doing AI research for us as part of his degree. Fran explains to me that HR is being very strict about the type of work he can do. “Commercial” work is not allowed, he can do “research” but nothing that goes to production is how I understand it. Some new law thing apparently. Ok, we’ll see what we can do, I tell him.
He then starts explaining to Pawel some of his department’s problems, and some of the issues with our current report. I tell Pawel about the POC I had built. He suggests we train a custom encoder model. Unlike generative models, it doesn’t write anything, he explains. It just reads a ticket and picks a category. It’s much smaller, so it’s cheap to run and to train. “It’s part of my studies and I think it could be a good use case for us,” he adds.
Since I don’t know if he can have access to our customer data (we’ll need to ask Juan about this), we decide to start with some synthetic data to validate the idea. We agree that I’ll share the schema we use for our records in the system, so he can start right away. We’re finished with 10 minutes to spare. I wish most meetings were like this. I like Pawel’s attitude, so eager to do things. I find it energizing.
The next day, he shows up at my desk, looking like he hasn’t slept much.
“Hi Ricardo, sorry to interrupt, I wanted to show you some results I have.”
“Already? Cool, Pawel, tell me.”
“So my model is working well. It scores 0.853 vs the AWS model which did 0.831.”
And he doesn’t say anything else… So I look at him thinking “wtf are you talking about?”, and reply.
“Ok but 0.8whatever out of what? What does that mean?”
“Oh sorry, yes so it’s an F1 score.”
“Ok, so?”
“Yes, yes, right. So it’s a number that comes from two other numbers. First we have Recall which is like, of all the tickets that were really serious, how many we caught. And then you have Precision, so like, of all the tickets we flagged as serious, how many really were. And F1 combines the two, and it drops if either one is bad.”
“Ok, so, low recall means we miss issues from customers who can’t pay, for example, and that costs us money. Low precision means Fran’s team wastes time on false alarms. And I guess we can’t have both, right?”
“Well, that’s the challenge we need to solve. I’ll send you later some instructions I made. Ok, bye.”
“Ok, later.”
And then he goes running back to where he came from.
Five minutes later, he shares a folder with the model’s weights and his instructions. I try to reproduce his results and get things running. I get very similar numbers. The next step is to test this with our real data.
I modify the report code to run Pawel’s model on a few months of historical tickets, then compare the results to the categories on record. First it crashes on tickets without a category. I skip those and run it again. The score is bad, F1 is 0.61. I look at the tickets where the model and the record disagree and sometimes the model flags a ticket as serious when it clearly isn’t. Other times the model looks right, and it’s our category that’s wrong. We’ll need Fran’s team to clean this up and label things properly.
I share my findings with Pawel on Slack, and he seems a bit upset we are stuck.
“Look, don’t be discouraged,” I tell him, “you made some really good progress in one day, you made an experiment that I could replicate, and defined how to evaluate the performance of the model. The real world is messy, you know. We need to keep iterating. On Monday we’ll have our meeting and discuss next steps.”
“Ok, ok, I have some ideas, we could try, I’ll think about it for our next meeting.”
“Good, just don’t stress over it. We’ll figure things out. Have a good weekend, see you on Monday!”
All the characters and companies are fictional. Inspired by real experiences.