Real data
How hard could it be? A story about running your own AI models. Episode 3.
Monday morning, 9am. I just got to the office. I go get myself a coffee and see Carlos having one in the kitchen.
“Hey Ricardo, how was your weekend?”
“It was good, thanks. I went to visit my in-laws in Guadalix de la Sierra. I love being in the mountains. What about you?”
“Not much. So… how’s the project with Fran going?”
“It’s good, we started running a daily report, it has room for improvement of course, but Pawel is investigating things and running some experiments.”
“Who did you say?”
“Pawel, the new intern. Have you met him?”
“Not yet.”
“We’re actually having a meeting in an hour, do you want to join us?”
“No thanks, I have enough meetings. Keep me posted. I have to go.”
Back at my desk I finish my coffee, while I check my emails and listen to some reggae. Cultura Profética radio is playing. When it’s time, I go to the meeting room.
During the meeting I bring Fran up to date. I tell him we need to get better data to properly evaluate the model. He doesn’t want to spend too much time on it, but he understands it’s important. I mention we need to check with Juan if we can give Pawel access to our data. Pawel says he can help with data collection. Fran agrees it’s a good idea, and pairs him with Marta, one of the most senior people on his team, so she can explain the business domain to him. I propose we create a new project board to make it easier to track the work. Nobody is against it. We go over some of the details, and add a few issues to it.
Afterwards, I send Juan an email.
To: [email protected]
CC: [email protected]
From: [email protected]
Subject: Data Access for PawelHi Juan,
I wanted to ask if we can give Pawel access to the tickets in the system to train some models. He doesn’t need access to the DB, I can prepare an export and share it with him.
Let me know if you have any questions.
Best,
Ricardo
A couple of hours later I get a reply:
To: [email protected]
CC: [email protected]
From: [email protected]
Subject: Re: Data Access for PawelHi Ricardo,
Listen, I checked with Caroline from HR. She said his work needs to be related to his research project and releasable to the public. He signed the NDA and he’s bound to the same policies as other employees. He can have access to the export, but not use it for his university work. That’s basically the line.
Cheers!
Juan
Good, Pawel can work with the export. I prepare it that same afternoon before I go home. I don’t want to be late. My wife is going out with some friends and I can understand why she needs to disconnect every now and then, just like I do.
On Thursday I go for a coffee and find Pawel in the kitchen. He tells me he’s been working with Marta and they have made a spreadsheet with about 500 old records. He also mentions that the support team is following a new process to close the tickets. Just a checklist, but it seems to be helping. I tell him that I’ve been provisioning a new account to run the models, to isolate things, and it also makes it easier for us to keep an eye on costs.
“The basic infra is ready now. We need to figure out how to deploy this thing next. I have a cron job running on my machine, which is not the best long-term solution, you know.”
He looks at me and just says:
“Yes, yes, ok, I’ll run some experiments today and tell you later.” And he quickly leaves.
This guy is funny. He reminds me of when I started working, and everyone looked so old and scary. So professional.
On Friday afternoon I come back from lunch and find Pawel waiting at my desk. He has his laptop and shows me a table:
high priority, 500 old tickets AWS model encoder
high priority tickets 82 82
caught 65 72
missed 17 10
false alarms 14 30
recall 79.27% 87.80%
precision 82.28% 70.59%
“I know the data is small and all, but the experiment is looking good, we catch a few more high-priority issues but we also have more false positives. And now with real data we can actually improve things.”
“Improve how?”
“My synthetic data didn’t have any examples for certain categories, like installments. I was just using some sample datasets I had found online for banking with some adaptations to our schema.”
“Good catch, installments are quite important. We make a lot of money on those.”
“I know. Marta told me.”
“Did you show this to Fran? What did he say?”
“Yes, so… he actually wants us to deploy this quickly, since he still has to go over all tickets anyway, the extra ones we catch are worth it for them.”
“Ok, makes sense. Did he say anything else?”
“No, that’s it.”
“Right, we’ll meet on Monday anyway. Is everything committed to the repo?”
“Not yet.”
“Well, merge your changes, and I’ll start investigating that next week. Anything I can help with?”
“No, I’m good.”
“Look, I have to wrap up some things before I leave today…”
“Sure, no problem. I’ll go push my changes.”
“Have a good weekend, Pawel.”
“Thanks, you too.” And off he runs.
All the characters and companies are fictional. Inspired by real experiences.