Starting from scratch
How hard could it be? A story about running your own AI models. Episode 1.
The last couple of weeks have been hard at work, we had some layoffs, which really sucks, the company’s future isn’t too bright and leadership is trying to come up with ideas about how to fix things. After I had breakfast with the colleagues this morning, I went back to my desk and received an email from the VP of engineering. I was being assigned to a new project together with the support lead, and a meeting was scheduled at noon that day.
I went to the meeting and there was Francisco, our support lead. Carlos, the VP of engineering, had not arrived. I hadn’t worked much with Fran, as we call him, he’s a nice guy, definitely very enthusiastic, he maybe went a bit hard on the drink during the last company party but, I think most people did anyways lol.
In Spain, there’s usually a company party close to xmas time, the “cena de empresa” is quite an event, and you can see some wild things. Most people get super drunk, there’s a lot of silly dancing, camaraderie and flirting. Many hookups start there and some divorces too.
I was just taking a seat when Carlos arrived. He told me that Fran had asked him for help. Lots of support tickets were being dropped and with the layoffs, they were not following up with the accounts as they should. He then said that I’d be working on this project for the next month or two, to figure out some reports that Fran needed. He’d managed to secure some other “resources” for the project, an intern called Pawel, who’d start in a few days. He then proceeded to remind me of the utmost importance of this project and that he appreciated my flexibility and commitment to the company. I think that’s the corporate way to say that I am in some deep shit and my job is on the line. Fran mentioned he was looking forward to collaborating with me, and Carlos said he had a bunch of fires to put out so he’d leave it in my hands.
After that the three of us went to lunch in a nearby restaurant. We had the “menú del día”, a fixed-price lunch with a starter, a main, dessert and a drink, which is how half of Madrid eats on weekdays. I had some gazpacho to start, then “albóndigas” (meatballs in sauce), they were ok. For dessert some pudding, which by the way was really good. We also had some “vino con Casera”. If you’ve never had it, it’s cheap red wine with sweet soda water. Don’t judge.
We had lunch from two until almost four, which was a bit long, but nothing too extraordinary in Madrid… After the lunch we were a bit tipsy, I went to the kitchen and had some coffee, spoke with some of the colleagues and before I realized it, it was time to go.
I had a few meetings the next day, with Fran and some of his team, and started picturing an easy way to do this. I decided I needed to use AI to solve the problem, otherwise I might be seen as crazy or something and I really didn’t need more pressure. AI is the new IBM, I won’t get fired for using AI ;)
The rest of the week went rather well, I managed to get some exports of the data and quickly built a prototype calling some LLMs for classifying the tickets, which I’d then feed to a program to generate the reports. This should be an easy job, I thought to myself.
It was pretty simple. For each ticket, I sent the text to an LLM API with a prompt like this:
You are a support assistant for a fintech company.
Classify the customer ticket below into exactly one category:
card_arrival, lost_or_stolen_card, compromised_card,
transfer_not_received_by_recipient, top_up_failed,
card_payment_not_recognised, ... (77 in total)
Mark it as serious if the customer may be losing money
or may be a victim of fraud.
Answer only with JSON:
{"ticket_id": "<id>", "intent": "<category>", "serious": true|false}
Ticket <id>:
<ticket text>
And the answer came back like this:
{"ticket_id": "48213", "intent": "lost_or_stolen_card", "serious": true}
Asking for JSON meant I didn’t have to parse free text. Then a small script grouped everything and wrote the report: serious cases at the top, then the counts per category. Nothing fancy, a few hundred lines of Python and a cron job.
I showed it to Fran on Thursday. He scrolled through it for a while without saying anything, and then he said “This is a great start! I’m starting to get some ideas… let’s talk tomorrow.” I had a feeling this report was going to grow a lot. Anyway, I checked a bunch of tickets by hand and the classification looked good, way better than I expected. Honestly, I felt a bit like a genius.
That lasted until the next Monday. Carlos and Fran showed the demo at the executive meeting, and that’s when the issues started. David, our head of security, wasn’t sure we had the permissions to send customer data to a third party. Then he went on about compliance, subprocessor agreements, data processing agreements, blah blah blah. David is a bit old school. He’s more into security paperwork and compliance than engineering. He knows his stuff, just a bit outdated. But he was right, we needed our clients’ permission for it.
After the meeting Carlos came to my desk. “We have a small problem,” he said. In Carlos language, small problems are never small.
That afternoon I had a call with David. The tickets have names, they could even have card numbers, account details, sometimes people’s whole life story. We’re a fintech, we have regulators, and every company that touches our customers’ data has to be in our contracts. The AI provider wasn’t, and adding it meant getting our clients’ permission. That meant bureaucracy, getting legal involved, new contracts, months of emails… and we hadn’t even validated the idea yet. Nobody was going to start all that for a demo.
The demo was dead. Fran looked like someone had taken his toy away.
That night, after putting Lucas to bed, I sat on the couch with a beer and thought about it. The problem wasn’t the AI, the problem was where it ran. There are open source models now, what if we run it ourselves? How hard could it be?
All the characters and companies are fictional. Inspired by real experiences.