Shopify

I’m on the Applied ML team, specifically the group building the shopping agent people talk to inside the app. I’m the only intern on that group, so I got real tickets in week one and there was nobody at my level to copy off. My first production change merged that same week, in a language and a monorepo I’d never opened before.

The agent wasn’t reliably showing brand pages when people searched for a brand. Four separate bug reports turned out to be the same underlying thing. I rewrote the parts of the system prompt dealing with brand intent and when to surface a brand card, then measured it properly: the same conversations run through the old prompt and the new one, scored by a model against a rubric, about 3,000 conversations in the final pass.

It came out at +13 points on personalization quality with safety flat at 98%. The number matters less to me than what it took to trust it. The scores are ordinal, so averaging them would have been wrong and I read the distributions instead. I went through every case where safety flipped, one at a time, to show the regressions came from a layer underneath mine rather than from my change. And I threw out one of my own metrics before it reached leadership, because I couldn’t defend how it had been calculated.

A new search feature was going to hit the agent’s intent classifier on every keystroke rather than once per query, and somebody had to prove the endpoint could take that before the other team shipped. I built the load test on the internal framework and ran it the boring way, working up from a smoke test to a baseline to a ramp and finally a thirty-minute soak. Seven runs and roughly 250,000 requests later it was holding 99.7% at 3,000 requests a minute with no server errors, so the answer was yes and I could say why.

Getting the test traffic through meant widening an authorization rule. The review bot flagged it, and when I went back and tried to break my own change I found the bot had actually undercounted. The obvious fix would have exposed more than ten internal endpoints, one of which could mutate data, so I shipped a much tighter allowlist. The other thing that fell out of the test was that nearly all the latency was in the model sidecar rather than the application, which changed where the team spent its scaling effort.

Mobile users kept reporting that the agent came back with nothing, and nobody on the team could reproduce it. Rather than try again myself I went and read more than 300 production traces for anonymous mobile requests. The server was succeeding every single time, just taking anywhere from 30 to 112 seconds to do it, and the client had given up long before then. It looked like a “no results” bug and it was really a latency problem, which is why error monitoring never caught it: every one of those requests returned a 200.

Two things at the moment. One is a retraining loop for the intent classifier that improves itself: it logs every inference with a confidence score, picks out the cases the model was least sure about, gets a frontier model to label those, retrains, checks the result against a held-out set, and swaps the new model into serving without a redeploy. The other is a background agent that keeps working on what a shopper needs over hours or days rather than answering one question and stopping. That second one is the same long-running-flow problem I spent two years on at Call Fusion, in a completely different context.

The habit I’ve picked up from this team is shipping in thin slices and saying my assumptions out loud, then making them configurable instead of sitting around waiting for someone to unblock me.