Shopify

I’m on the Applied ML team, specifically the group building the shopping agent people talk to inside the app. I got real tickets in week one, and my first production change merged that same week, in a language and a monorepo I’d never opened before.

When a shopper asked something vague, like help with “my order” when they had several, the agent would pick one and act on it. I built and shipped the fix: the agent can now ask a follow-up question with tappable options before it does anything. I owned the prompt policy and the server side, worked with the engineer who owned the mobile client, and took it from a timeboxed prototype to 100% of users. About 72% of shoppers answer the question when it comes up, and it hasn’t caused a single client failure.

The surprising part was why it hadn’t worked before. Four separate instructions in the prompt were all pushing the agent to just act, so adding a fifth rule saying “ask first” lost every time. Replacing them with one coherent policy is what actually fixed it.

I took over a change from an engineer who had left, a set of brand-page improvements, and ran it through a proper evaluation: about 2,900 conversations scored before and after against a rubric. On the model we tested, the share rated great on personalization went from 48.5% to 63.1%. Then I noticed production runs a different model, and on that one the gain was flat. So it shipped as a UX fix, which it was, and not as a quality win, which it wasn’t. The scores are ordinal too, so I compared distributions instead of averaging them.

The agent can hand hard questions to a slower, more expensive reasoning model. It was doing that even after it already had good results, which added around ten seconds before the shopper saw anything. I changed the tool so it needs a real reason to escalate and has a clear point to stop. Redundant escalations dropped 79% and affected conversations got a median 10.7 seconds faster, shipped to all of production.

A new search feature was going to call the agent’s intent classifier as people typed, and somebody had to prove the endpoint could take it before the other team shipped. I built the load test on the internal framework and ran it the boring way, from a smoke test to a baseline to a ramp and finally a thirty-minute soak at about 3,000 requests a minute. Latency held flat the whole time, with p95 under 80ms. It wasn’t perfect: about 0.1% of model calls errored and a few requests came back as server errors, so the zero-server-error target didn’t pass, and I wrote that up alongside the targets that did. Nearly all the latency turned out to be in the model process rather than the application.

Getting the test traffic through meant widening an authorization rule. The review bot flagged one endpoint, and when I went back and tried to break my own change I found it would have exposed more than ten, one of which could mutate data. I replaced it with a default-closed allowlist before it ever merged.

Later I moved the same classifier to a new Python service, built its infrastructure and parity tests (every output matched the old one), and load-tested it at five times the expected Black Friday peak. All 24,735 requests succeeded while it scaled from 3 to 15 servers.

Mobile users kept reporting that the agent came back with nothing, and nobody on the team could reproduce it. Rather than try again myself I went and read more than 300 production traces for anonymous mobile requests. The server was succeeding every single time, just taking anywhere from 30 to 112 seconds to do it, and the client had given up long before then. It looked like a “no results” bug and it was really a latency problem, which is why error monitoring never caught it: every one of those requests returned a 200.

The team had a retraining loop for the intent classifier that was supposed to improve itself, and it never promoted a new model. I worked out why: the classifier was already at its ceiling, retraining on top of the old weights made it worse (0.884 to 0.852 macro-F1), and 90% of the evaluation set guarding promotion had been labeled by the previous model. I also helped prototype a background agent that keeps working on what a shopper needs over hours or days, including the push notification that brings them back. That’s the same long-running-flow problem I spent two years on at Call Fusion, in a completely different context.

The habit I’ve picked up from this team is shipping in thin slices and saying my assumptions out loud, then making them configurable instead of sitting around waiting for someone to unblock me.