Four Questions to Ask Before You Build an AI Agent For Your Clients

I gave a session at GTIA ChannelCon called "AI Agents That Flopped." A field report from the lessons we learned through failures. Many people enjoyed the discussion so I decided to make it public here for everyone to see.

Over the past year at Bumblebee we've built AI agents for MSPs across profitability analysis, security alert triage, billing reconciliation, cost allocation, ticket triage, reporting, device monitoring, and provisioning. Some became products. Several died. A couple worked and still weren't worth scaling.

What came out of that year isn't a list of the right use cases — that list changes every time a new model releases. It's a scoring framework: four questions to ask about any proposed AI use case, before you spend a dollar building it.

1. Data volume — can the agent chew through this in one bite?

The first question is the least interesting and the most frequently fatal: how much data does the agent have to look at to answer the question?

Agents have a context window. It's large and getting larger, but it is finite, and the practical limit arrives well before the advertised number. We have run into this context window limitation before. Long before you fill it, quality degrades — the agent skims, loses the thread, and confidently returns something wrong.

What it looks like when this box is unchecked.

An MSP owner asked us a reasonable question: "Is this client profitable? Add up every hour we worked for them last year and tell me if I'm losing money."

Structurally this is easy — a multi-step math problem with a well-defined equation:

1. Pull every hour worked on the client through time entries
2. Multiply by the bill rate of the corresponding tech
3. Compare against what we actually billed for the time

We needed AI only to navigate the messy data; the arithmetic is trivial. The agent never got to the arithmetic. Step one alone flooded it.

Do the volume math before you build — it takes thirty seconds:

3,000 time entries × ~90 words each ≈ 270K tokens — past a typical 256K context window.

That's one client, one year. Stretch the window to three years and it breaks three times as badly. No prompt is clever enough to fix that.

2. Data consistency — does the data have a shape you can rely on?

When the agent goes looking for the fact it needs, is that fact in a predictable place? Or is it wherever the last person who touched the record happened to put it?

Structured data has a shape. Free text does not. An agent can work against unstructured data, but only when the pattern is stable enough for it to learn where to look.

What it looks like when this box is unchecked.

A security team told us: "When I see an Impossible Travel alert, I don't know if it's legit. I usually have to ask in the group chat whether anyone interacted with that contact or documented a leave request in a ticket."

Login in Dallas at 9:02, login in Berlin at 9:41. Is it a breach?

The exonerating answer always existed somewhere. But somewhere was the problem. Sometimes it was a comment buried thirty-eight deep on a different ticket — "Heads up, I'm OOO in the Berlin office all week." Sometimes a technician's free-form note. Sometimes a line in a client meeting transcript nobody had tagged. We encountered the same problem while building recurring issue detection workflows. Same fact, a different container every time, and no field that reliably held it.

That is what an agent cannot learn. With no stable place to look, it doesn't return "I don't know" — it reaches for the most plausible-sounding text in the pile and reasons from it. The failure mode isn't a miss, it's a confident wrong answer.

We ran it ten times on the same scenario. It found the real evidence four times. 40% success rate — and you cannot put a coin flip in front of a client on a security decision.

Consistency is fixable in principle, but the fix is upstream: change where your team logs things, on every technician, every ticket. That's a process change, not a sprint. Price it honestly.

3. Verifiability — can you cheaply tell whether the answer is right?

Here it gets harder, because the next two questions aren't about the technology at all. They're about the shape of the problem.

When the agent hands you an answer, what does it cost you to know whether it's right? If checking the work costs nearly as much as doing the work, you haven't automated anything. You've just moved the labor.

What it looks like when this box is unchecked.

Billing reconciliation. "I want to make sure I'm billing clients accurately in the face of price changes, PSA sync failures, and a growing customer base." We built an agent to pull what the PSA says we should bill, compare it line by line against what the vendor actually charged, and report differences in quantity, price, and proration.

The ceiling appeared in verification. Invoices generate fresh edge cases every month — a mid-cycle price change, a license added on the 14th and removed on the 22nd, a credit applied against the wrong period, or a vendor changing how it names a line item. Each difference has to go to a person who decides whether it is a real error or just an unusual month.

That manual review still takes one full day every month, and we could not reduce it further. We could teach the agent each edge case after the fact, but new ones kept arriving. Verification was not a backlog we could work down; it regenerated on its own schedule, and a reviewer had to be there every time.

That's the ceiling. The work left for a human is also the work that comes back. Before you build, ask what happens the first time the agent meets something new. If the honest answer is "a person looks at it" — and if new keeps arriving every month — you've found your ceiling before you've spent a dollar.

4. Repeatability — does the process hold its shape across cases?

The last question is the one that kills otherwise-good agents: is this the same problem every time, or a new problem wearing the same name?

What it looks like when this box is unchecked.

Cost allocation. "My team follows the same set of rules to build pivot tables for cost allocation. Some months we're drowning; other months we have no work." Dozens of branching rules — software license? Split by seats per client. Managed-services cost on a per-device contract? Allocate by endpoint count, or by ticket hours if not. Project cost? To that project's client. Everything else into an overhead pool, split by revenue share.

This one had everything the last one lacked. Closed books are finished and final, so we could check the agent's work against months that were already done and signed off. No new exceptions arriving, no reviewer in the loop — verification scaled on its own.

The wall was somewhere else. Every client build took 15 hours, and that number did not go down as we added more clients. Each client's inputs arrive in a different shape: payroll in ADP or Gusto or QuickBooks, time tracked in ConnectWise or Autotask or Halo, accounts grouped however that client groups them. The rules had to be modeled by hand each time, and nothing we learned on client seven made client eight cheaper.

So the real math isn't four hours saved times a hundred clients. It's fifteen hours of hand-modeling, a hundred times over, before any of that saving starts. Verifiability decides what each run costs you; repeatability decides what the next client costs you. We had the first and not the second — which leaves you with a custom project you'll be doing forever, priced as if it were a product.

The scorecard

Four questions, asked in a room, before anyone writes a line of code. Every failure above cost us weeks, and every one would have shown itself in ten minutes if we'd asked.

But this is a snapshot of what broke us this year, not a permanent law. I anticipate the list of criteria will continue to evolve as we learn more and as AI models get better.

There are also important things not on the list — such as domain expertise. Every project that worked had someone who understood the underlying work — what a cost allocation rule is trying to accomplish, which invoice discrepancies are real and which are noise. It's not a criterion because it doesn't help you choose; it's required for all of them. But if nobody on your team can explain the process end to end without hedging, the score doesn't matter.

We'll keep evolving this. If you're building agents for your clients and hitting a wall we haven't named, I'd like to hear about it.