How to Vet an AI SDR Vendor's Claims Before You Sign

Why this matters now

I run an AI SDR in my own stack, so this is not a "don't buy AI SDR tools" post. It is the opposite. I think the category is good and getting better fast, which is exactly why the vetting question matters more this year than last year. When a category is young, buyers ask "does this work at all." When a category is hot, buyers stop asking that question and start asking "which vendor has the best demo," and vendors know it. The gap between a great demo and a durable result is where inflated numbers live.

I read the reporting on one AI SDR vendor's internal numbers this month and it is the clearest, most specific case study I have seen for why you verify before you sign, not after.

The story in brief

According to a TechCrunch investigation into the a16z and Benchmark-backed AI SDR company 11x, one former employee said plainly: "We were losing 70-80% of customers that came through the door." The company's public position was a 79% retention rate, roughly the inverse of what the employee described. Another former employee told the reporter the company "absolutely massaged the numbers internally when it came to growth and churn."

On revenue, the same reporting says the company had claimed around $14 million in annual recurring revenue on the strength of signed contracts, but a former employee estimated that once you filtered to contracts that actually passed the three-month trial period, the real number was closer to $3 million. The reporting also names specific customers, ZoomInfo and Airtable among them, whose logos appeared on the company's site despite short or non-production trials; ZoomInfo told the reporter directly, "we did not move forward afterward."

The metrics vendors love to show you

Three numbers show up on almost every AI SDR sales deck, and all three are easy to make look better than reality without technically lying.

Logos. A logo wall proves a company signed a contract or ran a trial. It proves nothing about whether that company is still a customer, used the product in production, or would recommend it. The reporting above found logos still displayed for companies that had explicitly declined to continue.

Headline ARR. Contracted ARR at the moment of signature is not the same as revenue that survives a trial period, a renewal, or a budget review. A number quoted to a board or a prospect is only as honest as the definition behind it, and definitions vary company to company.

Blended retention. A single retention percentage across your entire customer base can hide a brutal first-90-days churn rate if it is averaged against a smaller pool of long-tenured, happy accounts. Ask for the cohort view before you trust the headline.

The metrics that actually matter

These are the numbers I'd actually want before committing budget, and none of them are exotic or unreasonable to ask a vendor for.

Contracted ARR vs. realized revenue

The TechCrunch reporting notes the company used the term "contracted ARR," or CARR, in board reporting. That term is not inherently dishonest; a lot of software companies track pipeline-adjacent metrics internally. The problem is when a contracted figure gets presented externally, in a sales conversation or a press mention, without the word "contracted" attached, because a prospect hears "ARR" and assumes it means realized, collected revenue from customers still using the product.

My rule of thumb: any time a vendor states a revenue number, ask directly whether it is contracted, collected, or annualized off a recent month. Then ask what percentage of contracted deals make it past the trial window. If a vendor cannot answer that second question cleanly, that itself is useful information.

Questions to ask before a pilot

I ask some version of these on every vendor call now, whether I am buying for a client or evaluating a tool for my own stack.

1. What is your gross revenue retention, not net, over the last four quarters? Net retention can look fine even with heavy churn if expansion revenue from your best accounts covers for it.

2. What percentage of new logos are still active at 90 days? This is the single number closest to the 11x story above, and it is the one vendors are least likely to volunteer unprompted.

3. Can I talk to a customer who churned, not just one who stayed? Most will say no. Ask anyway. The answer, and the reason given, tells you something.

4. How do you define a "meeting booked" and a "qualified opportunity"? Get the exact criteria in writing before the pilot starts, so you are not renegotiating the definition of success after you have already paid.

5. What does the product do when the input data is bad? A vendor that has thought about garbage-in failure modes will have a real answer. One that has not will talk around the question.

How to read a case study skeptically

A case study is marketing, not evidence, until you can verify it. Three quick checks before you let a case study move your decision: does it name a specific, checkable metric with a time window, or a vague "significant improvement"; can you find the referenced customer independently, on LinkedIn or their own site, rather than only through the vendor's link; and does the vendor offer to connect you directly with that customer, unprompted, or only after you push. A vendor confident in its results will make the introduction easy. One that stalls is telling you something.

Reference checks that actually work

Vendor-supplied references are curated by definition, so I treat them as a floor, not a verification. What actually works: ask the vendor's sales rep for the name of the account executive or customer success lead who managed the account you are being shown as a reference, then ask a mutual connection or your own network if anyone has worked with that same vendor outside the reference list. In a category as networked as B2B GTM tooling, someone in your circle has usually run a trial with any vendor you are seriously considering, and their unprompted take is worth more than three curated calls.

A pilot structure that protects you

Structure the trial so the vendor's incentives match yours instead of just theirs. Tie a meaningful share of the contract to a 90-day checkpoint against the specific metrics you defined in writing before you started, not the vendor's dashboard definitions. Keep the pilot on a real segment of your actual list, not a hand-picked easy list the vendor suggests. And put a plain-language exit clause in the contract before you sign, not after a renewal conversation goes badly. None of this is adversarial. A vendor that has confidence in its product will agree to all three without much friction.

Red flags checklist

A short list I'd walk away from, or at minimum slow down hard on: a vendor that will not break out retention by cohort; a case study you cannot verify independently within ten minutes of searching; a sales rep who answers "what counts as a meeting booked" with a vague or shifting definition; logos on the homepage the vendor cannot confirm are current, paying customers when asked directly; and any resistance to a written exit clause before signing. None of these alone is disqualifying. Two or more together is a pattern worth taking seriously.

Where Agent Frank fits in this

This whole exercise is exactly why I run Agent Frank, the AI SDR inside the Forge ecosystem, alongside Salesforge for sequencing and Leadsforge for the list layer, rather than a standalone black-box tool. The number I actually stand behind is my own: 2000+ meetings booked for B2B clients, which is a track record I can point to, not a vendor-supplied case study you have to take on faith. I'd rather a prospective client ask me the same hard questions in this article than take my word for it, and I'd expect any AI SDR vendor worth using to welcome the same scrutiny. Check current pricing directly with Salesforge for exact numbers; I'm not going to invent a figure here to make a point about honesty.

My take

The AI SDR category does not have a technology problem right now nearly as much as it has an incentive problem. The tools are genuinely useful. The reporting model most vendors use to sell them is not always honest, and it does not have to be, because most buyers do not ask the second or third follow-up question. I think that is going to change fast as more stories like the one in this article become public, and the vendors that already report clean, cohort-level numbers are going to win more deals in 2026 and 2027 precisely because they stopped hiding the ball. Ask the questions in this article before your next pilot. It costs you one extra call and it can save you a wasted quarter.

FAQ