AI and List Building: A Checklist for Telling Real Capability From a Good Demo | NPLUS Global

Every vendor pitch now includes some version of "our AI-powered platform." Half of them mean a large language model that reads NPI registries faster than a human intern. The other half mean a marketing team that added the word "AI" to a rules-based matching engine they built in 2019. Both can produce a usable list. Neither is magic. Here's a practical way to sort out what you're actually buying, organized around the points in the process where the truth tends to surface.

Where AI Genuinely Helps Right Now

• Ask which specific step uses a model, not the whole pipeline. "AI-powered" often means one component — entity resolution, specialty classification, or affiliation matching — sits inside an otherwise conventional ETL process. Get the vendor to name the step.
• Test it on messy source data, not clean data. Feed the vendor a batch of provider names with title suffixes, hyphenated names, or foreign credentials and see if matching holds up. This is where language-model-assisted matching genuinely outperforms older fuzzy-logic tools.
• Check how it handles specialty and subspecialty inference. Models trained on claims or NPI taxonomy data are legitimately better than manual tagging at inferring, say, an interventional cardiologist from a general cardiology NPI code plus procedure history. Ask for an example of this specific inference, not a general accuracy claim.
• See if it flags its own uncertainty. A model that returns a confidence score alongside a matched record is more useful — and more honest — than one that just outputs a clean-looking row. If every field looks equally confident, be suspicious.
• Ask what happens when the model doesn't know. Good implementations fall back to a null field or a lower-confidence tier. Bad ones guess and format the guess to look authoritative.
• Verify that affiliation mapping accounts for turnover. Providers change hospital systems, group practices merge, and locum tenens work muddies "current employer" fields. Ask how recently the affiliation data was refreshed and what triggers a re-check.
• Ask if the model is doing summarization or generation anywhere in the list-building process. If it's writing descriptive fields (bios, specialty summaries, "about this practice" text), check a sample for confidently wrong details. This is the single most common place hallucination shows up in list products.

Where AI Still Falls Short — Don't Take the Vendor's Word

• Don't assume AI improves license verification. State board data, DEA status, and sanctions checks are structured, regulated, and time-sensitive. A model doesn't verify these any faster or more reliably than a direct API pull from the source. If a vendor claims AI improves compliance accuracy here, ask them to explain the mechanism.
• Don't assume intent or "buying signal" scoring is reliable at the individual level. Aggregate trend scoring (a specialty segment showing rising engagement) is more defensible than claims about a specific physician's individual intent, which is usually inferred from thin behavioral signals dressed up with a confidence percentage.
• Ask what the model was trained on and whether it's healthcare-specific. A general-purpose LLM fine-tuned lightly on medical terminology is not the same as a model built on years of claims, NPI, and affiliation data. Ask directly — vendors that dodge this question usually have a generic model with a healthcare skin on it.
• Check for stale mental models of specialty structure. Models trained on older data sometimes miscategorize newer subspecialties or misunderstand scope-of-practice changes (e.g., expanded NP/PA prescribing authority by state). Test with a few recent regulatory edge cases.
• Don't let "AI-enriched" data skip human spot-checks. Enrichment fields — especially email format guesses, inferred job titles, or seniority tiers — need manual sampling before you trust volume claims. AI is good at scale, not at catching its own systematic errors.
• Ask how contact-level email and phone data is sourced, not just formatted. AI can clean and standardize a phone number field beautifully while the underlying number is three years stale. Formatting quality is not the same as accuracy.

Questions to Ask Before You Buy Into the Pitch

• Request a before/after sample on your own target list, not their case study list. Vendors demo on their best-performing verticals. Give them 200 records from your actual ICP and see what the model actually does with it.
• Ask what percentage of the list required human review before shipping. A vendor with a mature process will know this number and share it. One that says "the AI handles it end to end" is either overselling or hasn't measured it.
• Ask how errors get corrected once found. Is there a feedback loop where flagged bad matches retrain or adjust the model, or does every batch start cold? This tells you whether you're buying a system or a one-off run.
• Get specific about what "match confidence" means numerically. If a vendor says 90% confidence, ask what that's measured against — a held-out validation set, manual audit, or nothing concrete at all.
• Ask what the model does with records it can't resolve. Are unmatched or low-confidence records dropped, flagged, or silently included at face value? This single answer tells you a lot about vendor discipline.

After the List Lands — Verify Before You Deploy

• Pull a random 3-5% sample and manually verify affiliation and title against a source like a hospital directory or LinkedIn. Don't rely on the vendor's own confidence scores to self-report accuracy.
• Check for duplicate entities under slightly different name formats. This is a common artifact of AI-assisted matching that missed a normalization step — same provider, two records, two confidence scores.
• Look for over-homogenized job titles. Models sometimes normalize titles too aggressively, flattening "Director of Pharmacy Operations" and "Pharmacy Operations Manager" into the same bucket when they shouldn't be.
• Track downstream performance separately from list accuracy. A technically accurate list can still underperform if the AI-driven segmentation logic doesn't match how your sales team actually sells. Measure response rates by segment, not just match rates.
• Revisit the vendor relationship on a fixed cadence, not just at renewal. AI models and underlying data sources change quietly. What worked in the first delivery may quietly degrade six months later without any change to the sales pitch.

At NPLUS Global, we treat AI as one tool in a larger verification pipeline, not a replacement for it — which is really the standard any buyer should hold every vendor to, not just us. The technology is real and worth using. The oversell is the part worth pushing back on.