Project essay

Building Referral Intelligence: ranking with reasons

A production specialist-ranking system at Snap eHealth that shows its reasons, admits when evidence is thin, and never takes action on its own.

Thomas Howard · 6 min read

My role

Architecture & service implementation

Focus

Python · Explainable ranking · ML data design

Project stage

Live production · Decision support

When a patient needs a specialist, someone has to pick one. The evidence behind that choice is rarely complete: quality data for one provider, volume for another, no network information for a third. A system that turns this into a single confident number can send a patient to the wrong place, and nobody can say why.

I architect, implement, and operate Referral Intelligence, Snap eHealth’s production physician-recommendation system. It ranks specialist candidates, explains the factors behind each result, and withholds a recommendation when the evidence does not meet the eligibility policy. The hard part is making incomplete data useful while keeping the calculation, the gaps, and the authority to act visible.

A score should never carry permission to act

The risk with any recommendation engine is scope creep. Once it is wired into the workflow, a change to a scoring rule can start to behave like a change to who gets referred where, with no human decision in between.

The engine began inside Snap Workflows. I designed its extraction into a standalone Python package behind an authenticated FastAPI interface. Domain contracts and scoring rules live apart from HTTP validation and data adapters, and the ranker only sees normalized models, never vendor payloads or web-framework objects.

That boundary gives the service one job: assess and explain the candidates. Workflows keeps user sessions, operational queues, specialist selection, authorization, and EHR writes. A scoring change cannot acquire permission to act on a referral.

The tradeoff is an extra service to run. In return, the package can be tested on its own and its contracts can evolve independently.

Every point in the score has to be explainable

A ranking nobody can explain is a ranking nobody should trust in healthcare. If a care coordinator asks “why this doctor?”, the answer has to come from the math, not from a story written after the fact.

So the production ranker uses explicit weights. It puts each signal on a common scale, computes weighted contributions, and sorts by score, breaking ties on a stable provider identifier so the same inputs always give the same order. The factors include quality, condition experience, relevant volume, outcomes and utilization, network alignment, proximity, continuity, and evidence confidence.

Explanations are built from the contributions the calculation actually used. Two specialists can land on similar totals for different reasons: one has an established relationship and is nearby, another has stronger condition-specific evidence. The assessment shows that tradeoff.

Explicit weights also make our own assumptions reviewable. Condition experience and total volume, for example, may partly measure the same thing. Because the contributions are visible, that overlap can be examined and the weights revised on purpose.

Missing data must not look like good data

The quiet failure in ranking systems is the sparse profile that looks great. A provider with two strong signals and six unknowns can outscore one with a full, honest record if the system fills the gaps generously.

Normalization enforces the input contract first. Invalid percentages, nonfinite values, negative or fractional counts, mismatched conditions, and missing provider records are rejected or marked as unavailable. Network compatibility requires compatible plan information, because knowing who a provider is does not tell you whether the patient’s coverage applies.

Then the key decision: unavailable signals earn no points, and their weight is not redistributed to the signals that are present. Coverage is reported separately. The cost is real. Providers with less data can score lower. So the response lists the missing factors, and a reviewer can see when a low score reflects thin evidence rather than weak signals.

I keep three ideas apart. Confidence describes the size of the outcomes sample. Coverage describes which signals are present. Score is the weighted assessment. None of them is a calibrated probability that the referral will go well, and the consumer can explain a recommendation without pretending otherwise.

“Not enough evidence” is a real answer

Every ranking has a top result. That does not mean the top result is good enough.

I separate ranking from eligibility. The highest-scoring candidate must still meet the required signals, quality-profile requirements, sample-size confidence, and confirmed network compatibility. If it fails, the service returns the full ranked assessment with reasons and withholds a recommended provider.

That is a valid result, not an error. The API distinguishes it from an authentication failure, a malformed request, an unknown context, or an outage, so the workflow can route insufficient evidence to a person instead of retrying or crashing.

The policy checks the top candidate and does not silently fall through to the next one. Falling through would be a different recommendation policy, and changing policy should be a visible, tested decision.

Prove the extraction changed nothing

Moving a production engine out of the application it grew up in is exactly when behavior drifts unnoticed.

Before extraction, we captured full assessment responses for controlled synthetic scenarios. The standalone API is checked against those parity fixtures for ordering, contributions, coverage, confidence, missing evidence, and abstention.

Verification installs the built wheel in a separate environment, starts it outside the source checkout, and calls it over loopback HTTP. That catches missing package resources and accidental imports from the source directory, which ordinary unit tests can miss.

API tests cover bearer authentication, invalid input, unknown contexts, cache behavior, and the difference between abstention and a transport error. Fixture adapters are restricted to development and testing. CI runs locked package checks so the shipped package and its domain contract are verified together. A change to weights or eligibility then shows up as an explicit behavior change, ready for review.

Ranking is not prediction

A ranking score is not the “chance this referral works out.” The ranker assesses candidates under defined rules. Predicting whether an eligible referral completes within 30 days is a different problem, with a different target, dataset, and evaluation contract.

That prediction work is a separate educational project, Physician Referral Success Predictor. It is in development on synthetic data and has not shipped. Its NumPy and pandas package generates and validates 20,000 fictional referral candidates by default. Invented relationships among wait time, distance, capacity, experience, and specialty feed a logistic simulation. Insurance acceptance is an eligibility gate: ineligible candidates have no success label at all.

The main design concern is time. A model that peeks at the future looks brilliant in testing and fails in use. Features must be available when the referral is created. A success becomes observable at its completion event; a non-completion only after the full 30-day window. Pending referrals stay pending, and future or same-timestamp outcomes are excluded from historical features.

The generator updates histories as outcomes arrive, and validation recomputes them independently. Tests cover reproducibility, schema rules, eligibility, cold starts, and exact time boundaries. A physician with no history gets a neutral default and an explicit zero count.

What exists today is the data pipeline and temporal validation. Chronological splits, fitted baselines, and probability calibration come next. My Fourth Down project works through related questions on historical sports data: define the outcome, enforce the prediction-time boundary, and compare against a simple baseline.

Every failure gets its own answer

When a recommendation changes, someone will ask why. Without clear boundaries, the honest answer is “we’re not sure.”

The service contract keeps input validity, evidence coverage, ranking, eligibility, and workflow authorization separate. Each failure has a distinct result: reject invalid data, expose missing evidence, abstain when policy requires it, or return an explained recommendation to the authorized workflow.

That also makes changes reviewable. Engineers can tell whether a different recommendation came from new evidence, a normalization rule, a weight, or an eligibility decision, and verify the affected behavior before release.

What I’d tell a founder

Keep the scorer and the actor apart. If the component that ranks options can also act on them, every scoring tweak becomes an unreviewed operational change.

Make “I don’t know” a first-class output. A system that can abstain, with reasons, earns more trust than one that always has an answer.

Don’t let a score impersonate a probability. Ranking under rules and predicting outcomes are different problems; ship the one you can defend and label the other honestly.