← ragul.sh

Field notes · Ranking systems

The model reads. The rules rank.

Everyone’s instinct is to hand the whole matching problem to the model. We gave it the messy half and kept the ranking deterministic — here’s the argument, and the part we still haven’t solved.

19 Aug 2026 8 min read Ranking systems

The short version

Buyer matching is the kind of problem that looks like it was built for a language model. Messy inputs, fuzzy criteria, a judgment call at the end. The obvious move is to hand the model a deal and a list of buyers and ask it who fits.

We tried the obvious move. What we ship now puts the model in the middle of the pipeline and nowhere near the final order, and I’d defend that split to anyone building something similar.

I’m co-founder and CTO at Rehouzd. Wholesale operators bring us off-market single-family deals, and the system’s job is to answer one question: of the investors we know about, who should see this deal first? Get it wrong and an operator wastes a day calling people who were never going to buy.

01 · The design

The three stages

The pipeline looks like this, and the order matters more than any single component.

candidates = hard_filter(deal)     # market, price, asset type
                                   # deterministic. no model.

for buyer in candidates:           # the model reads the mess
    buyer.features = extract(
        buyer.transaction_history,
        buyer.notes,
        buyer.past_acquisitions,
    )

ranked = score(candidates, deal)   # deterministic + explainable

Hard constraints eliminate candidates first, deterministically. A buyer who doesn’t operate in the market or doesn’t touch this asset type isn’t a weak match, they’re not a match. There’s no reason to spend model tokens or model judgment on a question that a comparison answers.

Then the model does the part it’s genuinely good at, which I’ll come back to.

Then a scoring function ranks what survives. No model. Weights, inputs, an order.

02 · The argument

Why the ranker stays deterministic

This is the part people argue with, so here’s the actual case. Note what isn’t on the list: cost and latency. Those are real, but they’re not why. If model-driven ranking were better, it would be worth paying for.

An operator won’t act on an order they can’t interrogate. The first question anyone asks about a ranked list is “why is this buyer at the top?” A deterministic score answers that every time, in terms the operator recognizes — this buyer scores high because of these three things. A model can produce an explanation, but it’s a story generated after the fact, and it isn’t necessarily the reason for the ordering. Operators figure that out fast, and once they do they stop trusting the list. An explanation you can’t verify is worse than no explanation, because it spends trust you’ll need later.

Rankings have to be stable. The same deal should produce the same order today and next week. Model-driven ranking drifts — a prompt edit, a version bump, an unusual input, and suddenly the buyer who was third is seventh. Nothing broke, no test failed, and the operator who noticed now believes the system is arbitrary. Determinism isn’t a purity thing here. It’s what lets someone build a mental model of the system and keep it.

When it’s wrong, you need to know what to change. An operator says the ranking is off. With a scoring function you look at the weights, change one, and know exactly what moved and what didn’t. With a model-ranker you’re rewriting a prompt and hoping, with no bounded blast radius and no way to confirm you didn’t break the cases that were fine. The debugging story matters more than the day-one accuracy, because you will be debugging this for as long as it’s in production.

03 · The failure

What the first version got wrong

Three pilot operators used v1, and they caught two things fast.

The first was activity bias.

Activity bias

Activity bias is when a ranking optimizes the signal that’s easiest to measure instead of the outcome that actually matters — ranking on how recently and how often someone transacted, when the thing you care about is whether they’ll close.

We ranked buyers by recent transaction activity, because transaction activity is what the data gives you cleanly. Operators looked at the top of the list and told us those weren’t the people who close. Some high-activity buyers make a lot of offers and follow through on few. Some quieter buyers close nearly everything they engage on. The data said “active.” The operator knew “flaky.” Our ranking had confidently optimized a proxy.

The second was buy-box nuance. The model matched on the attributes that were sitting in structured fields — price, beds, market — and missed everything operators actually hold in their heads: how much condition a buyer will tolerate, how they finance, which streets they won’t touch regardless of the numbers. Technically-valid matches that any experienced operator would immediately discard.

Both failures have the same root. The system was reasoning over the data that was convenient rather than the data that was true.

04 · The split

Where the model earns its place

The buy-box problem is what convinced me the model belongs in the pipeline — just not at the end.

The naive fix is to ask operators to fill out a buy box per buyer. It works for about a week. Nobody maintains it, it goes stale, and now you have a form that lies.

So the model reads instead. It goes through a buyer’s transaction history, acquisition patterns, and the unstructured notes people actually write, and it infers the buy box: this buyer consistently takes heavier rehab, this one never goes above a certain age of construction, this one has only ever bought in three sub-markets despite being “active” across the metro. That’s a real constraint set, derived from behavior rather than from what someone typed into a field eighteen months ago.

This is the whole argument for the split in one sentence: put the model where the mess is, and keep it out of the decision. Reading unstructured history into structured features is a job that has no deterministic solution — it’s exactly what language models are for. Ordering a list of candidates by known features is a job that has a perfectly good deterministic solution, and using a model there trades away explainability and stability for nothing.

05 · The honest part

What’s still broken

Buy-box extraction fixed the second problem. Activity bias, I have not fixed.

I know that ranking on activity is measuring the wrong thing. I don’t yet have a signal that reliably captures who actually closes. The obvious candidates all have problems. Close rate needs enough engagements per buyer to mean anything, and the tail is thin. Operator judgment is the most accurate signal available and the hardest to collect consistently, because it requires busy people to record something at the exact moment they’d rather move on. Inferring reliability from outcomes takes a volume of completed cycles we’re still accumulating.

So today the ranking still leans on activity more than I’d like, softened by the buy-box work but not solved. If you’re building a matcher, this is the trap I’d flag hardest: the signal that’s easy to collect will quietly become the thing you optimize, and it takes a user telling you to notice.

Anyone who tells you their ranking captures reliability should be asked how they measured it.

06 · The result

Did it work?

I don’t have a clean number, and I’d rather say that than dress up a metric.

What I have is behavioral. Operators stopped overriding the list. Before, they’d scan the ranking, ignore it, and work from their own memory of who to call — which is the real signal that a matcher has failed, because the system is generating work instead of removing it. Now they start at the top and work down.

The lists also got shorter. v1 returned long sets of technically-valid buyers, which is functionally the same as returning nothing: a list of forty is a list you have to re-rank yourself. Hard filters plus a real buy box cut that to a handful an operator can act on in one sitting. Fewer, more defensible candidates beat comprehensive every time, because the operator’s scarce resource is attention, not options.

07 · Questions

Questions I get asked about this

Should an LLM rank search or recommendation results?

Usually not. Ranking has a deterministic solution, and using a model costs you explainability and run-to-run stability while making the system harder to tune. Use the model to turn unstructured inputs into features, then rank those features with code.

What is activity bias in a ranking system?

It’s optimizing the signal that’s easiest to measure instead of the outcome that matters — ranking on recent activity when you care about who actually follows through. It’s common because activity data is clean and outcome data isn’t.

How do you capture requirements that live in users’ heads?

Don’t ask them to maintain a form; it goes stale and then it lies. Infer the constraints from behavioral history with a model, and let users correct what’s wrong.

How do you keep an AI-driven ranking explainable?

Keep the model out of the ordering. If a deterministic function produces the rank, you can always answer “why is this first?” with the actual reason rather than a plausible-sounding reconstruction.

If you’re building a matcher

  • Filter deterministically, extract with the model, rank deterministically. The order of those three stages is the design.
  • Don’t let a model produce a final ordering a human has to trust and interrogate. Post-hoc explanations spend trust you need later.
  • Watch for activity bias. The signal that’s easy to collect will become the thing you optimize unless someone catches it.
  • Don’t ask users to maintain a form describing what they want. Infer it from what they’ve actually done.
  • A shorter, defensible list beats a comprehensive one. Attention is the scarce resource.
  • The honest measure of a matcher is whether people stop working around it.

The model is the best tool available for reading mess. It is not the best tool for making a decision you’ll have to defend.


Work with me

I take on a small number of outside engagements — production AI agents and workflow automation, the parts that have to hold up when a real operator is depending on them.

Fixed-scope sprint

4–8 weeks

An agent or automated workflow shipped to production, with the tool-boundary validation, contract tests, and approval gates that make it trustworthy.

Seed / Series-A teams with a workflow that should have been automated last quarter.

Fractional technical lead

Ongoing, part-time

Architecture, agent reliability, and technical direction without a full-time hire. Design calls, reviews, and hands-on work where it counts.

Teams shipping AI features who need someone senior on it continuously.

Tell me what’s breaking →

All writing  ·  ragul.sh