Case study
Job Matcher
Job Matcher scores how well a CV fits a job offer from 0 to 100 with a model of my own, then has Claude explain the gaps. The model was trained on Claude's scores, so that is what it measures up against.
Off from Claude's score by 8 points on average — not a recruiter's.
- 8.04
- points of mean error against Claude's own scores (5-fold CV)
- 15.6
- points of error from always guessing the average score (test split); the model's is 8.70
- 0.629
- R² on 100 held-out pairs the model never saw (test split)

The problem
Asking an LLM for the score itself means a slow, paid call that does not return the same number twice. A model trained for the job is quick and cheap to run, and gives the same score for the same features every time; Claude still reads the documents and writes the report.
But a model needs labelled examples, and there is no public record of which candidate fitted which offer. So Claude labelled synthetic pairs once, and a small model learned to reproduce its scores.
How it works
Scoring a real CV
- CV + job offer
- The CV comes as a PDF or DOCX; the offer as a URL or pasted text.
- Claude features
- Claude reads both into structured fields, and Python turns them into the 14 numbers the model expects.
- ML score
- The trained GradientBoosting model turns those numbers into a 0–100 fit score: the score itself comes from the model, not from an LLM.
- Fit report
- Claude writes the gaps and recommendations; matched and missing technologies show as green and red pills.
Training the model
- Synthetic pairs
- 500 candidate and job-offer profiles, generated in Python. No real people, no real hiring outcomes.
- Haiku 4.5 scores
- Claude Haiku 4.5, prompted as a technical recruiter, scored each pair once from 0 to 100.
- Regressor
- A scikit-learn GradientBoosting regressor learned to reproduce those scores from 14 features, trained on 400 pairs and tested on 100.
Results
Every error below is distance from Claude's score, not from a hiring decision. Against that target the model nearly halves the error of guessing the average, and the level gap between candidate and role drives most of it.
- Test error
- 8.70
- mean absolute error on the 100 held-out pairs
- Within 10 points
- 63%
- of held-out predictions land within 10 points of Claude's score
- Within 15 points
- 80%
- the other 20% miss by more
What it gets wrong
- It learns Claude, not recruiters. The ground truth is one Haiku 4.5 score per pair. I never measured whether Haiku gives the same pair the same score twice, so part of the 8 points may be noise in the labels themselves.
- Claude scores in steps. Its 500 scores use only 24 distinct values, and its 8 most frequent (15, 28, 35, 42, 45, 52, 62, 72) take 81% of them, 404 of 500. The model is chasing a coarse target.
- It pulls towards the middle. Strong fits, scored above 70 by Claude, come out 11.0 points lower on average. Regression to the mean, the price of a small dataset.
- It learns Claude's quirks. Claude scores candidates two levels above the role slightly lower on average than those one level above (58.8 against 60.5), a hint that it penalises over-qualification. A model trained on those labels learns whatever quirks they carry; I did not measure whether it reproduces this one.
- Small and synthetic. 500 invented profiles, not real people. A real version needs real hiring decisions to check against.
“The model learns Claude's scoring criterion, which may differ from actual recruiter judgment.”
What I learned
A metric is only as honest as its labels. MAE 8.04 sounds like accuracy; it is distance from another model's opinion. Saying which one is half the result.
A number needs something to beat. On its own 8.04 means little; on the held-out pairs the model's 8.70 against the 15.59 of always guessing the average shows it learned something real.
The average error hides where the model is wrong. Splitting it by score showed the pull towards the middle that a single MAE never would.
Let's talk
Remote preferred. Based in Spain.