Case study

Job Matcher

Job Matcher scores how well a CV fits a job offer from 0 to 100 with a model of my own, then has Claude explain the gaps. The model was trained on Claude's scores, so that is what it measures up against.

Off from Claude's score by 8 points on average — not a recruiter's.

8.04
points of mean error against Claude's own scores (5-fold CV)
15.6
points of error from always guessing the average score (test split); the model's is 8.70
0.629
R² on 100 held-out pairs the model never saw (test split)
Job Matcher result: a fit score of 77 out of 100, 'good fit, worth applying', the detected profile, and technology pills: Python, FastAPI, LangChain, Docker, AWS and Azure matched; Hugging Face Transformers, MLOps, CI/CD, Kubernetes and Google Cloud missing.
A frame from the demo video: the score ring, the detected profile, and matched (green) and missing (red) technologies.

The problem

Asking an LLM for the score itself means a slow, paid call that does not return the same number twice. A model trained for the job is quick and cheap to run, and gives the same score for the same features every time; Claude still reads the documents and writes the report.

But a model needs labelled examples, and there is no public record of which candidate fitted which offer. So Claude labelled synthetic pairs once, and a small model learned to reproduce its scores.

How it works

Scoring a real CV

CV + job offer
The CV comes as a PDF or DOCX; the offer as a URL or pasted text.
Claude features
Claude reads both into structured fields, and Python turns them into the 14 numbers the model expects.
ML score
The trained GradientBoosting model turns those numbers into a 0–100 fit score: the score itself comes from the model, not from an LLM.
Fit report
Claude writes the gaps and recommendations; matched and missing technologies show as green and red pills.

Training the model

Synthetic pairs
500 candidate and job-offer profiles, generated in Python. No real people, no real hiring outcomes.
Haiku 4.5 scores
Claude Haiku 4.5, prompted as a technical recruiter, scored each pair once from 0 to 100.
Regressor
A scikit-learn GradientBoosting regressor learned to reproduce those scores from 14 features, trained on 400 pairs and tested on 100.

Results

Every error below is distance from Claude's score, not from a hiring decision. Against that target the model nearly halves the error of guessing the average, and the level gap between candidate and role drives most of it.

Level gapdiff_nivel
0.361
Years gapdiff_anos
0.188
Tech coveragecobertura_tech
0.144
Years of experienceanos_experiencia
0.068
LLM experiencellm_match
0.055
9 others
0.183
What drives the score: GradientBoosting feature importance, summing to 1. The level gap alone carries over a third.
Claude ≤30MAE 5.96 · n = 21
+4.8
Claude 31–50MAE 9.25 · n = 39
+5.1
Claude 51–70MAE 8.38 · n = 22
−3.1
Claude >70MAE 11.08 · n = 18
−11.0
Where it is wrong: mean of predicted minus Claude's score on the held-out pairs, by Claude's score. It lifts weak fits and pulls strong ones down; on the test split it never predicts above 78.5, while Claude goes up to 92.
Test error
8.70
mean absolute error on the 100 held-out pairs
Within 10 points
63%
of held-out predictions land within 10 points of Claude's score
Within 15 points
80%
the other 20% miss by more

What it gets wrong

  • It learns Claude, not recruiters. The ground truth is one Haiku 4.5 score per pair. I never measured whether Haiku gives the same pair the same score twice, so part of the 8 points may be noise in the labels themselves.
  • Claude scores in steps. Its 500 scores use only 24 distinct values, and its 8 most frequent (15, 28, 35, 42, 45, 52, 62, 72) take 81% of them, 404 of 500. The model is chasing a coarse target.
  • It pulls towards the middle. Strong fits, scored above 70 by Claude, come out 11.0 points lower on average. Regression to the mean, the price of a small dataset.
  • It learns Claude's quirks. Claude scores candidates two levels above the role slightly lower on average than those one level above (58.8 against 60.5), a hint that it penalises over-qualification. A model trained on those labels learns whatever quirks they carry; I did not measure whether it reproduces this one.
  • Small and synthetic. 500 invented profiles, not real people. A real version needs real hiring decisions to check against.

“The model learns Claude's scoring criterion, which may differ from actual recruiter judgment.”

Job Matcher README, Limitations

What I learned

A metric is only as honest as its labels. MAE 8.04 sounds like accuracy; it is distance from another model's opinion. Saying which one is half the result.

A number needs something to beat. On its own 8.04 means little; on the held-out pairs the model's 8.70 against the 15.59 of always guessing the average shows it learned something real.

The average error hides where the model is wrong. Splitting it by score showed the pull towards the middle that a single MAE never would.

Let's talk

Remote preferred. Based in Spain.