Identity benchmark

Resolving a component name written one way to the identifier an advisory database uses. Four baselines and this resolver, scored on the same rows by the same scorer.

Our F1 is 0.1548. The best baseline is 0.1576.

We lose on the headline metric. The result is published because a benchmark whose author always comes first is marketing, and worth nothing as evidence.

What we win on is the column on the right. Across all rows this resolver assigns an identifier to 0 of the 250 unknown-class rows. prefix-strip assigns one to 4. In a tool whose output is submission evidence, a confident wrong identifier produces a clean vulnerability verdict for a component nobody actually identified, and that is a worse failure than declining to answer.

Those 250 rows are not 250 components. 36 are real components from the corpus that carry no identifier at all. The other 214 are path-stripped names built from components that do have one, where the identifier cannot be recovered from the final segment alone. Both are rows a resolver must decline, and the smaller number is the one to quote.

What the columns mean

Precision: Of the answers given, how many were right. An abstention is never counted as a wrong answer.
Ours is 1.000 on every subset. That is the number this product is built around.
Recall: Of the rows with a right answer available, how many were answered correctly.
F1: The harmonic mean of the two, which weights them equally.
We do not think they are equal here, and saying so while losing on F1 is the only honest order to say it in.
Abstained: How often the resolver declined to answer. High abstention is the cost of high precision, and it is a deliberate one.
Wrong on unknowns: How many rows with no correct identifier were nonetheless assigned one.
The benchmark has an unknown class for exactly this. A resolver that always guesses scores well on the other columns and fails here.

Clean

Names exactly as their tools wrote them, with no perturbation applied.

Resolver	Precision	Recall	F1	Abstained
always-abstain	0.000	0.000	0.0000	100.0%
exact-match	0.976	0.085	0.1571	91.6%
normalised-match best baseline, tied	0.977	0.090	0.1641	91.2%
prefix-strip best baseline, tied	0.977	0.090	0.1641	91.2%
StratifyPro	1.000	0.087	0.1609	91.6%

Perturbed

The same names after one of seven documented shape changes: an archive extension, a vendor prefix, a parenthetical, a case change, added spaces, a version, or a stripped path.

Resolver	Precision	Recall	F1	Abstained	Wrong on unknowns
always-abstain	0.000	0.000	0.0000	100.0%	0
exact-match	0.000	0.000	0.0000	100.0%	0
normalised-match	0.942	0.069	0.1278	93.4%	4
prefix-strip best baseline	0.952	0.084	0.1546	92.0%	4
StratifyPro	1.000	0.082	0.1520	92.5%	0

All rows

Both of the above together. This is the headline set.

Resolver	Precision	Recall	F1	Abstained	Wrong on unknowns
always-abstain	0.000	0.000	0.0000	100.0%	0
exact-match	0.976	0.027	0.0519	97.5%	0
normalised-match	0.955	0.075	0.1392	92.7%	4
prefix-strip best baseline	0.960	0.086	0.1576	91.7%	4
StratifyPro	1.000	0.084	0.1548	92.2%	0

Reproducing this

The dataset is generated, not committed by hand: python3 bench/identity/build.py rebuilds it and --check fails if it would change. scripts/bench-publish.py produces this page’s numbers by emitting one prediction per row and scoring them with bench/identity/run.py, the same scorer that produced the baselines. There is deliberately no second scorer.