How the matching thresholds were calibrated

The package asserts a retraction from an exact identifier (DOI or PMID) with high confidence. For a reference that carries no identifier it falls back to title matching, and two thresholds govern that fallback:

This article reports a calibration of those two numbers against a labeled corpus, so they are evidence-based rather than guessed. The labeled corpus ships with the package in inst/extdata/calibration_corpus.csv, and the two scripts that build and analyze it (calibration_corpus.R, calibration_analysis.R) live in the data-raw/ directory of the source repository, so the study is reproducible from a repository checkout.

The labeled corpus

599 references in three groups:

200 exact-title retracted — records sampled from the Retraction Watch corpus, cited with their exact title (as a well-formatted bibliography would).
200 perturbed retracted — the same records with a realistic citation variation (about 15% of title words dropped, lower-cased), to probe how the thresholds behave on imperfect titles.
199 clean — non-retracted articles sampled from OpenAlex.

Every reference is matched by title, year, and author only (the DOI is withheld), which is exactly the hard case the thresholds govern.

Result 1: the assertion gate never false-accuses

At the title_exact gate (0.985 + year + first author):

Metric	Value
Precision	1.000
Recall	0.532
Clean references false-flagged	0 / 199 (0.000)

Flag rate by group: exact-title retracted 1.000, perturbed retracted 0.065, clean 0.000.

Reading: no clean reference was ever asserted as retracted (zero false accusations), and every exact-title retracted reference was recovered. The perturbed titles almost never clear the gate (0.065) — by design they fall to “possible” rather than being asserted, which is the conservative behavior we want. The overall recall of 0.532 is dominated by the perturbed group; on citations that reproduce the title faithfully, recall is 1.0.

Result 2: 0.90 is the empirical sweet spot for surfacing

Sweeping the fuzzy threshold and measuring how often a retracted reference is surfaced (as flagged or possible) versus how often a clean reference is wrongly surfaced:

Threshold	Retracted surfaced	Clean surfaced	Precision
0.84	0.980	0.774	0.718
0.86	0.958	0.437	0.815
0.88	0.945	0.035	0.982
0.90	0.885	0.000	1.000
0.92	0.790	0.000	1.000
0.94	0.688	0.000	1.000

0.90 is the lowest threshold at which no clean reference is surfaced while still recovering 88.5% of retracted references. Below 0.88 the clean-surfacing rate climbs steeply (43.7% at 0.86), which would bury real hits in false positives. Above 0.90 precision stays perfect but recall falls with no benefit.

Conclusion

The defaults are validated by this corpus:

0.985 title_exact gate: perfect precision, zero false accusations — the right posture for a tool that could otherwise mislabel a clean paper.
0.90 fuzzy threshold: the operating point where clean false-positives reach zero while retaining high recall.

These are heuristics, not calibrated probabilities, and the corpus is modest (599 references); the numbers should be revisited as the corpus grows and across non-English titles. But they show the current thresholds are conservative in the direction that matters: the package prefers “possible, please verify” over a false assertion.

Reproducing

The two scripts below are in the data-raw/ directory of the source repository (they are not part of the installed package). Run them from a repository checkout:

retraction_sync()                       # local corpus
source("data-raw/calibration_corpus.R")   # rebuild the labeled set (online)
source("data-raw/calibration_analysis.R") # recompute the tables above