Validation¶

Two generated reports back the faithfulness claim. Both live in validation/bw_faithfulness/derived/ and are included on this page verbatim, by transclusion, so the site cannot drift from the artifacts it describes.

What reruns offline, and what does not¶

The Buse-Weimer lane is self-contained. Its raw inputs are tracked in validation/bw_faithfulness/derived/arbitration_inputs/: the 100 rated Java snippets, oracle.csv (the raw 121-row annotator matrix), and scores.csv (the per-snippet means derived from it). extract.py, train.py, arbitrate.py and report.py read those pins as their primary source, so with no network and no cache/ directory they regenerate every number quoted below. A missing tracked input is a hard error and exit 1; there is no skip-and-pass path anywhere in the lane. When the original archive happens to be present in the gitignored cache/, the scripts cross-check every pin against it byte for byte, and any difference is a hard error rather than a silent re-pin. pinned.py is the input contract; fetch.py is not on this lane's path at all.

Those inputs can be tracked because an author of the dataset gave written permission to redistribute the snippets and annotator scores and to publish data derived from them. What was asked, what was answered, and how the message can be checked are in PERMISSIONS.md.

The cross-corpus lane is not self-contained and never will be. The Scalabrino et al. (2018) and Dorn (2012) corpora carry no permission of any kind, so not one byte of either is tracked, now or later. They are fetched at run time into the gitignored cache/, measured there, and only aggregates are published. Reproducing validation/breadth/ therefore needs a network fetch (fetch.py --all), even though the Buse-Weimer lane never does. Nothing on this page should be read as "the whole project regenerates from the git tree alone".

The arbitration loop¶

Ambiguities in the published Buse-Weimer feature definitions (how wide is a tab? do error fragments count?) are not resolved by taste. Each candidate interpretation ships as a PROVISIONAL ruling. A pre-registered experiment on the paper's original dataset arbitrates it, with the decision criteria written down before any result was seen. The winner is adopted by superseding the old ruling under a new immutable ID, the spec MAJOR version is bumped, and the faithfulness reproduction is re-run under the adopted rulings. Both reports below are the output of that loop.

To rerun it:

pip install -e ".[retrain]" -c constraints/ci.txt -c constraints/retrain.txt
python validation/bw_faithfulness/extract.py     # -> derived/features.csv
python validation/bw_faithfulness/train.py       # -> derived/train_results.json
python validation/bw_faithfulness/report.py      # -> derived/bw_faithfulness_report.{json,md}
python validation/bw_faithfulness/arbitrate.py   # -> derived/arbitration_report.{json,md}

BW 2010 faithfulness reproduction¶

Reproduction of the Buse-Weimer readability study with codecaliper's public snippet-granularity extractor (ARCHITECTURE.md §8.3).

Citing the original work¶

An author of the dataset asked that the work be cited as both papers; this project honours that requested citation form:

Raymond P. L. Buse and Westley Weimer, 'Learning a Metric for Code Readability', IEEE Transactions on Software Engineering 36(4):546-558, 2010, DOI 10.1109/TSE.2009.70
Raymond P. L. Buse and Westley Weimer, 'A Metric for Software Readability', ISSTA 2008:121-130, DOI 10.1145/1390630.1390647

Headline numbers¶

10-fold accuracy: 0.820 (bootstrap 95% CI [0.770, 0.870]) vs the paper's ~0.80; CI OVERLAPS the paper's figure
AUC: 0.828
fold accuracies: 0.90, 0.70, 0.80, 0.70, 0.90, 0.80, 0.80, 0.80, 1.00, 0.80
convergence warnings during fitting: 0
sign agreement vs Fig. 9: 21/24 agree, 3 disagree, 1 excluded (direction unclear in the paper)

Protocol¶

labels: label = 1 iff snippet mean score >= 3.14 (the paper's Figure 5 bimodal cutoff; scores below 3.14 are 'less readable') (paper Fig. 5 cutoff 3.14; 59 high / 41 low)
classifier: LogisticRegression(max_iter=1000); CV: StratifiedKFold(n_splits=10, shuffle=True, random_state=0)
AUC: roc_auc_score over cross_val_predict(method='decision_function') on the same folds
Deviation from the paper: (1) The paper evaluated a battery of classifiers (Bayesian classifier, logistic regression, multilayer perceptron, ...) under 10-fold cross-validation; its ~0.80 accuracy summarizes that setting. This reproduction runs logistic regression ONLY (LogisticRegression(max_iter=1000), raw unscaled features). (2) The paper repeats the entire 10-fold validation 10 times over fresh random partitionings and averages across runs; this reproduction uses ONE fixed-seed partitioning (random_state=0) for byte-reproducibility (ARCHITECTURE.md section 8.3).

Per-feature Spearman vs Fig. 9 directionality¶

Spearman rho of each feature against the snippet mean score; expected sign is the Fig. 9 direction of correlation with HIGH readability. A disagreement localizes a candidate divergence to a specific BW-/TOK- ruling. It is an empirical-arbiter input (ARCHITECTURE.md §8.3), not a failure to hide.

feature	Spearman rho	expected sign (Fig. 9)	rel. power	agree
avg_line_length	-0.544	-	0.96	yes
max_line_length	-0.412	-	0.78	yes
avg_identifiers	-0.660	-	1.00	yes
max_identifiers	-0.494	-	0.64	yes
avg_identifier_length	-0.535	unclear	0.00	excluded (unclear)
max_identifier_length	-0.239	-	0.40	yes
avg_indentation	-0.230	-	0.55	yes
max_indentation	-0.254	-	0.50	yes
avg_keywords	-0.331	-	0.55	yes
max_keywords	-0.206	-	0.13	yes
avg_numbers	-0.167	-	0.23	yes
max_numbers	-0.129	-	0.16	yes
avg_comments	+0.291	+	0.33	yes
avg_periods	-0.442	-	0.78	yes
avg_commas	-0.442	-	0.45	yes
avg_spaces	+0.039	-	0.21	NO
avg_parentheses	-0.619	-	0.93	yes
avg_arithmetic_ops	-0.230	+	0.07	NO
avg_comparison_ops	-0.335	-	0.21	yes
avg_assignments	-0.067	-	0.26	yes
avg_branches	-0.213	-	0.20	yes
avg_loops	-0.131	-	0.19	yes
avg_blank_lines	+0.262	+	0.53	yes
max_char_occurrences	+0.090	-	0.37	NO
max_identifier_occurrences	-0.285	-	0.41	yes

Extraction quality (reported, not tuned away)¶

parse_ok 29/100 snippets, 10 scaffolded (CORE-JAVA-0001), 0 with an EMPTY token vector. On parse errors BW token-family features are computed over the full lexical stream, ERROR subtrees included (BW-ALL-0007, bw-lexical-fallback diagnostic), which is why this count is zero; metrics remain error-opaque per CORE-ALL-0002. This is measured extractor behaviour on bare snippets, an arbitration outcome (see arbitration_report.md), never tuned away.

Dataset¶

100 snippets; 121 annotator rows in the archive vs 120 in the paper, reported as-is, never reconciled silently.
Permission (a grant, not a licence): an author of the dataset (W. Weimer) granted redistribution and derived-data publication by email 2026-07-11 (PERMISSIONS.md, dataset.toml). Under that grant the repository tracks the raw inputs this lane consumes: the 100 snippets, the per-annotator score matrix (oracle.csv) and the per-snippet mean ratings (scores.csv), all under derived/arbitration_inputs/. Every number reported here therefore regenerates from pinned tracked inputs with no network and no cache/. Re-downloading the archive is an optional byte-for-byte cross-check, never a prerequisite. This grant covers the Buse-Weimer corpus alone; the other readability corpora this project measures carry no permission and are never redistributed.

Anti-circularity¶

Anti-circularity note (stated in the paper too): ambiguity rulings are arbitrated on the same 100-snippet dataset whose reproduction accuracy we report. The reproduction is evidence of faithful operationalization of the published feature definitions, not an independent validation of BW's construct (ARCHITECTURE.md §8.3).

Provenance¶

codecaliper 0.2.1, spec 1.2.1
grammar: tree-sitter-java 0.23.5 (ABI 14, validated=True)
BW feature-order sha256: 8adae3992539a4eb2cf8d3b2386558612c77ad8e2e850f5bab940bbf430ad56d

Reading the two reports together: the faithfulness report's headline AUC (0.828, the final instrument run) and the arbitration report's adopted-cell AUC (0.827) are the same configuration. They differ by one ranked pair out of 2419 (about 0.0004), a matter of feature-serialization precision, as the arbitration report's reconciliation note details. 0.828 is the instrument's number.

BW arbitration experiment (A1 tab width / A2 arithmetic ops / A3 lexical fallback)¶

Empirical-arbiter loop, ARCHITECTURE.md §8.3. Every cell runs the exact train.py protocol; the baseline cell reproduced the committed pre-fallback training record (derived/arbitration_inputs/) exactly before the rest of the matrix was trusted.

Citing the original work¶

The experiment is scored against the Buse-Weimer dataset. An author of the dataset asked that the work be cited as both papers; this project honours that requested citation form:

Raymond P. L. Buse and Westley Weimer, 'Learning a Metric for Code Readability', IEEE Transactions on Software Engineering 36(4):546-558, 2010, DOI 10.1109/TSE.2009.70
Raymond P. L. Buse and Westley Weimer, 'A Metric for Software Readability', ISSTA 2008:121-130, DOI 10.1145/1390630.1390647

Pre-registered decision rule (verbatim)¶

Primary criterion: the number of sign agreements over the 24 clear-signed Fig. 9 features (fig9_signs.toml; avg_identifier_length is 'unclear' and excluded). Tie-break: AUC. A winner must hold under BOTH extraction modes to be adopted for the instrument. The lexical fallback is adopted iff it does not reduce sign agreements or AUC materially (its independent justification is construct fidelity + coverage 29/100 -> 100/100). Operationalization: within each mode the 16 (tab, ops) cells are ranked by (n_sign_agree desc, AUC desc); a candidate beats the current setting (tab=1, V0_current) iff it strictly increases n_sign_agree in at least one mode, never decreases n_sign_agree in either mode, and never decreases AUC by more than 0.01 in either mode; ties keep the current ruling. Among clearing candidates: highest summed n_sign_agree, then summed AUC, then fewer changed dimensions, smaller tab, earlier variant. 'Materially' for the fallback: at the adopted (tab, ops), fallback_on must not have fewer sign agreements than fallback_off and its AUC must not be lower by more than 0.01. No candidate clearing the bars => recorded null result, current rulings stand.

Extraction modes¶

fallback_off: derived/features_fallback_off.csv (pre-BW-ALL-0007 extraction, error-opaque token stream); empty-token vectors 8/100.
fallback_on: derived/arbitration_inputs/features_fallback_on_tab1.csv (extract.py with BW-ALL-0007 implemented: full lexical stream on parse errors; tab=1, spec 0.1.0); empty-token vectors 0/100 (parse_ok 29/100, BW-ALL-0007 gives the other snippets a full lexical stream).

Ops variants (Java operator sets)¶

V0_current: % * + - /
V1_minimal: * + - /
V2_incdec: % * + ++ - -- /
V3_compound: % %= * *= + += - -= / /=

Matrix (n_sign_agree / AUC / accuracy)¶

fallback_off¶

tab \ ops	V0_current	V1_minimal	V2_incdec	V3_compound
tab=1	20/24, auc 0.7834, acc 0.720	20/24, auc 0.7830, acc 0.720	20/24, auc 0.7830, acc 0.720	20/24, auc 0.7830, acc 0.720
tab=2	21/24, auc 0.7854, acc 0.720	21/24, auc 0.7859, acc 0.720	21/24, auc 0.7859, acc 0.720	21/24, auc 0.7859, acc 0.720
tab=4	21/24, auc 0.7929, acc 0.770	21/24, auc 0.7929, acc 0.770	21/24, auc 0.7929, acc 0.770	21/24, auc 0.7929, acc 0.770
tab=8	21/24, auc 0.7983, acc 0.770	21/24, auc 0.7974, acc 0.770	21/24, auc 0.7987, acc 0.770	21/24, auc 0.7979, acc 0.770

fallback_on¶

tab \ ops	V0_current	V1_minimal	V2_incdec	V3_compound
tab=1	20/24, auc 0.8297, acc 0.790	20/24, auc 0.8297, acc 0.790	20/24, auc 0.8305, acc 0.790	20/24, auc 0.8297, acc 0.790
tab=2	21/24, auc 0.8260, acc 0.800	21/24, auc 0.8260, acc 0.800	21/24, auc 0.8251, acc 0.800	21/24, auc 0.8260, acc 0.800
tab=4	21/24, auc 0.8210, acc 0.800	21/24, auc 0.8214, acc 0.800	21/24, auc 0.8210, acc 0.800	21/24, auc 0.8210, acc 0.800
tab=8	21/24, auc 0.8272, acc 0.820	21/24, auc 0.8280, acc 0.820	21/24, auc 0.8272, acc 0.820	21/24, auc 0.8276, acc 0.820

Sign disagreements per cell¶

cell	disagreements
fallback_off/tab=1/V0_current	avg_indentation, avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=1/V1_minimal	avg_indentation, avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=1/V2_incdec	avg_indentation, avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=1/V3_compound	avg_indentation, avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=2/V0_current	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=2/V1_minimal	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=2/V2_incdec	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=2/V3_compound	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=4/V0_current	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=4/V1_minimal	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=4/V2_incdec	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=4/V3_compound	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=8/V0_current	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=8/V1_minimal	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=8/V2_incdec	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_off/tab=8/V3_compound	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=1/V0_current	avg_indentation, avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=1/V1_minimal	avg_indentation, avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=1/V2_incdec	avg_indentation, avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=1/V3_compound	avg_indentation, avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=2/V0_current	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=2/V1_minimal	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=2/V2_incdec	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=2/V3_compound	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=4/V0_current	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=4/V1_minimal	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=4/V2_incdec	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=4/V3_compound	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=8/V0_current	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=8/V1_minimal	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=8/V2_incdec	avg_spaces, avg_arithmetic_ops, max_char_occurrences
fallback_on/tab=8/V3_compound	avg_spaces, avg_arithmetic_ops, max_char_occurrences

Decision (per-dimension, with disclosed deviation)¶

DISCLOSED DEVIATION from the pre-registered operationalization (the decision rule itself is unchanged): the joint (tab, ops) preference chain put summed AUC before parsimony, allowing a dimension with zero primary-criterion evidence to change on ~4e-4 summed-AUC noise, contradicting the rule's own 'ties keep the current ruling / a spec change requires positive evidence' clause. Dimensions were therefore evaluated marginally; the joint chain's outcome is recorded here for transparency.

Pre-registered joint chain would pick: tab=8, V2_incdec (ops component carried by AUC noise only).
A1 clearing tab widths (ops at V0_current): [2, 4, 8].
A2 clearing ops variants (tab at adopted): none; max AUC spread across variants within any (mode, tab) row: 0.001240.
A3 at the adopted setting: sign agreements on/off = 21/21, AUC on/off = 0.8272/0.7983.

Recommendation¶

tab_width = 8
ops_variant = V0_current
lexical fallback (BW-ALL-0007): adopt

A1 ADOPTED: tab=8. Every tab>=2 fixes the avg_indentation sign disagreement in BOTH extraction modes (+1 sign agreement, the primary criterion); among clearing tab widths [2, 4, 8], tab=8 is the (n_sign_agree, AUC) top cell in both modes (the pre-registered tie-break). It also moves max_indentation's Spearman rho from ~0 to clearly negative, matching Fig. 9. A2 NULL RESULT: no ops variant changes ANY Fig. 9 sign in any of the 32 cells (avg_arithmetic_ops disagrees with the paper's near-zero-power positive bar everywhere), and the AUC spread across variants within any (mode, tab) row is <= 0.00124018189334, within noise. The current ruling stands (BW-ALL-0006, Java arithmetic_ops as-is). A3: at the adopted setting, fallback_on has 21/24 sign agreements vs 21/24 and AUC 0.827201322861 vs 0.798263745349: no material reduction, so BW-ALL-0007 is ADOPTED. Its independent justification is construct fidelity (the original BW instrument was grammar-less) and coverage (empty-token vectors 8 -> 0, full token streams 29/100 -> 100/100).

Scoping notes¶

Tab dimension: only avg_indentation/max_indentation were re-derived; avg_line_length, max_line_length and avg_spaces remain raw character counts (a tab is 1 character there), a possible future arbitration.
Ops dimension: avg_arithmetic_ops for V1-V3 was recomputed from a DIRECT parse of the raw snippet (lex(include_error_tokens=True) for fallback_on, plain lex() for fallback_off); the instrument itself may engage the CORE-JAVA-0001 scaffold at snippet granularity. V0_current cells keep the matrix's own column (the true instrument path); the 'approximation' block reports recomputed-V0 vs that column.
V3_compound counts compound assignments (+= -= *= /= %=) in avg_arithmetic_ops while avg_assignments (unchanged base column) still counts them too; adopting V3 in the instrument would additionally require a precedence decision in BW-ALL-0006.
Anti-circularity (README.md): these rulings are arbitrated on the same 100-snippet dataset whose reproduction accuracy is reported; the reproduction is evidence of faithful operationalization, not an independent validation of BW's construct.
The baseline cell (fallback_off, tab=1, V0_current) was asserted to reproduce the committed pre-fallback training record exactly (fold accuracies, CI, AUC, per-feature Spearman, sign table) before the rest of the matrix was computed.
Reconciliation with the final instrument run: the adopted cell's AUC (fallback_on/tab=8/V0, 0.827201322861) differs from the headline AUC of the final re-extraction (derived/train_results.json, 0.827614716825, first produced under spec 1.0.0 and byte-identical on every number when re-stamped under spec 1.1.0) by exactly one ranked pair (1/2419 = 0.000413): the matrix splices full-precision recomputed indentation values into the feature array, while the instrument's train.py consumes the canonical 12-significant-digit features.csv. The feature VALUES agree under 12-significant-digit quantization for all 100 snippets: the gap is serialization precision flipping one near-tied AUC pair, not a semantic difference. 0.828 (the final run) is the instrument's number.

Cross-corpus parse anatomy¶

The breadth lane measures three Java corpora with the shipped public API and reports what the parser met. It is the reason the BW lane is careful about error recovery: most snippets in the readability corpora are truncated fragments of larger units, not broken code.

corpus	N	clean parse	at least one tab	recovered	of those, brace-imbalanced
Buse-Weimer (2010)	100	27 (27.0%)	32	73	69
Scalabrino et al. (2018)	200	190 (95.0%)	151	10	0
Dorn (2012), Java only	121	19 (15.7%)	108	102	76

These aggregates are spec- and grammar-dependent: they hold for the versions codecaliper env prints in this tree, and a change that moves them is a spec event. Reproducing them needs the archives, hence a network fetch. See validation/breadth/README.md.

Why this page says 73 and the Quickstart says 71¶

Both counts are of the same 100 Buse-Weimer snippets under the same grammar, and they differ because the two lanes measure at different granularities.

The breadth lane calls measure(src, language="java") and takes the default granularity="file". The CORE-JAVA-0001 snippet scaffold is gated on granularity == "snippet" (api._measure), so at file granularity it cannot fire: a snippet either parses as a bare program or it does not. 27 do, 73 are recovered, and that is the row above.

The faithfulness lane calls measure(src, language="java", granularity="snippet") (validation/bw_faithfulness/extract.py), where the scaffold does fire. It is adopted on 10 of the 100 snippets, and on 2 of those 10 the scaffolded parse is clean. So 29 snippets reach parse_ok: true and 71 keep parse_ok: false, which is the number the Quickstart breaks down. The two-snippet gap between the pages is the scaffold, and nothing else.