Ultrasound radiomics in breast neoadjuvant therapy

An evidence synthesis collects the results of several published studies and asks what they say together. Doing this by hand is slow and hard to audit. Here an AI agent did the reading and wrote the code, but the code it produced runs without any AI: it reads saved copies of the articles, applies documented rules and always gives the same answer. Each step below has one job, and every number can be traced back to the sentence or table it came from.

AI agents selected the studies, wrote the specification and built the pipeline under my direction. The building agent read the four studies, documented the extractions and implemented the checks, following the written specification (AGENTS.md). The pipeline it produced is deterministic and offline: it uses no LLM or API at runtime, so the same inputs always regenerate the same scores, deltas and classifications.

Every second-pass check carries one of four labels:

Neoadjuvant therapy is chemotherapy (sometimes with targeted drugs) given before breast surgery. Knowing early whether a tumour is responding helps doctors adjust treatment. Radiomics turns medical images into hundreds of quantitative features describing shape, brightness and texture, which a model can then use for prediction. The question is not whether radiomics models score well, but whether they add something beyond what doctors already know from clinical data and a standard ultrasound read.

Does ultrasound-based radiomics provide incremental predictive value for predicting response to neoadjuvant therapy in breast cancer beyond clinical and conventional ultrasound-based models?

Secondary question: how consistent is performance across imaging timepoints, radiomics strategies, model components and validation designs?

This is a proof of concept: four studies were prespecified, with no comprehensive search, screening or automatic meta-analysis.

Four open-access studies were chosen in advance, so this is a demonstration of method rather than a full review of the literature. They differ in size, in how patients were split for model building and testing, and in whether the model was tested at a second hospital. The bars show how many patients each study used for training (building the model), internal validation (testing on held-out patients from the same hospital) and external validation (testing at a different hospital, the strongest check).

A synthesis is only as reliable as its sources. Each article is stored locally and fingerprinted with a SHA-256 hash, a code that changes if even one character of the file changes. This means the pipeline can prove it is reading exactly the same documents on every run, and it can regenerate everything without an internet connection.

All four primary full texts are archived and checked for DOI identity, document completeness and hash. Numerical table transcriptions are checked by an independent parser; narrative facts have reviewed source anchors. Exact source selectors are exported in the provenance table.

An endpoint is the outcome a model tries to predict. The four studies use three different ones. RECIST measures how much the tumour shrinks on imaging. Miller–Payne grades how many tumour cells are left under the microscope after surgery; grades 4–5 count as a major histological response (MHR). Pathological complete response (pCR) means no invasive cancer remains in the breast or lymph nodes. These are related but not interchangeable, so a model that predicts one well cannot be assumed to predict another.

The four studies use three different response targets. Binary coding is an analysis convenience and does not make them equivalent.

Ultrasound images can be taken before treatment (baseline), after the first cycles (early-treatment), or both. Delta radiomics measures how the features change between the two scans, and longitudinal models combine both timepoints. Later images contain more information about the response already under way, so comparing a baseline model with an early-treatment model is not a like-for-like comparison. The pipeline labels every result with its timepoint so these differences stay visible.

“Early” means after two cycles in Yang and Wang; Wu’s midpoint is not assumed to mean two cycles. Baseline, delta and longitudinal information are kept separate.

Extraction means reading each article and filling in a structured form: who was studied, how images were acquired and segmented (the tumour outlined on the image), which features and algorithms were used, and how models were tested. The form has 54 fields per study. When an article does not report something, the field is marked NR (not reported) instead of being guessed or set to zero, so gaps in the literature remain visible.

Each study is extracted against a 54-field schema covering design, population, reference standard, ultrasound acquisition, segmentation, radiomics, models and validation. Missing fields stay visible.

The main performance measure is AUC (area under the ROC curve). It describes how well a model ranks responders above non-responders: 0.5 is a coin toss, 1.0 is perfect. The 95% confidence interval (CI) shows the range of plausible values given the sample size; wide intervals mean the estimate is uncertain. Training results usually look better than validation results because the model has already seen those patients, so the validation columns are the ones that matter most.

Reported AUC (95% CI) by model and dataset role. Values marked “review” are provisional transcriptions with an open source discrepancy.

Zhou 2024 also reports nine additional radiomics-only algorithms (LR, NB, SVM, KNN, RF, ET, LGBM, AdaBoost, MLP); their internal-validation AUCs range from 0.447 to 0.694. All rows, with sensitivity, specificity, calibration and provenance, are in the full report.

Incremental value asks whether adding radiomics improves on a simpler model, for example clinical data alone. ΔAUC is the difference between two models’ AUCs on the same patients. A positive ΔAUC is not proof by itself: the difference could be due to chance. The DeLong test is the standard statistical test for comparing two AUCs measured on the same patients, and a low p-value (typically below 0.05) suggests the difference is real. Each comparison is classified according to how strong the supporting evidence is.

Seven comparison families are searched in every study: (A) radiomics vs clinical-only, (B) radiomics vs conventional ultrasound, (C) combined vs clinical, (D) early vs baseline, (E) delta vs baseline, (F) radiomics+deep learning vs radiomics-only, (G) fusion vs components. Equivalent contrasts prioritise external validation → internal validation → test → training.

Primary matched contrasts: 4 Demonstrated, 5 Suggestive, 2 Not demonstrated, 4 Indeterminate. Contrasts within a study are correlated and are not independent study counts. Families without an isolated, source-supported contrast (e.g. B and F in every study) are Indeterminate.

Study design affects how much a result can be trusted. Single-centre studies with one scanner may not generalise to other hospitals. Manual tumour outlining depends on the operator. Feature selection done before splitting the data can let information from test patients leak into the model and inflate performance. Rather than labelling studies “high” or “low risk”, the pipeline records what each article does or does not report, so readers can judge for themselves.

Considerations are recorded neutrally as observed, unclear, not reported or not applicable. The pipeline does not assign automatic “high risk” judgements.

Before results from different studies can be combined, they need to be measuring the same thing in similar patients. Each pair of studies is compared on nine criteria, such as endpoint, treatment, imaging timing and validation design. Each criterion is scored aligned, partially aligned or not aligned, and criteria that cannot be judged because information is missing are left out rather than counted against the pair. The score summarises similarity; it is not by itself permission to pool.

Nine criteria per pair: endpoint definition, regimen, timepoint, imaging modality, radiomics strategy, comparator, validation, population, subtype. aligned = 1, partially aligned = 0.5, not aligned = 0; unavailable criteria are excluded from the denominator. High ≥ 0.75, moderate ≥ 0.50, low < 0.50. Scores describe alignment and do not establish poolability.

Populations are partially aligned across all pairs: every study includes breast cancer patients receiving NAC, with differences in eligibility, completion and surgery requirements. Subtypes are partially aligned: all include hormone-receptor, HER2 and triple-negative categories, with distributions that differ or are incompletely reliable.

Pooling means statistically combining results from several studies into one estimate, as in a meta-analysis, usually shown as a forest plot. It only makes sense when studies predict the same outcome, with the same kind of model, at the same imaging timepoint, tested in the same way. The pipeline checks whether at least two independent studies share all of these features. If none do, no pooled estimate is produced, because a single number would hide real differences.

No joint stratum contains two independent studies with the same endpoint, imaging role, model composition and validation role. The only pCR pair (Wu and Zhou) compares baseline radiomics against pre+midpoint radiomics, and its operational pathology definition and source precision also need review. Strata were checked separately for baseline, early-treatment, delta and pCR studies; each contains a single study.

No forest plot or quantitative meta-analysis was generated.

This is the summary the pipeline “froze” before looking at any external review. Freezing means the conclusions were saved with a fingerprint, so they cannot be quietly adjusted later to match someone else’s findings. Each point is limited to what the four studies actually support.

• —Four prespecified studies; three endpoint categories (RECIST, Miller–Payne MHR, pCR). Not an exhaustive systematic review.
• —Zhou 2024 provides a direct clinical+radiomics versus clinical-only contrast on baseline imaging, supported by a verified DeLong test (Figure 5H), within one small internal validation cohort.
• —Wu 2025 validates multimodal models on an external center, but longitudinal information, clinical and deep features complicate attribution to radiomics alone; table/prose discrepancies leave effect sizes provisional.
• —Wang 2025: exact validation AUCs and the dataset attribution of its tests remain unresolved, so validation incremental value is Indeterminate.
• —Yang 2022 separates early and delta from baseline radiomics; training tests are not borrowed for validation contrasts.
• —18 REVIEW_REQUIRED checks remain visible. Clinical usefulness beyond discrimination is not established by AUC differences alone.

Published articles sometimes contradict themselves, for example when a number in the abstract differs from the same number in a table. The pipeline does not choose a version on its own. It records both, explains which results depend on the value, and flags the item as REVIEW_REQUIRED for a human to check. Making this list visible is part of the result: it shows where the evidence is less certain than it first appears.

Unresolved discrepancies between sections of the same article are kept with both versions, the impact on the analysis, and the number of resolution attempts (stopped after two when no new evidence emerged).

A benchmark is an independent reference to compare against. Here it is a 2024 systematic review with meta-analysis on the same topic. It was read only after the synthesis was frozen, to avoid being influenced by it. Sensitivity is the share of true responders a test correctly identifies; specificity is the share of non-responders correctly identified. RQS (Radiomics Quality Score) is a checklist that rates the methodological quality of radiomics studies.

Benchmark: Ultrasound-based radiomics for early predicting response to neoadjuvant chemotherapy in patients with breast cancer: a systematic review with meta-analysis (2024; DOI 10.1007/s11547-024-01783-1; PMID 38630147). Only the publisher abstract and preview were accessible.

The review searched four databases through 1 January 2023 and included eight studies. Mean RQS was 12.88 (35.8% of the maximum). Pooled sensitivity was 0.87 (95% CI 0.81–0.92) and specificity 0.78 (0.72–0.83). The authors describe potential predictive usefulness and call for rigorous multicenter prospective confirmation.

The review predates the two 2025 studies and reports pooled accuracy, not a pooled incremental contrast against clinical or conventional ultrasound models. Its findings support cautious interpretation of combined-model performance and do not change the primary synthesis.

Automation makes analysis faster and repeatable, but it does not replace expert judgement. This section lists what the pipeline cannot do, so its results are read with the right level of confidence. The commands at the bottom regenerate every output from the saved sources.

• —Four protocol-selected studies; no exhaustive search or screening.
• —Narrative transcriptions and semantic mappings were curated by the building agent. Deterministic source checks do not replace independent clinical appraisal.
• —Source contradictions, missing exact validation metrics and uncertain feature-selection nesting remain open.
• —No patient data, model fitting, invented confidence intervals or imputed confusion matrices.
• —Previous benchmark exposure prevents a claim of blind synthesis.
• —The workflow reproduces reviewed transcriptions, interpretations and calculations. It is not a general autonomous extractor for new articles, and repeated execution cannot repair source contradictions.

Regenerate from the project folder with the existing virtual environment: