My day job is building voice agents that answer phone calls for medical clinics. My Master’s research at Unitec looks at the same signal from the other side: not what a patient says on a call, but what their voice might reveal about their health.
The specific question is Parkinson’s disease. Voice changes (reduced loudness, monotone pitch, breathy or hoarse quality, imprecise articulation) appear in a large majority of people with Parkinson’s, often early, and a phone microphone can capture them. That makes voice an appealing screening signal: cheap, non-invasive and available anywhere.
The published literature looks very promising. Many models report accuracy above 90%. And then, when they’re tested on data from a different study, they commonly lose 10 to 30 points.
This post is about why that happens and how I’ve designed my thesis to measure it honestly.
Why in-dataset accuracy is misleading
Three problems come up again and again in the voice-PD literature.
1. The same person in training and test. Many datasets contain several recordings per participant. If recordings are split randomly, the model sees one recording from a person in training and another in testing. It can learn to recognise the speaker rather than the condition, and the accuracy reflects that.
2. One recording setup. A dataset collected with one microphone, in one room, by one protocol bakes those conditions into every sample. A model can pick up on channel and environment differences that happen to correlate with labels.
3. One population. Most data comes from one country, one language and a narrow age range. Voice varies with all of them. A model that has only heard one population has no reason to work on another.
None of these show up if you train and test within the same dataset. All of them show up the moment the model meets new people.
The evaluation design
Train once, on one source
Models are trained on mPower, a large smartphone-based study where participants recorded sustained vowel phonation at home. It’s big, it’s realistic about recording conditions, and it’s predominantly English-speaking participants in the US.
Split by subject, always
Cross-validation is done at the subject level: every recording from a participant goes into the same fold. This alone removes the speaker-identity shortcut, and in the literature it’s one of the changes that most reduces reported in-dataset performance. That reduction isn’t a failure; it’s the real number.
Freeze, then test elsewhere
After training, the model and its decision threshold are frozen. They’re then evaluated, with no retraining and no re-tuning, on three external datasets:
- Bridge2AI-Voice: English, clinically collected, a different protocol and population from mPower. Access is controlled, and the ethics application for it was a project in itself.
- PC-GITA: Spanish speakers from Colombia.
- IPVS: Italian speakers.
That gives a change in protocol, a change in language, and a change in both, so the drop can be attributed rather than just observed.
Accuracy isn’t the right question
Even a fair accuracy number doesn’t answer the question a clinic would actually ask: if we use this to decide who gets referred, what happens?
Two things answer that better.
Calibration
A model that outputs 0.8 should be right about 80% of the time it says 0.8. Many models aren’t: they’re over-confident, and they become more miscalibrated on external data. A badly calibrated model can still have a high AUC while producing probabilities that mean nothing.
I report reliability curves and expected calibration error in-domain and on each external dataset, and test whether simple recalibration, fitted only on training data, holds up after the shift.
Decision curve analysis
Decision curve analysis asks, for each threshold a clinician might choose, whether using the model gives more net benefit than the simple strategies of referring everyone or referring no one. It makes the trade-off between false positives (unnecessary referrals, anxiety) and false negatives (missed early diagnosis) explicit.
Of the voice-PD studies I surveyed for the proposal, none reported calibration or decision curves. That’s the gap the thesis is aimed at.
Why an engineer cares about this
This research reads differently because of my day job. In production, the question is never “what’s the accuracy on the benchmark?” It’s “what happens when this meets real traffic that doesn’t look like the test set?” Distribution shift is the normal condition of any deployed system, not an edge case.
The habits carry over directly:
- Separate the thing you tune on from the thing you test on, and don’t let them leak.
- Measure behaviour on the population you’ll actually serve, not the one you happened to collect.
- Report numbers in terms of decisions and their costs, not just scores.
I’ll write up results as the thesis progresses. For now, the design is the contribution: a way of evaluating voice biomarker models that tells you what you’d actually be deploying.