Text to speech has improved enormously. A modern neural voice will pronounce almost every word correctly, sound clean at the segment level, and avoid the artefacts that characterised earlier generations of synthesis. And yet many synthetic voices are still recognisable within a sentence or two, not because any sound is wrong, but because the sentence as a whole does not appear to mean anything in particular. It is read rather than said.
The difference lives in prosody: the pitch, timing, loudness and voice quality that sit above the individual sounds and carry a substantial part of what a speaker intends. Prosody is where synthesis remains genuinely unsolved, and on a phone call, where the listener has no face to read, it is also where most of the communicative burden falls.
What prosody actually consists of
Prosody is not a single property. It is usually decomposed into several interacting dimensions:
- Pitch, or fundamental frequency (F0). The rate of vocal fold vibration, perceived as how high or low the voice is. What matters is rarely the absolute value but the contour: where the pitch rises, where it falls, how steeply, and on which syllable the movement is anchored.
- Duration and rhythm. How long each segment, syllable and word lasts, including the systematic lengthening that tends to occur at the end of a phrase. Rhythm is the resulting pattern of relative timing.
- Intensity. Loudness, which usually co-varies with pitch and duration rather than acting alone.
- Pauses. Their placement is one of the strongest cues to phrase structure, and their length signals how large a boundary is.
- Voice quality. Breathiness, creak, tension. These carry stance and emotion, and they are the dimension least often modelled explicitly.
These combine into what phonologists describe as intonational structure: a grouping of words into phrases, with prominences inside them and boundary behaviour at their edges. The autosegmental metrical tradition, and the ToBI annotation conventions derived from it, formalise this for English and have been adapted with varying success elsewhere.
What prosody is for
Prosody is not decoration. It does specific linguistic work.
Sentence type. In many varieties of English, a polar question can be signalled by a final rise where a statement would fall, so that "you are coming tomorrow" becomes a question purely through intonation.
Contrastive focus. The canonical demonstration is one sentence read with the stress moved:
- I did not say she took the money. (Someone else said it.)
- I did not say she took the money. (I implied it.)
- I did not say she took the money. (Someone else took it.)
- I did not say she took the money. (She borrowed it.)
- I did not say she took the money. (She took something else.)
The words are identical. The meaning differs entirely, and the only carrier of the difference is which word receives prominence. No amount of correct pronunciation recovers this.
Phrasing. Where a speaker places a boundary can resolve an attachment ambiguity. "We invited the manager, from the Pune office, and the auditor" describes two people or three depending on the grouping, and the grouping is audible.
Turn management. Falling pitch, final lengthening and a drop in intensity together tend to signal that a speaker has finished; sustained pitch tends to signal that they have not. A synthetic voice with the wrong final contour invites interruption or produces awkward silence.
Discourse structure and stance. Speakers commonly reset pitch higher at the start of a new topic and compress their range within one, which helps a listener track a longer explanation. Range, tempo and voice quality also convey deference, confidence or warmth, in culturally variable ways that are easy to get wrong.
Text underdetermines prosody
Here is the central difficulty. Given a written sentence, there is no single correct prosodic realisation. There are many, each appropriate to a different context, and the text does not say which applies.
This is the one-to-many mapping problem. During training, a model sees the same or similar text paired with recordings that differ in phrasing and emphasis. If the training objective is to minimise average reconstruction error over those recordings, the lowest-loss prediction is something close to the conditional mean of the distribution. The conditional mean of several distinct, expressive readings is a reading that is smooth, moderate and specific to nothing.
This framing explains why models with explicit stochastic or generative objectives tend to produce more varied prosody than regression-style predictors, and why adding data does not necessarily help: more data may simply sharpen the estimate of an average that was never the target.
A second, related problem is that the information needed to choose the right reading is often absent from the input entirely. Contrastive focus depends on what was said earlier. A sentence handed to a synthesiser in isolation has been stripped of exactly the context that determines its prosody.
How systems try to regain control
The research literature offers several broad families of approach. All of them are ways of supplying the missing information.
Explicit markup. SSML and similar schemes let an author annotate emphasis, breaks, rate and pitch directly. This is predictable, auditable and well suited to fixed content, but it does not scale to freely generated text, and support for the finer tags varies across implementations.
Reference and style encoders. A model can be conditioned on an embedding derived from a reference recording, learning a latent space of speaking styles. Global style tokens, introduced in 2018, are a widely cited instance: the model learns a small set of style components without explicit labels. The difficulty is that such spaces entangle style with speaker identity and recording conditions, so transferring a style across speakers is not straightforward.
Prosody transfer. A stronger version copies the prosodic contour of a reference utterance onto new text. It works well when reference and target are similar in length and structure, and degrades when they are not.
Explicit prosody prediction. Some architectures predict pitch, energy and per-phoneme duration as intermediate, inspectable variables, then condition the acoustic model on them; FastSpeech 2 is the best-known example. The advantage is editability: predicted values can be adjusted before synthesis, making targeted emphasis achievable without retraining.
Richer conditioning. Conditioning on surrounding text, dialogue history or representations from a language model supplies some of the context that determines the correct reading. This is an active research direction and, in principle, the most promising, since it attacks the underdetermination rather than working around it.
Indic prosody has its own requirements
Most prosodic research and annotation infrastructure was developed for English. The following are offered as tendencies reported in the descriptive literature rather than settled facts.
Phrasing in Hindi. Hindi is verb-final, with relatively free ordering of preverbal constituents. Descriptive accounts commonly report a recurring rising pattern across successive phrases with a final fall on the verb. Since constituent order carries information structure, prosodic phrasing interacts with word order in ways that have no direct English analogue.
Questions. Polar questions may be marked lexically with kya, or left unmarked and carried by intonation alone. A synthesiser reading aap kal aa rahe hain must decide from context whether that is a statement or a question, and the text may not say.
Rhythm in Indian English. Indian English is frequently described as more syllable-timed than British or American English, with less vowel reduction in unstressed syllables and different word stress in some items. This is a broad generalisation over very diverse varieties and differs by region, first language and register. The practical implication is that prosodic targets copied from a British or American model may sound foreign to an Indian listener even when every phoneme is correct.
Other languages. For Gujarati and most other Indian languages, comparable descriptive work is thinner and annotated prosodic corpora remain scarce. That scarcity, rather than any linguistic obstacle, is often the binding constraint.
Evaluating something you cannot measure directly
Prosody is difficult to evaluate, and the difficulty is frequently underestimated.
- Word error rate is silent on prosody. Running a recogniser over synthetic audio tells you whether the words are intelligible, and a completely flat reading transcribes perfectly. Intelligibility metrics can improve while expressiveness gets worse.
- Mean opinion score asks listeners to rate naturalness on a scale. It is the standard instrument, but scores are not comparable across studies, are sensitive to the stimulus set and listener pool, and compress once systems are all reasonably good.
- Preference tests on the same sentence are generally more sensitive for prosody, since listeners find relative judgements easier than absolute ones.
- Objective measures such as correlation between predicted and reference F0 contours, or per-phoneme duration error, are useful for detecting regressions. They are not measures of appropriateness: a contour can match a reference closely and still be the wrong reading for the context.
- Task-specific tests are often the most informative. Present listeners with a context and ask which of two renderings answers it. That measures whether the prosody communicates, which is the actual requirement.
Why this matters on a telephone call
On a call, prosody is doing work that has commercial consequences.
A caller who cannot hear which word is emphasised must reconstruct meaning from words alone, which is slower and more error prone across a narrowband channel that has already discarded much of the spectrum. A confirmation that should have contrasted one option against another lands ambiguously, and the caller asks again. A final contour that fails to signal the end of a turn produces either an interruption or a pause that feels like a fault on the line. A uniformly flat delivery across a long explanation gives the listener no signal about what is important, so nothing is.
Listeners also form judgements about competence and sincerity from prosody very quickly. A voice that sounds indifferent while delivering good news, or brisk while apologising, produces a mismatch people notice even when they cannot say what was wrong.
What to do about it in practice
Several things are achievable without waiting for the research to converge.
- Preserve context. Prosody depends on what came before. Architectures that hand a synthesiser isolated sentences discard the information it needs, and the loss is irreversible downstream.
- Write for speech. Shorter sentences with a single clear point of focus have fewer plausible readings to average over. Long subordinate structures invite mistakes in phrasing.
- Treat critical utterances differently. Amounts, confirmations, names and negations are where wrong emphasis costs most, and they deserve attention rather than the default reading.
- Test with listeners from the target audience. Prosodic appropriateness for an Indian English or Hindi listener is not something a metric developed on American English corpora reports accurately.
- Watch for regression. Work aimed at intelligibility, latency or stability can quietly reduce expressiveness, and no automatic metric in common use will flag it.
The field is moving towards systems that condition prosody on discourse context rather than on the sentence alone, which is the right attack on the underlying problem. Until that is routine, prosody has to be deliberately engineered and deliberately evaluated. At Cirio, we treat how something is said as part of whether it was understood, because on a phone call those are not separate questions.
Frequently Asked Questions
What is prosody in speech synthesis?
Prosody is the set of speech properties that sit above individual sounds: pitch movement, the duration of syllables and pauses, loudness, and voice quality. It signals whether an utterance is a question or a statement, which word is being contrasted, and where one phrase ends and the next begins. A synthesiser can pronounce every phoneme correctly and still sound wrong if its prosody does not match the intended meaning.
Why does neural text to speech often sound flat?
Text underdetermines prosody, because the same sentence can be said in many valid ways. A model trained to minimise average error across all of those readings tends to produce something near the middle of the distribution, which is smooth and safe but rarely emphatic. Flatness is therefore often a property of the training objective rather than a limitation of the audio quality.
Can you control emphasis in a text to speech system?
Yes, to varying degrees. Common approaches include markup such as SSML, which lets an author mark emphasis or insert breaks, reference encoders that copy the style of an example recording, and models that expose explicit pitch and duration predictions for editing. Each buys control at the cost of extra authoring effort or extra data.
How is prosody evaluated?
Prosody is usually assessed with listening tests, either absolute ratings such as mean opinion score or paired preference tests between two systems. Objective measures such as pitch contour correlation and duration error are useful for tracking regressions but correlate imperfectly with listener judgements. Word error rate from a recogniser says almost nothing about prosody, since a flat reading is still perfectly transcribable.