Photo: Stas Knop
In my previous post I described how I cloned my voice and generated an audiobook for my grandma using XTTS v2.
It worked.
Until I switched to a different book and publisher.
A biography of Wislawa Szymborska quickly put the first version of my script to the test. Different EPUB structure, lots of quotes, long sentences, footnotes, decorative drop caps.
The model "worked," but the result was far from natural. On top of that, listening through successive chapters, I compiled a substantial list of "fixes."
The list of problems was concrete:
- reading started from chapter 6, because earlier ones lacked proper markup,
- "Chapter 3" sounded like "Chapter three," not "third,"
- the model read editorial notes, ISBN, and publisher info,
- drop cap initials were split ("W islawa"),
- long quotes blended into the narration,
- there were no pauses between chapters,
- 3000-character chunks caused artifacts and token errors,
- footnote markers like [1], [2] were read aloud.
The conclusion was simple: the TTS model is the smallest part of the problem. The real work begins before and after audio generation.
Preprocessing — text preparation
Chunking: 3000 characters is too many
The first version split text mechanically every 3000 characters. XTTS has a limit of roughly 400 tokens per call — for Polish text, this meant regular errors and truncations.
I started by coding possible edge cases. But it quickly turned out that rules based solely on regexps weren't enough. Code doesn't understand context. I tried using a local language model to segment text and return ordered fragments in JSON format. In theory — intelligent sentence splitting. In practice:
- the quantized model was unstable — it generated invalid JSON, lost structure, appended comments,
- larger models worked better but were too computationally heavy to process an entire book locally.
The solution turned out to be the pySBD library — rule-based sentence segmentation. No GPU or ML model required. It handles ellipses and most abbreviations. For the Polish language, I had to add a custom abbreviation list (prof., dr, nr, im., ul., tzn., tzw.) to avoid false splits.
Final settings:
- default chunk: 300 characters,
- limit of 150 words,
- splitting only at sentence boundaries.
The model receives fragments that are naturally divided — the way a human narrator would split them, not a character counter.
Numbers and inflection
XTTS reads digits literally. "Chapter 3" sounds technically correct as "Chapter three," but in speech we use the ordinal form "Chapter third" (in Polish, inflection changes the word form entirely).
I added number-to-ordinal conversion with gender inflection: chapter third, page fifth, part second (each with proper Polish grammatical endings).
The conversion handles numbers up to 999, correctly building compound forms (e.g., "twenty-first").
EPUB cleanup
Polish EPUBs have a specific structure. Besides content, they contain: editorial notes, publishing info, footnotes (<sup>, [1]), decorative drop cap initials in separate HTML tags, images, and captions.
I expanded the filter to handle:
- removing footnotes and subscript markers,
- removing images and captions,
- merging drop cap initials with the rest of the word (e.g.,
<span class="dropcap">W</span>islawa), - reading chapters according to EPUB spine order,
- preserving forewords and prefaces as part of the book.
Every publisher structures their EPUB slightly differently. The parser must account for that.
As the cherry on top, I added a strongly accented title and author at the beginning of the book (the filters were stripping them out).
Quotes and pauses
Long quotes caused disorientation. The listener couldn't tell where narration ended.
For quotes longer than 50 characters, I add:
- "Quote:" at the beginning,
- "End of quote." at the end,
- audio pauses around them.
After headings, I inject special markers into the text (e.g., [PAUSE_CHAPTER], [PAUSE_SECTION]). During audio assembly, these are replaced with silence:
- 2-2.5 seconds after a chapter title — a clear entry into a new section,
- 2 seconds after a subheading — structural separation of content,
- 1-1.5 seconds around longer quotes — a subtle change in rhythm.
In version 1.0 the text was correct but "flat." After introducing controlled pauses, the narration started to breathe.
Retry mechanism
Even with short chunks, XTTS sometimes throws a token limit error.
I added a recursive retry mechanism:
- if a fragment is too long, it gets split in half,
- each half is processed separately,
- the process repeats until it succeeds.
Each generated fragment goes through micro-postprocessing:
- 3 ms fade-in (eliminates the "pop" at the start),
- trimming excess silence,
- 50 ms safe tail for crossfade.
Postprocessing — FFmpeg
Raw audio from XTTS has characteristic artifacts: vocoder noise, clicks, and uneven loudness.
The vocoder is the TTS model component that transforms the acoustic representation (e.g., mel-spectrogram) into the final audio signal. It "assembles" the sound. Depending on model quality, it can introduce subtle background noise, metallic timbre, or micro-distortions.
The FFmpeg postprocessing pipeline includes:
- Denoising (afftdn) — noise reduction without aggressive speech smoothing,
- Bandpass filter (40 Hz - 15 kHz) — removing extreme frequencies and DC offset,
- Declick (adeclick) — automatic removal of impulses from chunk joins,
- EBU R128 normalization to -16 LUFS — the standard loudness for audiobooks.
This isn't about "beautifying" the sound. It's about removing signals that betray its synthetic origin.
Photo: cottonbro studio
TTS parameter optimization
During one of the "code review" sessions, Claude suggested varying the reading style depending on text type. Uniform parameters meant that dialogue, description, and question all sounded identical.
This led to the TTSOptimizer module, which analyzes each fragment and dynamically adjusts generation parameters:
- dialogues — higher temperature and top_p (more expressiveness),
- questions — slightly slower pace,
- lists and enumerations — calmer pace and lower temperature.
Parameters are thus matched to the type of content. The result is a more natural reading experience.
Additionally, the --pause-stretch parameter extends natural pauses detected in the generated audio (silence below -35 dBFS, minimum 150 ms).
This parameter was created specifically with Grandma in mind. At a factor of 1.5, pauses between phrases are 50% longer. The tempo stays the same, but the narration becomes calmer and easier to follow.
TL;DR — if you want to make your own TTS audiobook
If you're considering a similar solution, keep in mind:
- The model isn't everything. The model reads, but you have to prepare the text. Most of the work is in preprocessing and postprocessing.
- Chunking must be semantic, not character-based. Splitting every 3000 characters is asking for artifacts and unnatural sound.
- Language inflection matters. Numbers, abbreviations, declension — these details build naturalness (especially critical for highly inflected languages like Polish).
- EPUB is not plain text. Spine, footnotes, drop caps, metadata — everything needs to be consciously filtered before reaching the model.
- Pauses build narration. Without them, an audiobook sounds like a continuous stream of text. Speed, intonation, and rhythm also matter — dialogue, quote, enumeration, and description shouldn't be read the same way. Long quotes should be clearly marked and separated.
- Retry and fallback are essential. The model will eventually throw an error — the pipeline must be ready for it.
- Postprocessing makes a huge difference. Denoising, declick, bandpass filtering, and normalization are a standard, not a luxury.
- Vary your TTS parameters. A dialogue shouldn't sound like a description, and a question shouldn't sound like a footnote.
- Test on a real listener. Especially if you're making it for a specific person — practical reception verifies technical assumptions.
Technically, it's a TTS project. In practice — it's the engineering of reading.
Audio sample
Below you can listen to a sample from the book: "Pamiatkowe rupiecie, biografia Wislaway Szymborskiej" (Memorable Trinkets, a biography of Wislawa Szymborska), by Anna Bikont and Joanna Szczesna.
What it all comes down to
A TTS model doesn't "read a book." It processes a string of characters.
It's preprocessing that determines whether the text becomes narration or a jumble of artifacts.
It's postprocessing that determines whether the sound resembles a voice or synthesis.
Every book is different. Every EPUB has its own pitfalls. But each one was an opportunity for me to learn something new (or revisit material from university and the "Digital Signal Processing" course).
Grandma listens and says it sounds almost like me.
Almost. And that "almost" is the most demanding part of the entire project.
The source code for the entire pipeline is available on GitHub — if you want to try the solution or adapt it to your own needs.
Stay tuned, I have a feeling this story isn't over yet...
