Popiai-skill/audio-creation/audiobook/phases/04-generation.md

Phase 4: Chapter Audio Generation

Script Setup (Run Once Per Session)

Read skillScriptsPath from the state file — written by Phase 0 with the skill's real install path:

export AUDIOBOOK_SCRIPTS=$(python3 -c "
import json, os
from pathlib import Path

audiobook_root = Path(os.getcwd()) / '.audiobook'
hits = list(audiobook_root.glob('*/.audiobook-state.json')) if audiobook_root.exists() else []
if not hits:
    print('ERROR: state file not found — run Phase 0 first'); exit(1)
data = json.loads(hits[0].read_text())
val = data.get('skillScriptsPath', '').strip()
if not val:
    print('ERROR: skillScriptsPath missing from state — re-run Phase 0'); exit(1)
if not Path(val).is_dir():
    print(f'ERROR: skillScriptsPath does not exist: {val}'); exit(1)
print(val)
")
echo "AUDIOBOOK_SCRIPTS=$AUDIOBOOK_SCRIPTS"

A clean path (no ERROR) means scripts are ready.

If you see skillScriptsPath missing from state: Do NOT re-run Phase 0 just for this. Instead, derive the path directly from the available_skills list visible in your system context — find the <location> entry for the audiobook skill (e.g. file:///Users/alice/.hub/skills/audiobook/SKILL.md), strip file:// and /SKILL.md, append /scripts. Then patch the state file:

import json
from pathlib import Path
audiobook_root = Path(os.getcwd()) / ".audiobook"
state_path = next(audiobook_root.glob("*/.audiobook-state.json"))
state = json.loads(state_path.read_text())
state["skillScriptsPath"] = "/derived/path/to/audiobook/scripts"  # replace with actual
state_path.write_text(json.dumps(state, ensure_ascii=False, indent=2))

Then re-run the Script Setup block above.

---

4.1 Core Principle: TTS Built-in Silence

Never add silence or crossfade via ffmpeg in post-processing. All silence must come from <#X#> pause markers in the TTS text. TTS-generated silence shares the same noise floor and audio characteristics as the speech, preventing waveform discontinuities at splice points.

4.2 TTS Text Silence Padding

Each segment's TTS text needs head/tail silence padding. This is applied automatically by the Step 1 script — do NOT manually edit segment text.

The chapter script from Phase 2 already contains inline <#X#> pause markers. Step 1 preserves ALL existing markers and adds head/tail padding on top.

Padding rules (applied automatically in Step 1):

Position	Head Padding	Tail Padding
First segment of book	<#2#>	<#0.2#>
Last segment of book	<#0.2#>	<#3#>
Chapter ending (not last)	<#0.2#>	<#2#>
Normal segment	<#0.2#>	<#0.2#>

4.3 TTS Generation (Parallel with Pre-Indexed Plan)

The ONLY rule that matters: 1 text per call, max 3 calls per wave. Run every script below — do NOT skip any.

Generation uses parallel audios_generation calls with a pre-indexed generation plan to guarantee correct playback order regardless of completion timing.

Each segment from the chapter script becomes one entry in the generation plan — no merging or grouping.

Step 1: Parse Chapter Script & Build Generation Plan (MANDATORY script)

This script does everything:

1. Reads the chapter script (preserving ALL <#X#> markers from Phase 2)
2. Parses segment delimiters and headers
3. Maps voice labels to voice_ids via voice_settings.json
4. Applies 4.2 head/tail padding
5. Outputs generation_plan.json
6. Prints every wave's calls — use these directly for Step 2

python3 $AUDIOBOOK_SCRIPTS/build_generation_plan.py <chapter_number>

The script auto-detects chapter title from chapters.md and first/last chapter from .audiobook-state.json. It also validates voice assignment and warns if dialogue segments fall back to narrator voice.

This script is MANDATORY — run it first, then use its output for Step 2.

Step 2: Fire TTS calls wave by wave

Use the exact calls printed by Step 1. Each call has exactly 1 text and 1 filename.

Rules:

• Fire up to 3 calls per wave (copy the calls from Step 1 output) — fire all 3 simultaneously (in parallel), NOT one by one
• Wait for ALL calls in a wave to complete before starting the next wave
• No sleep needed between successful waves
• If any call fails → sleep 30 seconds, then continue to the next wave. Do NOT retry the failed call inline. All failures are handled after Step 3.5 by the Failure & Retry script, which reads parameters from generation_plan.json (guaranteed correct). Inline retry risks using wrong parameters from LLM context.

Example (from Step 1 output):

# Wave 1 (fire these 3 calls in parallel):
audios_generation(texts=["<#0.2#>She opened the door.<#0.5#>"], voice_id="narrator", speed=0.9, filenames=["ch1_idx_0"])
audios_generation(texts=["Hello!"], voice_id="alice", speed=1.0, filenames=["ch1_idx_1"])
audios_generation(texts=["<#0.2#>He waved back.<#0.2#>"], voice_id="narrator", speed=0.9, filenames=["ch1_idx_2"])

# Wait for wave 1 to complete. Then wave 2:
audios_generation(texts=["Good morning."], voice_id="bob", speed=1.0, filenames=["ch1_idx_3"])
...

DO NOT:

• Put multiple texts in one call: texts=["a", "b", "c"] is WRONG
• Invent your own filenames — use the ch{N}_idx_{M} from Step 1
• Skip Step 1 and construct calls yourself

Step 3: Match blobs to plan (MANDATORY script)

After all TTS calls complete, run this to map generated audio files back to plan entries:

python3 $AUDIOBOOK_SCRIPTS/match_blobs.py <chapter_number>

The script reads generation_plan.json + assets.json, matches by filename (last-write-wins for retries), and updates generation_plan.json with blob_path for each entry.

Step 3.5: Verify blob content (MANDATORY script)

Verify every matched blob's metadata matches the plan (catches wrong voice_id or text from misfired calls):

python3 $AUDIOBOOK_SCRIPTS/verify_blobs.py <chapter_number>

The script compares voice_id and cleaned text from assets.json metadata against generation_plan.json. Mismatched entries have their blob_path cleared so the Retry script picks them up.

How clean_text() works (important for debugging false positives):

The script normalises both sides before comparing:

1. html.unescape() — the chapter script (Phase 2 output) may store pause markers as <#0.6#> (HTML-escaped) rather than <#0.6#>. The plan inherits this encoding. The TTS metadata always stores raw markers. Unescaping before stripping ensures both sides match.
2. Smart-quote → ASCII normalisation — the plan may contain Unicode curly quotes (" / " / ' / '). The TTS engine normalises these to straight ASCII quotes (" / ') in stored metadata. Both sides are normalised before comparison.

If you see unexpected "text mismatch" failures on Step 3.5, check generation_plan.json for < / > or \u201c / \u201d — these indicate the source script was HTML-escaped or contained smart quotes. The script handles both automatically; no manual fix needed.

Step 3.6: Organize segment files (MANDATORY script)

Move segment audio files from the project root into audio/chapter_{N}/segments/ and update assets.json paths:

python3 $AUDIOBOOK_SCRIPTS/organize_segment_files.py <chapter_number>

The script reads generation_plan.json for filenames, moves .mp3 and _subtitle.json files, and updates assets.json path fields. Idempotent — safe to re-run after retries.

Step 4: Write manifest (MANDATORY script)

python3 $AUDIOBOOK_SCRIPTS/write_manifest.py <chapter_number>

The script reads generation_plan.json, writes manifest.txt in playback order, and verifies all blob files exist on disk.

Failure & Retry (MANDATORY script)

If Step 3 reports unmatched segments, run this to get retry calls:

python3 $AUDIOBOOK_SCRIPTS/print_retry_calls.py <chapter_number>

The script cleans old workspace .mp3 files (prevents gateway dedup suffix), then prints exact retry calls grouped by wave (max 3 per wave).

Retry procedure:

1. Run Step 3 (blob matching) + Step 3.5 (verification) to identify all failures
2. Run the Failure & Retry script above — it reads generation_plan.json and prints exact calls
3. Sleep 20 seconds (let system recover from transient issues)
4. Fire the printed calls (max 3 per wave) — do NOT modify the parameters
5. Re-run Step 3 + Step 3.5 → re-run this script → repeat until all matched

CRITICAL: Never construct retry calls yourself. Always use this script's output. The script reads parameters from generation_plan.json, ensuring text, voice_id, speed, and filename are all correct. Manual retry from memory is the #1 cause of wrong audio content.

Never skip a failed segment. The manifest requires every segment to have a blob_path.

TTS text limits

• Each text should not exceed 2500 characters
• If a single segment is too long, split by sentence into sub-segments connected with <#0.2#>
• Sub-segments share the same voice_id and get consecutive indices in the plan

4.4 Concatenation & Export

Step 1: Verify manifest

python3 $AUDIOBOOK_SCRIPTS/write_manifest.py <chapter_number>

This is the same script as Step 4 — it writes manifest.txt and verifies every file exists. Run it again here to confirm nothing changed between generation and concatenation.

Step 2: Delegate to editing sub-agent

Pass the exact ordered file list from manifest.txt to the editing sub-agent. Do NOT re-sort or re-order — the manifest is the single source of truth.

请帮我拼接以下音频片段成一个完整的章节文件。
不需要 crossfade、fade 或 DC removal — 直接拼接即可。

片段列表（按顺序排列，不要重新排序）：
1. /abs/path/to/.hilo/.blobs/{uuid1}.mp3
2. /abs/path/to/.hilo/.blobs/{uuid2}.mp3
3. /abs/path/to/.hilo/.blobs/{uuid3}.mp3
...

格式要求：MP3, 44100Hz, 256kbps
输出文件名：chapter_{N}.mp3
完成后请返回输出文件路径和时长。

Important:

• Input file list MUST come from manifest.txt
• Number each file to make order unambiguous
• Do NOT specify an output path — let the editing agent decide
• The editing sub-agent automatically handles canvas registration (writes to both project root and .hilo/.blobs/, registers in assets.json)

Filename rules

• Individual segments: ch{N}_idx_{M} (generated by Step 1 script, do not change)
• Final chapter audio: chapter_{N}.mp3
• Never use Chinese characters in any audio filename — canvas registration silently fails

4.5 Audio Quality Check

After each chapter's final audio is exported, run the quality checker:

python3 $AUDIOBOOK_SCRIPTS/audio_check.py /path/to/exported_chapter.mp3

ffmpeg dependency: audio_check.py and generate_assembly_report.py both shell out to ffmpeg / ffprobe. If ffmpeg is not on $PATH, prepend the binary location:

PATH="/opt/homebrew/Cellar/ffmpeg/8.1/bin:$PATH" python3 $AUDIOBOOK_SCRIPTS/audio_check.py ...

Find the binary with: find /usr /opt /Users -name "ffmpeg" -type f 2>/dev/null | head -3

If issues are found, delegate fixes to the editing sub-agent:

• Pops/clicks: Apply adeclick/adeclip filter
• DC offset > 1%: Apply dcshift correction
• Clipping: Reduce volume and regenerate the affected segments

Energy spikes are normal: The quality checker reports "energy spikes" at segment boundaries (silence → speech onset from hard-cut concatenation). These are expected and require no action — they are not audio artifacts.

4.6 Assembly Report

After the quality check passes, generate a markdown assembly report. This is the single source of truth for users to inspect, locate, and request re-edits of any segment.

What it contains

A markdown table with one row per segment:

Column	Content
#	Segment index (primary identifier for re-edit requests)
Start / End	Precise timestamps hh:mm:ss.ss via ffprobe
Voice ID	The actual voice_id used
Local File	Root-directory filename (matches canvas display)
Text	Clean spoken text with <#X#> markers stripped

Generation script

python3 $AUDIOBOOK_SCRIPTS/generate_assembly_report.py <chapter_number>

ffmpeg dependency: This script uses ffprobe to compute timestamps. If ffprobe is not on $PATH, prepend the binary location (same as 4.5 above).

The script reads generation_plan.json, uses ffprobe to compute timestamps, maps blob refs to local filenames via assets.json, and writes {book_name}_chapter_{N}_assembly.md to the project root. Auto-detects language for tips text.

Canvas note: The canvas does not render markdown tables — only header lines are previewed. Users can open the assembly report in any markdown viewer for the full table.

Re-edit workflow

When the user requests changes (e.g. "segment #14 sounds too fast"):

1. Look up index 14 in generation_plan.json for voice_id, speed, text, filename
2. Regenerate: audios_generation(texts=[new_text], voice_id=..., speed=..., filenames=[filename])
3. Re-run Step 3 blob matching to pick up the new blob (latest entry with same filename wins)
4. Rewrite manifest.txt from updated plan
5. Re-concatenate via editing sub-agent
6. Re-run quality check and regenerate assembly report

4.7 Final User Message (REQUIRED)

After the assembly report is generated, send a completion message. Match the user's language.

Include:

• Final audio filename (UUID on canvas)
• Duration
• Cast list (narrator + characters)
• Assembly report filename
• How to request fixes (mention segment # number)
• Options: generate next chapter / re-edit a segment

4.8 Multi-Chapter Generation

Note: True background/parallel chapter generation is not currently supported. Chapters are processed one at a time (sequentially) to ensure quality and allow user review between chapters.

After the first chapter is confirmed by the user:

1. Tell the user how many chapters remain and ask if they want to continue generating them
2. For each remaining chapter, follow the full 4.2–4.6 workflow in order:
   • Build generation plan → Fire TTS waves → Match blobs → Verify → Organize → Write manifest → Concatenate → Quality check → Assembly report
3. After each chapter completes, update chaptersCompleted in .audiobook-state.json
4. Notify the user that chapter N is done, show duration and assembly report filename, then ask whether to proceed to the next chapter or stop

Error Handling

• TTS generation failure: Sleep 20s then retry. Never skip — keep retrying until success.
• File parsing failure: Prompt the user to check file format
• Audio concatenation failure: Check file paths and formats