Flatten skill category directory structure
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
# 歌词拆解
|
||||
|
||||
## 目标
|
||||
|
||||
将歌曲音频转为带时间戳的歌词段落,标注纯音乐段(前奏/间奏/尾奏)。
|
||||
|
||||
## 输入
|
||||
|
||||
| 输入 | 必需 | 说明 |
|
||||
|------|------|------|
|
||||
| 音频文件 | 是 | 歌曲音频文件路径 |
|
||||
| LRC 文件 | 否 | 如提供则直接解析,跳过 ASR |
|
||||
|
||||
## 处理流程
|
||||
|
||||
如果用户未提供音频文件,直接在对话中向用户提问,要求提供音频文件路径。音频文件是整个 MV 创作流程的前提,必须先获取。
|
||||
|
||||
音频文件确认后,扫描 `.mv/*/lyric.md` 中的 `音频:`/`Audio:` 字段,检查是否已有该音频路径对应的歌词拆解:
|
||||
|
||||
- **已存在** → 读取已有的 `lyric.md` 内容,展示给用户,通过 `AskUserQuestion` 询问用户:
|
||||
- 直接使用已有歌词拆解,进入下一阶段
|
||||
- 重新拆解(覆盖已有文件)
|
||||
- **不存在** → 继续正常拆解流程
|
||||
|
||||
正常拆解流程:如果用户未提供 LRC 文件,通过 `AskUserQuestion` 询问用户是否有 LRC 文件可以提供(选项:有/没有)。
|
||||
- 用户提供了 → 走路线 A
|
||||
- 用户没有 → 走路线 B(ASR 兜底)
|
||||
|
||||
### 路线 A:用户提供 LRC 文件
|
||||
|
||||
直接解析 LRC 文件,提取时间戳和歌词文本。
|
||||
|
||||
### 路线 B:ASR + LLM(兜底方案)
|
||||
|
||||
1. **获取音频时长**:通过 ffprobe 获取音频总时长
|
||||
```bash
|
||||
ffprobe -v error -show_entries format=duration -of csv=p=0 <audio_path>
|
||||
```
|
||||
2. **音频预处理**:检查文件大小,如果超过 5MB 则先压缩再调用 ASR(`hilo_tools_audio_transcribe_lyrics` 限制 10MB)
|
||||
```bash
|
||||
# 检查文件大小(字节)
|
||||
stat -f%z <audio_path> # macOS
|
||||
stat -c%s <audio_path> # Linux
|
||||
wc -c < <audio_path> # 跨平台
|
||||
(Get-Item <audio_path>).Length # Windows PowerShell
|
||||
# 如果 > 5242880(5MB),压缩为 64kbps 单声道
|
||||
ffmpeg -y -i <audio_path> -b:a 64k -ar 16000 -ac 1 <compressed_path>
|
||||
```
|
||||
压缩后使用 `<compressed_path>` 调用 ASR,原始音频保留用于后续阶段。
|
||||
3. **ASR 识别**:调用 MCP 工具 `hilo_tools_audio_transcribe_lyrics`,获取词级时间戳
|
||||
- 中文自动使用 Tencent Cloud ASR,其他语言使用 Whisper ASR
|
||||
- 返回的 JSON 包含 segments(时间戳 + 歌词 + 词级信息)
|
||||
4. **纯音乐段检测**:工具自动识别并标注无歌词段落
|
||||
- 开头无歌词 > 0.5s → `[prelude]`
|
||||
- 歌词间间隙 > 5s → `[interlude]`
|
||||
- 结尾无歌词 > 0.5s → `[outro]`
|
||||
5. **歌词矫正**:调用 MCP 工具 `hilo_tools_read_media` 传入音频文件,获取对音频内容的听写结果,然后 LLM 结合 ASR 结果和听写结果进行歌词矫正(纠正错别字、断句、标点等)
|
||||
|
||||
## 输出格式
|
||||
|
||||
每个段落包含时间码和歌词文本,纯音乐段用方括号标记。**文件头部必须包含音频文件路径**(相对项目根目录),用于后续阶段引用和歌词拆解去重判断。
|
||||
|
||||
完整示例见 `references/lyric-example.md`(非中文用户参考 `references/lyric-example-en.md`)。
|
||||
|
||||
> 注:每个时间段的歌词按实际演唱的句子逐行拆分,一句一行。
|
||||
|
||||
## 格式校验
|
||||
|
||||
歌词写入 `lyric.md` 后,**必须**运行校验脚本:
|
||||
|
||||
```bash
|
||||
python3 .opencode/skills/mv-creator/scripts/validate_lyric.py .mv/{song_name}/lyric.md
|
||||
```
|
||||
|
||||
校验规则:时间码格式、时间段连续性、单段建议不超过 20 秒(WARN)、纯音乐段无歌词、歌词段非空。
|
||||
|
||||
- **通过** → 调用 MCP 工具 `preview_and_collect_feedback` 启动预览:
|
||||
- **script**: `.opencode/skills/mv-creator/scripts/render_preview.py`
|
||||
- **args**: `[".mv/{song_name}/lyric.md", "--type", "lyric", "--lang", "{lang}"]`
|
||||
- **feedback_path**: `.mv/{song_name}/feedback.md`
|
||||
|
||||
工具会启动预览服务器、打开浏览器,等待用户提交反馈后直接返回反馈内容:
|
||||
- 内容为 `LGTM` → 用户确认无修改,进入下一阶段
|
||||
- 其他内容 → 用户的编辑建议,按建议修改后重新校验和预览
|
||||
- **未通过** → 通过 `AskUserQuestion` 询问用户处理方式:
|
||||
- **用户手动修改**:暂停流程,等待用户修改 `lyric.md` 后重新校验
|
||||
- **重新 ASR**:重新执行 ASR 识别和 LLM 矫正,覆盖当前文件
|
||||
- **模型自动修复**:由 LLM 根据校验错误信息自动修正 `lyric.md`,修复后重新校验
|
||||
- **中止流程**:终止 MV 创作流程
|
||||
|
||||
## 文件存储
|
||||
|
||||
```
|
||||
./.mv/{song_name}/lyric.md
|
||||
```
|
||||
@@ -0,0 +1,35 @@
|
||||
# 故事概念
|
||||
|
||||
## 目标
|
||||
|
||||
基于歌词内容生成 MV 的故事主题、故事简要和核心人物。
|
||||
|
||||
## 流程
|
||||
|
||||
### 1. 预生成
|
||||
|
||||
LLM 结合歌词和用户输入,自动生成初版故事概念。
|
||||
|
||||
通过 `AskUserQuestion` 询问用户的额外要求(风格偏好、特定意象等)。
|
||||
|
||||
### 2. 用户修正循环
|
||||
|
||||
依次通过 `AskUserQuestion` 与用户确认以下内容,每项支持反复修改直到用户满意:
|
||||
|
||||
1. **故事主题** — 用户指定或从建议中选择
|
||||
2. **故事简要** — 用户提出修改意见,迭代至满意
|
||||
3. **核心人物** — 用户提出修改意见,迭代至满意
|
||||
|
||||
## 输出格式
|
||||
|
||||
完整格式示例见 `references/story-concept-example.md`(非中文用户参考 `references/story-concept-example-en.md`)。关键要点:
|
||||
|
||||
- 包含三个字段:`故事主题:`、`故事简要:`、`核心人物:`
|
||||
- 故事主题一句简短的话(20~50字)
|
||||
- 核心人物用【角色名】开头,多个角色各占一行
|
||||
|
||||
## 文件存储
|
||||
|
||||
```
|
||||
./.mv/{song_name}/{theme}/story_concept.md
|
||||
```
|
||||
@@ -0,0 +1,84 @@
|
||||
# 美术风格
|
||||
|
||||
## 目标
|
||||
|
||||
在人设图和分镜图生成之前,确定 MV 的整体美术风格,为后续视觉生成提供统一的风格基准。
|
||||
|
||||
## 内置风格预设
|
||||
|
||||
内置风格的预览图首次预览时由 `scripts/download_style_presets.py` 自动下载到 `.mv/{song_name}/{theme}/style_cache/{style}/` 本地目录,后续直接复用本地缓存。下游 `art_style.md` 的 `参考图:` 字段统一记录本地路径,确保生图工具的 `image_paths` 可直接读取。
|
||||
|
||||
| 目录 | 风格 | 说明 |
|
||||
|------|------|------|
|
||||
| `cinematic/` | 电影质感写实 | 35mm胶片颗粒、暖琥珀色调、自然光、宽银幕 |
|
||||
| `oil-painting/` | 油画风格 | 厚重笔触、印象派色彩、painterly 质感 |
|
||||
| `noir-illustration/` | 黑色插画 | 高对比剪影、图像小说风、琥珀色点缀 |
|
||||
|
||||
## 流程
|
||||
|
||||
调用 MCP 工具 `preview_and_collect_feedback` 启动美术风格选择页面:
|
||||
- **script**: `.opencode/skills/mv-creator/scripts/render_preview.py`
|
||||
- **args**: `[".mv/{song_name}/{theme}", "--type", "art_style", "--lang", "{lang}"]`
|
||||
- **feedback_path**: `.mv/{song_name}/{theme}/feedback.md`
|
||||
|
||||
页面是一个交互式选择界面,包含:
|
||||
|
||||
- **内置风格展示区**:脚本首次启动时通过 `download_style_presets.py` 把内置风格预设图缓存到 `.mv/{song_name}/{theme}/style_cache/{style}/`,按分类展示,附风格关键词标签
|
||||
- **自定义风格区**:允许输入 **风格描述文字**、上传 **1 张** 自定义参考图,或两者兼用(上传后保存到项目 `style_references/` 目录)
|
||||
- **选择提交区**:用户选定风格方向后提交
|
||||
|
||||
工具返回反馈内容后:
|
||||
|
||||
- **选择内置预设** → 直接结合预设信息生成 `art_style.md`
|
||||
- **选择自定义风格** → 根据用户提供的内容生成 `art_style.md`:
|
||||
- 仅文字描述 → 基于描述提取风格关键词,生成 `art_style.md`
|
||||
- 仅参考图 → 用 `read_media` MCP 工具分析参考图的美术风格特征(线条、色彩、构图、质感等),再基于分析结果生成
|
||||
- 文字 + 参考图 → 结合文字描述和参考图分析结果,综合生成
|
||||
|
||||
**确保参考图可用**:`art_style.md` 的 `参考图:` 列表必须包含至少一张图片。
|
||||
如果用户选择的是纯文本自定义风格(无内置预设图、无上传参考图),则在生成 `art_style.md` 之前,先用 Midjourney 根据风格关键词生成一张风格参考图,保存到 `style_references/` 目录,并写入 `参考图:` 列表。这确保后续人设图和分镜图生成阶段始终有可用的风格参考图。**此处 MJ 生成的四宫格不需要裁切**,仅需一张完整图作为风格参考即可。
|
||||
|
||||
生成的 `art_style.md` 记录:
|
||||
|
||||
- 风格名称和描述
|
||||
- 选定的参考图路径(**至少一张**)
|
||||
- 色调、光影、质感等关键词
|
||||
- 适用的生图模型推荐
|
||||
|
||||
**风格 JSON 提取**:`art_style.md` 生成后,**必须**从参考图中提取结构化的风格 JSON。调用 `read_media` MCP 工具,传入参考图路径和以下提示词:
|
||||
|
||||
```
|
||||
请以专业的艺术品鉴观角,深度分析这张图片的视觉风格。请忽略画面具体内容(比如人物、剧情),色彩风格、色彩调色板(hex代码)、构图风格、纹理质感、材质效果、线条风格、艺术风格关键词。请将分析结果输出为一段精简的 JSON 格式或描述,不要任何解读,就只要 JSON 结构。
|
||||
```
|
||||
|
||||
将返回的 JSON 结果写入 `art_style.md` 的 `风格JSON:` 字段中。此 JSON 将在后续人设图、背景图和关键物件图生成时作为风格描述注入提示词,替代直接传入参考图。
|
||||
|
||||
## 输出格式
|
||||
|
||||
完整格式示例见 `references/art-style-example.md`(非中文用户参考 `references/art-style-example-en.md`)。关键要点:
|
||||
|
||||
- 首行 `美术风格:` + 风格名称
|
||||
- 包含字段:`风格描述:`、`关键词:`、`色调:`、`光影:`、`推荐生图模型:`、`参考图:`、`风格JSON:`
|
||||
- 关键词使用英文,用于后续生图 prompt
|
||||
- 参考图路径相对项目目录,每行一个
|
||||
- 风格JSON 为 `read_media` 分析参考图返回的结构化风格数据
|
||||
|
||||
## 文件存储
|
||||
|
||||
```
|
||||
./.mv/{song_name}/{theme}/art_style.md # 美术风格说明
|
||||
./.mv/{song_name}/{theme}/style_cache/ # 内置风格预设 CDN 缓存(首次预览自动下载)
|
||||
./.mv/{song_name}/{theme}/style_references/ # 用户上传的参考图(如有)
|
||||
```
|
||||
|
||||
## 风格路由
|
||||
|
||||
美术风格确定后,判断风格是否属于**写实类**,决定后续工作流:
|
||||
|
||||
**写实类判定**:以下情况视为写实风格:
|
||||
- 用户选择了内置预设「电影质感写实(cinematic)」
|
||||
- 用户自定义风格的描述或关键词偏向写实(如包含 photorealistic / realistic / cinematic / 写实 / 真人 / 实拍 等特征)
|
||||
|
||||
**路由规则**:
|
||||
- **写实风格** → 停止 mv-creator 主流程,告知用户写实风格将使用写实分支流程(支持口型同步等写实 MV 能力)。读取 `phases/realistic-pipeline.md`,从 STEP 0 开始执行,将已有的音频文件路径和歌词数据(`.mv/{song_name}/lyric.md`)传递过去,跳过已完成的音频获取和歌词提取步骤
|
||||
- **非写实风格(油画、插画、动画、自定义非写实等)** → 继续 mv-creator 的下一阶段(脚本生成)
|
||||
@@ -0,0 +1,60 @@
|
||||
# 脚本生成
|
||||
|
||||
## 目标
|
||||
|
||||
根据歌词拆解的段落结构,为每个段落生成详细的分幕脚本。
|
||||
|
||||
## 重点
|
||||
|
||||
- 脚本的时间段可以对歌词段落进行拆分或合并,**不需要**与 lyric.md 一一对应
|
||||
- 各阶段产物相互独立:修改脚本不需要回改歌词,修改分镜不需要回改脚本,以此类推
|
||||
- **场景拆分后必须重新编号**:若某幕时长超过 15s 需要拆分成多幕,拆分后**必须**将所有场景从第一幕开始重新连续编号(第一幕、第二幕、……第 N 幕),**禁止**使用 `a/b` 后缀(如"第九幕b")。连续编号确保下游分镜图文件命名(scene_01.jpg、scene_02.jpg...)与数组索引严格一致
|
||||
- 脚本格式**必须**严格遵循模板 `references/script-example.md`(非中文用户参考 `references/script-example-en.md`),否则 HTML 预览解析器无法正确渲染
|
||||
- **场景描述必须写具体动态**:脚本的场景描述是后续视频提示词的核心输入,描述越具体、动态越丰富,生成的视频动作越自然。**禁止笼统静态描述**(如"旅人站在路边"),必须写清:
|
||||
- **角色动作**:具体的肢体动作和运动轨迹(走、跑、转身、抬手、低头等)
|
||||
- **表情/情绪**:面部表情变化(微笑渐收、皱眉凝视等)
|
||||
- **物件交互**:角色与道具/环境的互动(抚摸琴弦、推开车门、拾起信物等)
|
||||
- **镜头运动**:明确的镜头语言(从脚部缓缓上摇、跟随角色横移、推近面部特写等)
|
||||
- **环境动态**:风吹发丝、水面倒影晃动、光影变化等非静态元素
|
||||
|
||||
## 格式模板
|
||||
|
||||
完整格式示例见 `references/script-example.md`(非中文用户参考 `references/script-example-en.md`)。关键要点:
|
||||
|
||||
- 二级标题 `##` 用于每幕场景标题(如 `## 第一幕:黎明启程`)
|
||||
- 时间码行格式:`* M:SS - M:SS (Xs)`
|
||||
- 不加粗、不标注段落类型(Verse/Chorus 等)
|
||||
- 括号内为时长(秒),必须为 **4~15 秒的整数**
|
||||
- 歌词引用以 `> ` 开头,纯音乐幕无歌词引用
|
||||
- 每幕必须包含三个字段:`**场景描述:**`、`**视觉风格:**`、`**转场:**`
|
||||
- 可选字段 `**关键物件:**`:列出该幕出现的重要非人物视觉元素,格式为物件名 + 简短外观描述(括号内),多个物件逗号分隔
|
||||
- 示例:`**关键物件:** 旧皮卡(深蓝色、锈迹斑驳、车顶行李架)、吉他(原木色民谣吉他、琴身有贴纸)`
|
||||
- 场景之间用 `---` 分隔
|
||||
|
||||
## 文件存储
|
||||
|
||||
```
|
||||
./.mv/{song_name}/{theme}/script.md # 脚本文件
|
||||
```
|
||||
|
||||
## 格式校验
|
||||
|
||||
脚本写入 `script.md` 后,**必须**运行校验脚本:
|
||||
|
||||
```bash
|
||||
python3 .opencode/skills/mv-creator/scripts/validate_script.py .mv/{song_name}/{theme}/script.md
|
||||
```
|
||||
|
||||
校验规则:场景标题格式、时间码格式(含时长标签)、时长范围(4~15s 整数)、时间段连续性、必填字段完整性、与 lyric.md 交叉校验。
|
||||
|
||||
- **通过** → 调用 MCP 工具 `preview_and_collect_feedback` 启动预览:
|
||||
- **script**: `.opencode/skills/mv-creator/scripts/render_preview.py`
|
||||
- **args**: `[".mv/{song_name}/{theme}/script.md", "--type", "script", "--lang", "{lang}"]`
|
||||
- **feedback_path**: `.mv/{song_name}/{theme}/feedback.md`
|
||||
|
||||
预览页面以时间线卡片形式展示各幕,用户可点击场景卡片的 `@` 按钮引用到反馈框中。
|
||||
工具返回反馈内容后:
|
||||
- `LGTM` → 进入下一阶段
|
||||
- 其他 → 按建议修改后重新校验和预览
|
||||
- **未通过** → 由 LLM 根据校验错误信息自动修正 `script.md`,修复后重新校验。
|
||||
若连续 3 次自动修复仍未通过,通过 `AskUserQuestion` 询问用户处理方式
|
||||
@@ -0,0 +1,72 @@
|
||||
# 人设图生成
|
||||
|
||||
## 目标
|
||||
|
||||
基于确定的美术风格,为核心人物生成角色设计图。
|
||||
|
||||
## 重点
|
||||
|
||||
- 每个角色独立一个目录和设计文件
|
||||
- **统一使用 `all_in_one_image_generation`(model_name: `all_in_one_pro`)/ `nano_banana_image_generation`(model_name: `nano_banana_2`)生成候选图**
|
||||
- 保留每张图和对应的用户指令(prompt / 描述)
|
||||
- 最终由用户在预览页面中挑选人设图
|
||||
- **宽高比统一使用 2:3 竖版**(`aspect_ratio: "2:3"`)
|
||||
- **禁止生成 N 宫图**(多视角拼合图):提示词中必须包含 `single character, single view, no split screen, no multiple panels, no collage, no reference sheet`
|
||||
|
||||
## 流程
|
||||
|
||||
1. 根据故事概念中的核心人物描述 + 美术风格,生成角色设计 prompt(提示词末尾追加反 N 宫图指令)
|
||||
2. **风格 JSON 注入**:从 `art_style.md` 的 `风格JSON:` 字段读取结构化风格数据,将 JSON 内容融入生图提示词中(如在提示词末尾附加 `Art style: {JSON 内容}`)
|
||||
3. 使用 `all_in_one_image_generation`(model_name: `all_in_one_pro`)或 `nano_banana_image_generation`(model_name: `nano_banana_2`)生成多版本候选图,**aspect_ratio 统一设为 2:3**
|
||||
4. 将角色信息和候选图写入各角色的 `design.md`
|
||||
|
||||
## 输出格式
|
||||
|
||||
每个角色独立一个目录和设计文件。**严格按用户语言选择对应的 example 和字段名**(参见 SKILL.md「输出控制」和「Markdown 字段名中英对照表」):
|
||||
- 中文用户:参考 `references/character-design-example.md`,字段用中文(`外貌特征`、`性格气质`、`标志物件`、`生图设定`、`候选图`、`选定图`),内容用中文
|
||||
- 非中文用户:参考 `references/character-design-example-en.md`,字段用英文(`Appearance`、`Personality`、`Signature Items`、`Image Settings`、`Candidates`、`Selected`),内容用对应语言
|
||||
|
||||
关键要点:
|
||||
- 外貌特征/Appearance 用列表逐项描述(年龄/性别、体型、发型、面部、服装、配饰)
|
||||
- 生图设定/Image Settings 包含三个子字段:`提示词/Prompt`(英文)、`模型/Model`、`参考图/Reference`
|
||||
- 候选图/Candidates 记录文件名、模型和构图说明
|
||||
- `选定图/Selected` 生成后留空,待用户在预览页面中挑选
|
||||
|
||||
## 格式校验
|
||||
|
||||
design.md 写入后,**必须**运行校验脚本:
|
||||
|
||||
```bash
|
||||
# 校验单个角色
|
||||
python3 .opencode/skills/mv-creator/scripts/validate_character_design.py .mv/{song_name}/{theme}/characters/{character_name}/design.md
|
||||
|
||||
# 校验 theme 下所有角色
|
||||
python3 .opencode/skills/mv-creator/scripts/validate_character_design.py .mv/{song_name}/{theme}/
|
||||
```
|
||||
|
||||
校验规则:必需分区完整性、外貌特征列表项、生图设定子字段、候选图非空且文件存在。
|
||||
|
||||
- **通过** → 调用 MCP 工具 `preview_and_collect_feedback` 启动预览:
|
||||
- **script**: `.opencode/skills/mv-creator/scripts/render_preview.py`
|
||||
- **args**: `[".mv/{song_name}/{theme}", "--type", "character_design", "--lang", "{lang}"]`
|
||||
- **feedback_path**: `.mv/{song_name}/{theme}/feedback.md`
|
||||
|
||||
预览页面展示各角色信息卡片和候选图网格。
|
||||
每个角色默认选中第一张候选图作为兜底,用户点击图片直接切换选定。
|
||||
角色卡片的每张候选图有 `@` 按钮,点击可将 `@角色目录/候选图文件名` 插入底部反馈框,用于引用特定候选图。
|
||||
工具返回反馈内容后:
|
||||
- 以 `LGTM` 开头 → 将各角色的选定候选图复制为 `selected.jpg`,更新 `design.md`,进入下一阶段
|
||||
- 包含 `修改建议:` → 先处理选图,再解析建议中的 `@` 引用(候选图 `@角色/候选图` 或上传参考图 `@上传路径`)和修改意见,调整 prompt 重新生成候选图。更新 `design.md`,重新校验和预览
|
||||
- **未通过** → 由 LLM 根据校验错误信息自动修正,修复后重新校验
|
||||
|
||||
## 文件存储
|
||||
|
||||
**目录名规则**:`{character_name}` 统一使用英文小写 + 下划线(如 `traveller`、`bar_owner`),不使用中文或显示名。该目录名是后续阶段(分镜、视频生成)引用角色的唯一标识。
|
||||
|
||||
```
|
||||
./.mv/{song_name}/{theme}/characters/{character_name}/
|
||||
├── design.md # 角色设计说明
|
||||
├── candidate_01.jpg # 候选图
|
||||
├── candidate_02.jpg
|
||||
└── selected.jpg # 最终选定图(用户挑选)
|
||||
```
|
||||
@@ -0,0 +1,82 @@
|
||||
# 背景图生成
|
||||
|
||||
## 目标
|
||||
|
||||
基于脚本中的场景描述,为每个独立场景环境生成统一背景参考图。每一幕都需要对应一个背景,相似场景可合并复用同一背景。
|
||||
|
||||
## 背景类别
|
||||
|
||||
- **室外场景**:公路、荒野、山谷、海滩、街道等开阔环境
|
||||
- **室内场景**:酒馆、卧室、教室、工厂等封闭空间
|
||||
- **特殊环境**:梦境、回忆、超现实场景等
|
||||
|
||||
## 流程
|
||||
|
||||
### 1. 自动提取与去重
|
||||
|
||||
读取 `script.md` 中所有场景描述,提取每幕的场景环境。将相似的场景环境归并为同一背景(如第一幕和第七幕都在"荒漠公路"则合并),列出去重后的背景列表,附带每个背景的出场幕列表和环境描述摘要。
|
||||
|
||||
**每一幕都必须被覆盖** — 确保所有幕的场景环境都被归类到某个背景中。
|
||||
|
||||
### 2. 生成参考图
|
||||
|
||||
为每个背景生成 2~3 张候选图。**宽高比统一使用 16:9 宽屏**(`aspect_ratio: "16:9"`),展示场景全貌。
|
||||
|
||||
使用 `art_style.md` 中 `推荐生图模型:` 指定的模型生成候选图。提示词融合:场景环境描述(从 script.md 汇总)+ 美术风格关键词。**风格统一通过注入 `art_style.md` 的 `风格JSON:` 字段实现**(在提示词末尾追加 `Art style: {JSON 内容}`),**不直接通过 `image_paths` 传参考图**——避免预设图为 CDN 路径时无法被生图工具读取。如确需传图作为风格参考,必须使用 `art_style.md` 中本地路径(如 `style_cache/{style}/01_highway.png` 或 `style_references/user_ref.png`),并在提示词中说明该图仅用于风格参考(如 `use the reference image for art style only`)。
|
||||
|
||||
**提示词必须包含 `no people`**,确保背景图不包含人物,以便后续视频生成阶段单独叠加角色参考。
|
||||
|
||||
**Midjourney 四宫格裁切**:当推荐模型为 Midjourney 时,`midjourney_image_generation` 输出为 2x2 四宫格拼图,生成后**必须**用 ffmpeg 裁切为 4 张独立图片(1 次 MJ 调用 = 4 张候选图产出)。裁切方法:先用 `ffprobe` 获取宽高,再用 `crop` 滤镜分别截取左上、右上、左下、右下四个象限。
|
||||
|
||||
### 3. 用户选图
|
||||
|
||||
启动预览,展示背景卡片和候选图网格,用户为每个背景选定最终参考图。调用 MCP 工具 `preview_and_collect_feedback`:
|
||||
- **script**: `.opencode/skills/mv-creator/scripts/render_preview.py`
|
||||
- **args**: `[".mv/{song_name}/{theme}", "--type", "backgrounds", "--lang", "{lang}"]`
|
||||
- **feedback_path**: `.mv/{song_name}/{theme}/feedback.md`
|
||||
|
||||
预览页面展示各背景信息卡片和候选图。
|
||||
反馈处理逻辑同人设图阶段:
|
||||
- 以 `LGTM` 开头 → 将各背景的选定候选图复制为 `selected.jpg`,更新 `design.md`,进入下一阶段
|
||||
- 包含 `修改建议:` → 解析建议,调整 prompt 重新生成候选图,更新后重新校验和预览
|
||||
|
||||
## 输出格式
|
||||
|
||||
每个背景独立一个目录和设计文件。**严格按用户语言选择对应的 example 和字段名**(参见 SKILL.md「输出控制」和「Markdown 字段名中英对照表」):
|
||||
- 中文用户:参考 `references/background-design-example.md`,字段用中文(`场景描述`、`出场幕列表`、`生图设定`、`候选图`、`选定图`),内容用中文
|
||||
- 非中文用户:参考 `references/background-design-example-en.md`,字段用英文(`Scene Description`、`Scene List`、`Image Settings`、`Candidates`、`Selected`),内容用对应语言
|
||||
|
||||
关键要点:
|
||||
- 一级标题 `#` 用于背景名
|
||||
- 场景描述/Scene Description 用列表逐项描述(环境类型、地形/空间、色调/氛围、关键元素、天气/时段)
|
||||
- 生图设定/Image Settings 包含三个子字段:`提示词/Prompt`(英文,必须包含 `no people`)、`模型/Model`、`参考图/Reference`
|
||||
|
||||
## 格式校验
|
||||
|
||||
design.md 写入后,**必须**运行校验脚本:
|
||||
|
||||
```bash
|
||||
# 校验单个背景
|
||||
python3 .opencode/skills/mv-creator/scripts/validate_background_design.py .mv/{song_name}/{theme}/backgrounds/{bg_name}/design.md
|
||||
|
||||
# 校验所有背景
|
||||
python3 .opencode/skills/mv-creator/scripts/validate_background_design.py .mv/{song_name}/{theme}/backgrounds/
|
||||
```
|
||||
|
||||
校验规则:必需分区完整性、场景描述列表项、生图设定子字段、候选图非空且文件存在。
|
||||
|
||||
- **通过** → 启动预览供用户选图
|
||||
- **未通过** → 由 LLM 根据校验错误信息自动修正,修复后重新校验
|
||||
|
||||
## 文件存储
|
||||
|
||||
**目录名规则**:`{bg_name}` 统一使用英文小写 + 下划线(如 `desert_highway`、`neon_bar`),不使用中文或显示名。该目录名是分镜阶段引用背景的唯一标识。
|
||||
|
||||
```
|
||||
./.mv/{song_name}/{theme}/backgrounds/
|
||||
├── {bg_name}/
|
||||
│ ├── design.md # 背景设计说明
|
||||
│ ├── candidate_01.jpg # 候选图
|
||||
│ ├── candidate_02.jpg
|
||||
│ └── selected.jpg # 最终选定图(用户挑选)
|
||||
```
|
||||
@@ -0,0 +1,90 @@
|
||||
# 关键物件图生成
|
||||
|
||||
## 目标
|
||||
|
||||
基于脚本中标注的关键物件,为需要跨镜头保持视觉一致的非人物元素生成统一参考图。
|
||||
|
||||
## 物件类别
|
||||
|
||||
- **道具**:武器、信物、乐器、书信等手持/随身物品
|
||||
- **载具**:马、车、船、摩托等交通工具
|
||||
- **场景地标**:反复出现的特定建筑、标志性场所、门牌等
|
||||
- **动物/宠物**:有辨识度的伴随动物
|
||||
|
||||
## 流程
|
||||
|
||||
### 1. 自动提取
|
||||
|
||||
读取 `script.md` 中所有 `**关键物件:**` 字段,汇总并去重。列出在 **2 幕及以上** 重复出现的物件,附带每个物件的出场幕列表和外观描述摘要。
|
||||
|
||||
### 2. 用户确认
|
||||
|
||||
通过 `AskUserQuestion` 让用户确认/增删物件列表(选项:确认当前列表 / 我要增减物件)。
|
||||
|
||||
如果最终列表为空(脚本中无跨镜头物件),通过 `AskUserQuestion` 确认后跳过此阶段。
|
||||
|
||||
### 3. 生成参考图
|
||||
|
||||
为每个物件生成 2~3 张候选图。宽高比根据物件类型选择:
|
||||
|
||||
| 物件类型 | 宽高比 | 说明 |
|
||||
|----------|--------|------|
|
||||
| 道具/小物件 | **1:1** | 正方形,突出物件细节 |
|
||||
| 载具/动物 | **3:2** 横版 | 展示整体形态和比例 |
|
||||
| 场景地标 | **16:9** 宽屏 | 展示建筑/场所全貌 |
|
||||
|
||||
使用 `all_in_one_image_generation`(model_name: `all_in_one_pro`)或 `nano_banana_image_generation`(model_name: `nano_banana_2`)生成候选图。提示词融合:物件外观描述(从 script.md 汇总)+ 美术风格关键词 + `art_style.md` 的 `风格JSON:` 内容(如在提示词末尾附加 `Art style: {JSON 内容}`)。
|
||||
|
||||
### 4. 用户选图
|
||||
|
||||
启动预览,展示物件卡片和候选图网格,用户为每个物件选定最终参考图。调用 MCP 工具 `preview_and_collect_feedback`:
|
||||
- **script**: `.opencode/skills/mv-creator/scripts/render_preview.py`
|
||||
- **args**: `[".mv/{song_name}/{theme}", "--type", "props", "--lang", "{lang}"]`
|
||||
- **feedback_path**: `.mv/{song_name}/{theme}/feedback.md`
|
||||
|
||||
预览页面展示各物件信息卡片和候选图。
|
||||
每个物件默认选中第一张候选图作为兜底,用户点击图片切换选定。
|
||||
反馈处理逻辑同人设图阶段:
|
||||
- 以 `LGTM` 开头 → 将各物件的选定候选图复制为 `selected.jpg`,更新 `design.md`,进入下一阶段
|
||||
- 包含 `修改建议:` → 解析建议,调整 prompt 重新生成候选图,更新后重新校验和预览
|
||||
|
||||
## 输出格式
|
||||
|
||||
每个物件独立一个目录和设计文件。**严格按用户语言选择对应的 example 和字段名**(参见 SKILL.md「输出控制」和「Markdown 字段名中英对照表」):
|
||||
- 中文用户:参考 `references/props-design-example.md`,字段用中文(`外观描述`、`叙事作用`、`生图设定`、`候选图`、`选定图`),内容用中文
|
||||
- 非中文用户:参考 `references/props-design-example-en.md`,字段用英文(`Appearance`、`Narrative Role`、`Image Settings`、`Candidates`、`Selected`),内容用对应语言
|
||||
|
||||
关键要点:
|
||||
- 一级标题 `#` 用于物件名
|
||||
- 外观描述/Appearance 用列表逐项描述(类型、形态、颜色/材质、尺寸/比例、细节特征)
|
||||
- 生图设定/Image Settings 包含三个子字段:`提示词/Prompt`(英文)、`模型/Model`、`参考图/Reference`
|
||||
|
||||
## 格式校验
|
||||
|
||||
design.md 写入后,**必须**运行校验脚本:
|
||||
|
||||
```bash
|
||||
# 校验单个物件
|
||||
python3 .opencode/skills/mv-creator/scripts/validate_props_design.py .mv/{song_name}/{theme}/props/{item_name}/design.md
|
||||
|
||||
# 校验所有物件
|
||||
python3 .opencode/skills/mv-creator/scripts/validate_props_design.py .mv/{song_name}/{theme}/props/
|
||||
```
|
||||
|
||||
校验规则:必需分区完整性、外观描述列表项、生图设定子字段、候选图非空且文件存在。
|
||||
|
||||
- **通过** → 启动预览供用户选图
|
||||
- **未通过** → 由 LLM 根据校验错误信息自动修正,修复后重新校验
|
||||
|
||||
## 文件存储
|
||||
|
||||
**目录名规则**:`{item_name}` 统一使用英文小写 + 下划线(如 `guitar`、`pickup_truck`),不使用中文或显示名。该目录名是分镜阶段引用物件的唯一标识。
|
||||
|
||||
```
|
||||
./.mv/{song_name}/{theme}/props/
|
||||
├── {item_name}/
|
||||
│ ├── design.md # 物件设计说明
|
||||
│ ├── candidate_01.jpg # 候选图
|
||||
│ ├── candidate_02.jpg
|
||||
│ └── selected.jpg # 最终选定图(用户挑选)
|
||||
```
|
||||
@@ -0,0 +1,110 @@
|
||||
# 分镜规划
|
||||
|
||||
## 目标
|
||||
|
||||
根据脚本为每个场景规划分镜,**引用已生成的人设图、背景图和关键物件图**作为视频生成的参考素材,不再单独生成分镜图。
|
||||
|
||||
## 重点
|
||||
|
||||
- 分镜不再独立生成图片,而是为每幕映射已有的参考图(角色 `selected.jpg`、背景 `selected.jpg`、物件 `selected.jpg`)
|
||||
- 每幕需要编写视频生成提示词,描述画面动态和镜头运动
|
||||
- 分镜文档是视频生成阶段的直接输入
|
||||
|
||||
## 流程
|
||||
|
||||
### 1. 解析脚本与关联素材
|
||||
|
||||
读取 `script.md` 提取每幕信息(场景名称、时间码、场景描述、歌词、视觉风格、关键物件)。
|
||||
结合 `story_concept.md` 中的核心人物描述和各幕场景描述,为每幕标注:
|
||||
|
||||
- **出场角色**:场景中出现的角色名 → 对应 `characters/{character_name}/selected.jpg`
|
||||
- **场景背景**:该幕的场景环境 → 对应 `backgrounds/{bg_name}/selected.jpg`
|
||||
- **关键物件**:该幕出现的关键物件 → 对应 `props/{item_name}/selected.jpg`
|
||||
|
||||
判断规则:
|
||||
- 场景描述中明确提及角色名或角色特征 → 标注该角色
|
||||
- 场景描述为纯空镜/风景/物件特写 → 出场角色标注为 `(无)`
|
||||
- 不确定时倾向于标注角色(宁多勿漏,用户可在预览时修正)
|
||||
|
||||
### 2. 生成分镜文档
|
||||
|
||||
将每幕信息写入 `storyboard.md`,为每幕生成:
|
||||
- **出场角色**:该幕出现的角色**目录名**(即 `characters/` 下的子目录名,不是 design.md 中的 H1 显示名)
|
||||
- **场景背景**:该幕对应的背景**目录名**(即 `backgrounds/` 下的子目录名)
|
||||
- **关键物件**:该幕出现的关键物件**目录名**(即 `props/` 下的子目录名)
|
||||
- **场景描述**:中文画面描述(从 script.md 同步)
|
||||
- **视频提示词**:英文 prompt,描述画面动态、镜头运动、角色动作,引用参考图(如 `image 1 as background, image 2 as character`)
|
||||
- **参考图列表**:背景 `selected.jpg` + 角色 `selected.jpg`(如有)+ 物件 `selected.jpg`(如有)
|
||||
|
||||
### 3. 生成分镜图
|
||||
|
||||
为每幕生成分镜合成图,让用户在视频生成前预览实际画面构图。
|
||||
|
||||
**生成方式**:使用 `all_in_one_batch_image_generation`(model_name: `all_in_one_pro`)或 `nano_banana_batch_image_generation_v2`(model_name: `nano_banana_2`)批量生成,每幕 1 张。
|
||||
|
||||
**参考图传入**:通过 `image_paths` 传入背景 `selected.jpg` + 角色 `selected.jpg`(如有)+ 关键物件 `selected.jpg`(如有),并在提示词中用 `image 1 as background, image 2 as character` 等引用。
|
||||
|
||||
**提示词**:基于分镜文档的 `视频提示词:` 字段,但去除镜头运动描述(pan、dolly、tracking 等),仅保留画面构图、角色动作、光线氛围和风格描述。这是**静态画面**提示词,不同于视频提示词。
|
||||
|
||||
**宽高比**:统一 `16:9`。
|
||||
|
||||
**存储**:保存到 `storyboard/` 目录,命名 `scene_01.jpg`、`scene_02.jpg` 等(序号与幕序一致)。
|
||||
|
||||
**写入分镜文档**:在 `storyboard.md` 每幕末尾追加 `候选图:` 和 `选定图:` 字段,记录生成的候选图路径。首次生成后默认选定第一张。
|
||||
|
||||
### 4. 用户审阅
|
||||
|
||||
调用 MCP 工具 `preview_and_collect_feedback` 启动预览:
|
||||
- **script**: `.opencode/skills/mv-creator/scripts/render_preview.py`
|
||||
- **args**: `[".mv/{song_name}/{theme}", "--type", "storyboard", "--lang", "{lang}"]`
|
||||
- **feedback_path**: `.mv/{song_name}/{theme}/feedback.md`
|
||||
|
||||
预览页面按时间线展示各幕分镜卡片,显示每幕的参考图缩略图和视频提示词。
|
||||
工具返回反馈内容后:
|
||||
- `LGTM` → 进入视频生成阶段
|
||||
- 其他 → 按建议修改分镜映射或视频提示词,更新 `storyboard.md`,重新校验和预览
|
||||
|
||||
## 输出格式
|
||||
|
||||
**严格按用户语言选择对应的 example 和字段名**(参见 SKILL.md「输出控制」和「Markdown 字段名中英对照表」):
|
||||
- 中文用户:参考 `references/storyboard-example.md`
|
||||
- 非中文用户:参考 `references/storyboard-example-en.md`
|
||||
|
||||
关键要点:
|
||||
|
||||
- 二级标题 `##` 用于每幕场景标题,与 `script.md` 的幕名保持一致
|
||||
- 时间码行格式:`* M:SS - M:SS (Xs)`,与 `script.md` 一致
|
||||
- 歌词引用以 `> ` 开头(从 script.md 同步)
|
||||
- 每幕必须包含以下字段(中文/English):
|
||||
- `出场角色/Characters` — 角色名(逗号分隔),无角色写 `(无)`/`(None)`
|
||||
- `场景背景/Scene Background` — 对应的背景名
|
||||
- `关键物件/Key Props` — 物件名(逗号分隔),无物件可省略
|
||||
- `场景描述/Scene Description` — 画面描述(用户语言),缩进 2 空格
|
||||
- `视频提示词/Video Prompt` — 英文 prompt,描述动态和镜头,引用参考图编号,缩进 2 空格
|
||||
- `参考图/Reference` — 列表形式,每行注明用途,路径相对 theme 目录
|
||||
- `候选图/Candidates` — 列表形式,每行记录候选图文件名和简要说明
|
||||
- `选定图/Selected` — 用户选定的最终分镜图文件名
|
||||
- 场景之间用 `---` 分隔
|
||||
|
||||
## 格式校验
|
||||
|
||||
storyboard.md 写入后,**必须**运行校验脚本:
|
||||
|
||||
```bash
|
||||
python3 .opencode/skills/mv-creator/scripts/validate_storyboard.py .mv/{song_name}/{theme}/storyboard.md
|
||||
```
|
||||
|
||||
校验规则:场景标题格式、时间码格式、必填字段完整性、出场角色目录及 `selected.jpg` 存在性、场景背景目录及 `selected.jpg` 存在性、关键物件目录及 `selected.jpg` 存在性、与 `script.md` 交叉校验(幕数和时间码一致)。
|
||||
|
||||
- **通过** → 启动预览供用户审阅
|
||||
- **未通过** → 由 LLM 根据校验错误信息自动修正,修复后重新校验
|
||||
|
||||
## 文件存储
|
||||
|
||||
```
|
||||
./.mv/{song_name}/{theme}/storyboard.md # 分镜规划文档
|
||||
./.mv/{song_name}/{theme}/storyboard/ # 分镜图目录
|
||||
├── scene_01.jpg # 各幕分镜图
|
||||
├── scene_02.jpg
|
||||
└── ...
|
||||
```
|
||||
@@ -0,0 +1,78 @@
|
||||
# 视频生成
|
||||
|
||||
## 目标
|
||||
|
||||
基于分镜规划和脚本,生成 AI 视频片段。
|
||||
|
||||
## 生成模型
|
||||
|
||||
使用 **Seedance 2.0** 多参考图模式(`seedance_multimodal_video`):
|
||||
- 模型:`seedance2.0`
|
||||
- 输入:多张参考图(背景参考图 + 角色参考图 + 物件参考图),通过 `reference_image_paths` 传入
|
||||
- 时长:与脚本每幕的秒数一致(4~15 秒),从时间码标签直接读取
|
||||
- 分辨率:`720p`,比例 `16:9`
|
||||
- 音频:`generate_audio: false`(最终合成时统一嵌入歌曲音频)
|
||||
- **并发度:1**(逐幕顺序生成,避免 API 限流)
|
||||
|
||||
## 参考图策略
|
||||
|
||||
视频生成使用分镜选定图作为画面构图基准,配合角色和物件参考图确保一致性:
|
||||
|
||||
| 场景类型 | reference_image_paths | prompt 引用示例 |
|
||||
|----------|----------------------|----------------|
|
||||
| 角色场景 | [分镜选定图, 角色 selected.jpg] | `use image 1 as the scene composition, image 2 as the character reference` |
|
||||
| 角色 + 物件场景 | [分镜选定图, 角色 selected.jpg, 物件 selected.jpg] | `use image 1 as the scene composition, image 2 as the character, image 3 as the prop reference` |
|
||||
| 多角色场景 | [分镜选定图, 角色1 selected.jpg, 角色2 selected.jpg] | `use image 1 as the scene composition, image 2 and image 3 as character references` |
|
||||
| 纯空镜 | [分镜选定图] | `use image 1 as the scene composition` |
|
||||
|
||||
> 分镜选定图路径:`storyboard/scene_{NN}.jpg`(其中 NN 为幕序号,如 scene_01.jpg)
|
||||
|
||||
## 生成流程
|
||||
|
||||
1. 读取 `storyboard.md`,提取每幕的参考图路径、出场角色、场景背景和时长
|
||||
2. 为每幕编写视频提示词(英文),描述画面动态和镜头运动,引用参考图
|
||||
3. 逐幕调用 `seedance_multimodal_video`,传入参考图列表和提示词
|
||||
4. 每幕生成完成后,将视频保存到 `clips/` 目录
|
||||
5. 全部完成后,生成 `video_prompts.md` 记录每幕的提示词和生成参数
|
||||
|
||||
## 提示词结构
|
||||
|
||||
```
|
||||
[参考图引用], [主体描述], [动作/运动], [环境/场景], [光线/氛围], [镜头语言], [风格修饰]
|
||||
```
|
||||
|
||||
## 提示词原则
|
||||
|
||||
- 使用英文撰写(AI 视频工具对英文支持最佳)
|
||||
- 先引用参考图(image 1, image 2...),再写主体和动作
|
||||
- 明确指定镜头运动方式(camera pan left, dolly in, static shot...)
|
||||
- 包含光线描述(golden hour, neon lights, dramatic shadows...)
|
||||
- 添加风格标签(oil painting, thick brushstrokes, impressionist...)
|
||||
- 避免否定描述,用正面表述替代
|
||||
|
||||
## 文件存储
|
||||
|
||||
```
|
||||
./.mv/{song_name}/{theme}/video_prompts.md # 视频提示词
|
||||
./.mv/{song_name}/{theme}/clips/{scene_name}.mp4 # 生成的视频片段
|
||||
```
|
||||
|
||||
## 视频预览
|
||||
|
||||
全部视频生成完成后,调用 MCP 工具 `preview_and_collect_feedback` 启动视频预览:
|
||||
- **script**: `.opencode/skills/mv-creator/scripts/render_preview.py`
|
||||
- **args**: `[".mv/{song_name}/{theme}", "--type", "videos", "--lang", "{lang}"]`
|
||||
- **feedback_path**: `.mv/{song_name}/{theme}/feedback.md`
|
||||
|
||||
预览页面展示:
|
||||
- 每幕视频播放器(支持播放/暂停/拖拽)
|
||||
- 场景信息(标题、时间码、歌词、出场角色、场景描述)
|
||||
- 视频提示词(可折叠)
|
||||
- 参考图缩略图
|
||||
- 生成状态(已生成 / 缺失)
|
||||
|
||||
用户反馈处理(工具返回反馈内容后):
|
||||
- `LGTM` → 进入视频剪辑阶段
|
||||
- `@scene_03 镜头运动改为缓慢推进` → 修改指定幕的提示词并重新生成
|
||||
- `@scene_05 重新生成` → 使用原提示词重新生成指定幕
|
||||
- 可上传参考图辅助修改:`@scene_02 参考 @uploaded/ref.jpg 的构图`
|
||||
@@ -0,0 +1,115 @@
|
||||
# 视频剪辑
|
||||
|
||||
## 目标
|
||||
|
||||
将所有生成的视频片段按脚本时间线拼接,嵌入歌曲音频,输出完整 MV 成片。
|
||||
|
||||
## 工具
|
||||
|
||||
本阶段全部使用本地 `ffmpeg` / `ffprobe` 命令完成,不调用任何远程 MCP 工具。
|
||||
|
||||
## 流程
|
||||
|
||||
### 1. 素材校验
|
||||
|
||||
检查 `clips/` 目录下所有视频片段是否齐全(与 `storyboard.md` 幕数一致),通过 `ffprobe` 检查分辨率/帧率是否统一。如有不一致,先用 `ffmpeg` 统一转码:
|
||||
|
||||
```bash
|
||||
# 检查视频信息
|
||||
ffprobe -v error -select_streams v:0 -show_entries stream=width,height,r_frame_rate,codec_name -of csv=p=0 <input.mp4>
|
||||
|
||||
# 统一转码(示例:统一为 1280x720, 30fps)
|
||||
ffmpeg -i <input.mp4> -vf "scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2" -r 30 -c:v libx264 -preset medium -crf 18 -an <output.mp4>
|
||||
```
|
||||
|
||||
### 2. 裁剪时长
|
||||
|
||||
根据脚本时间码,用 `ffmpeg` 将每个视频片段裁剪至对应幕的精确时长:
|
||||
|
||||
```bash
|
||||
ffmpeg -i <input.mp4> -t <duration_seconds> -c copy <output.mp4>
|
||||
```
|
||||
|
||||
如裁剪后需要重编码(例如精确到帧),去掉 `-c copy`。
|
||||
|
||||
### 3. 拼接视频
|
||||
|
||||
使用 `ffmpeg` concat demuxer 将所有裁剪后的视频片段按脚本顺序拼接:
|
||||
|
||||
```bash
|
||||
# 1. 生成 filelist.txt
|
||||
echo "file 'clip_01.mp4'" > filelist.txt
|
||||
echo "file 'clip_02.mp4'" >> filelist.txt
|
||||
# ... 按顺序列出所有片段
|
||||
|
||||
# 2. 拼接
|
||||
ffmpeg -f concat -safe 0 -i filelist.txt -c copy <merged.mp4>
|
||||
```
|
||||
|
||||
如果片段编码参数不完全一致,改用重编码拼接:
|
||||
|
||||
```bash
|
||||
ffmpeg -f concat -safe 0 -i filelist.txt -c:v libx264 -preset medium -crf 18 -an <merged.mp4>
|
||||
```
|
||||
|
||||
### 4. 嵌入音频
|
||||
|
||||
用 `ffmpeg` 将歌曲音频嵌入拼接后的视频,替换原有音频(如有):
|
||||
|
||||
```bash
|
||||
ffmpeg -i <merged.mp4> -i <audio_file> -c:v copy -c:a aac -b:a 192k -map 0:v:0 -map 1:a:0 -shortest <output_with_audio.mp4>
|
||||
```
|
||||
|
||||
音频文件路径从 `lyric.md` 的 `音频:` / `Audio:` 行读取。
|
||||
|
||||
### 5. 字幕烧录(可选)
|
||||
|
||||
通过 `AskUserQuestion` 询问用户是否需要烧录歌词字幕(选项:需要 / 不需要)。
|
||||
|
||||
- **需要** → 先从 `lyric.md` 提取时间码和歌词,生成 SRT 字幕文件,再用 `ffmpeg` 烧录:
|
||||
|
||||
```bash
|
||||
# 烧录字幕(硬字幕)
|
||||
ffmpeg -i <input.mp4> -vf "subtitles=<subtitle.srt>:force_style='FontSize=24,PrimaryColour=&HFFFFFF,OutlineColour=&H000000,Outline=2,Alignment=2'" -c:v libx264 -preset medium -crf 18 -c:a copy <output.mp4>
|
||||
```
|
||||
|
||||
- **不需要** → 跳过此步
|
||||
|
||||
### 6. 输出成片
|
||||
|
||||
将最终成片保存到 `output/final.mp4`。
|
||||
|
||||
向用户输出项目总结(见下方「完成总结」模板),包含所有阶段的关键产物路径和统计信息。
|
||||
|
||||
## 完成总结
|
||||
|
||||
成片输出后,向用户汇报以下信息:
|
||||
|
||||
```
|
||||
MV 创作完成总结
|
||||
-----------------
|
||||
歌曲:{song_name}
|
||||
主题:{theme}
|
||||
总时长:M:SS
|
||||
总幕数:N 幕
|
||||
|
||||
项目目录:.mv/{song_name}/{theme}/
|
||||
├── 歌词拆解:lyric.md
|
||||
├── 故事概念:story_concept.md
|
||||
├── 脚本:script.md(N 幕,含动态描述)
|
||||
├── 美术风格:art_style.md
|
||||
├── 角色设计:{character_count} 个角色
|
||||
├── 背景图:{background_count} 个场景背景
|
||||
├── 关键物件:{props_count} 个物件(如有)
|
||||
├── 分镜规划:storyboard.md(N 幕)
|
||||
├── 视频片段:clips/(N 段)
|
||||
└── 成片:output/final.mp4
|
||||
|
||||
成片路径:.mv/{song_name}/{theme}/output/final.mp4
|
||||
```
|
||||
|
||||
## 文件存储
|
||||
|
||||
```
|
||||
./.mv/{song_name}/{theme}/output/final.mp4 # 最终成片
|
||||
```
|
||||
@@ -0,0 +1,279 @@
|
||||
# Music Video (MV) Production Skill
|
||||
|
||||
When the user wants to create a Music Video (MV), follow this workflow **instead of** the default video generation process.
|
||||
|
||||
## STEP 0: CHECK RESOURCES
|
||||
|
||||
1. **Music file**: If not provided, ask the user to provide one or generate using `hilo_tools_music_generation_with_official` (audio agent).
|
||||
2. **LRC lyrics**: Ask the user: "Do you have LRC lyrics? If yes, provide them. If not, I'll use audio analysis to extract lyrics."
|
||||
3. **MV duration**: If user specifies a duration (e.g. "30 seconds"), call **editing agent** with `hilo_tools_audio_subclip_batch` to trim the audio first.
|
||||
4. Get audio duration via **editing agent** using `hilo_tools_audio_meta`.
|
||||
|
||||
## STEP 0.5: ANALYZE MUSIC (DIRECT — do NOT delegate to sub-agent)
|
||||
|
||||
**IMPORTANT**: `hilo_tools_read_media` is YOUR direct tool (orchestrator), NOT a sub-agent tool. Call it directly — do NOT delegate this step to the editing agent or any other sub-agent.
|
||||
|
||||
Call `hilo_tools_read_media` with the music file to understand:
|
||||
- **question**: "Analyze this music track in detail: 1) How many singers? Solo or duet? 2) Gender of each singer (male/female) 3) Music genre and style (pop, rock, ballad, electronic, hip-hop, etc.) 4) Mood and emotion (romantic, energetic, melancholic, etc.) 5) Tempo (slow/medium/fast, approximate BPM) 6) Language of lyrics 7) Any notable instruments or production style 8) Suggested visual aesthetic for an MV (realistic, cyberpunk, fantasy, anime, etc.)"
|
||||
|
||||
Use this analysis throughout the workflow:
|
||||
- **Step 1**: Inform `language` parameter for ASR
|
||||
- **Step 2**: Guide visual style, character design, and scene choices in the storyboard
|
||||
- **Step 2**: Determine `is_duet` and `singer_gender` for each lyric segment
|
||||
|
||||
## STEP 1: EXTRACT LYRICS WITH TIMESTAMPS
|
||||
|
||||
**If user provided LRC lyrics**: Parse them directly into timestamped format.
|
||||
|
||||
**If no LRC lyrics**: Call **editing agent** with `hilo_tools_audio_transcribe_lyrics` to run ASR:
|
||||
- `audio_path`: the music file path
|
||||
- `total_duration`: from Step 0's `hilo_tools_audio_meta` result
|
||||
- `language`: "zh" for Chinese, "en" for English, "auto" for auto-detect
|
||||
- Chinese uses Tencent Cloud ASR, other languages use Whisper ASR
|
||||
- Returns segments with word-level timestamps, plus [prelude]/[interlude]/[outro] markers
|
||||
- Result is saved to a JSON file — read it to get the full segment data
|
||||
|
||||
Format the extracted lyrics as:
|
||||
```
|
||||
[segment_start]lyric_text|singer_gender:male/female/mixed/null
|
||||
[0.00][prelude]|singer_gender:null
|
||||
[10.72]First lyric line|singer_gender:female
|
||||
[14.68]Second lyric line|singer_gender:male
|
||||
...
|
||||
[total_duration:XXX.XX]
|
||||
[is_duet:true/false]
|
||||
```
|
||||
|
||||
## STEP 2: GENERATE MV SCRIPT
|
||||
|
||||
As the orchestrator, generate a complete MV storyboard script yourself (this is an LLM task, no sub-agent needed). The script must include:
|
||||
|
||||
**Be creative and cinematic** — don't just illustrate the lyrics literally. Use your imagination to create visually compelling scenes that capture the *emotion* of the music:
|
||||
- Use **metaphorical imagery**: a breakup song doesn't need to show two people arguing — it could show autumn leaves falling, an empty chair, rain on a window
|
||||
- Create **visual contrast**: alternate between intimate close-ups and grand wide shots, warm and cool tones, stillness and motion
|
||||
- Design **scene variety**: 4-6 distinct scenes minimum. Don't reuse the same location for every narrative segment — take the viewer on a journey (city streets -> rooftop -> seaside -> forest -> studio)
|
||||
- Add **cinematic moments**: slow-motion, silhouettes, reflections, light flares, time-lapse transitions
|
||||
- Match **energy to music**: verse = gentle/intimate scenes, chorus = dramatic/expansive scenes, interlude = atmospheric/abstract visuals
|
||||
|
||||
### Visual Style
|
||||
Based on lyrics mood/genre, choose one: `realistic`, `cyberpunk`, `fantasy`, `cartoon_3d`. Apply consistently to ALL prompts.
|
||||
|
||||
### Characters
|
||||
```json
|
||||
{
|
||||
"character_id": "char_singer_male",
|
||||
"name": "Singer Name",
|
||||
"image_prompt": "style keywords + ethnicity, appearance, clothing, pose, expression. Simple background.",
|
||||
"is_singer": true,
|
||||
"singer_gender": "male"
|
||||
}
|
||||
```
|
||||
- MUST include singer character(s): one for solo, two for duet
|
||||
- Singer character_id format: `char_singer_male` / `char_singer_female`
|
||||
- Make characters visually attractive and distinctive
|
||||
|
||||
### Scenes
|
||||
```json
|
||||
{
|
||||
"scene_id": "scene_performance",
|
||||
"name": "Performance Stage",
|
||||
"image_prompt": "Realistic photography, recording studio with warm lighting, clean background. NO people."
|
||||
}
|
||||
```
|
||||
- MUST include `scene_performance` for all performance segments
|
||||
- NO people in scene images — pure environment only
|
||||
|
||||
### Script Segments
|
||||
```json
|
||||
{
|
||||
"start": 0.0, "end": 10.5,
|
||||
"lyrics": "[prelude]",
|
||||
"scene_id": "scene_001",
|
||||
"characters_in_scene": [],
|
||||
"action": "...", "camera": "...", "lighting": "...", "visual_focus": "...",
|
||||
"segment_type": "narrative",
|
||||
"singer_gender": null
|
||||
}
|
||||
```
|
||||
|
||||
**Timing rules**:
|
||||
- Segments MUST be continuous (no gaps)
|
||||
- First segment starts at 0.0, last ends at total_duration
|
||||
- Duration: 3-15 seconds (hard limits)
|
||||
- **Preferred duration: 7-10 seconds per segment**. Fewer, longer segments produce more coherent storytelling — each video clip has enough time to develop a visual narrative. Avoid splitting lyrics into many short 3-4s segments; instead, merge adjacent lyrics into one 7-10s segment whenever possible.
|
||||
- Performance segments: <= 10 seconds
|
||||
|
||||
**Segment types**:
|
||||
- `narrative`: Storytelling scenes, no lip-sync. Can include characters (walking, playing guitar, gazing into distance, etc.) or pure environment shots. For [prelude], [interlude], [outro], and most lyric segments.
|
||||
- `performance`: Singer performing directly to camera, lip-sync required. MUST use `scene_performance`. Only for key emotional moments.
|
||||
|
||||
**Performance segment selection — vary it**:
|
||||
- Do NOT only pick choruses for performance. Scatter performance segments **unpredictably** across the song — a quiet verse line sung with intense eye contact can be more impactful than yet another chorus lip-sync.
|
||||
- Good candidates: opening hook, emotional turning point in a verse, bridge, a single powerful line before the final chorus, the last line of the song.
|
||||
- Avoid patterns like "every chorus = performance" — this feels repetitive. Mix it up so the viewer is surprised when the singer suddenly looks at them.
|
||||
- **Never place two performance segments back-to-back** — always separate them with at least one narrative segment. Consecutive lip-sync shots feel jarring and break the visual flow.
|
||||
|
||||
**Segment type ratio — IMPORTANT**:
|
||||
- **~60% narrative, ~40% performance** by total duration. This is validated by `hilo_tools_validate_mv_storyboard` and will FAIL if performance exceeds 50%.
|
||||
- Do NOT make every lyric segment a performance segment. Most lyric segments should be narrative (storytelling visuals with the lyrics as subtitles).
|
||||
- [prelude], [interlude], [outro] are always `narrative`.
|
||||
- A typical 3-minute MV should have ~6-8 performance segments and ~15-18 narrative segments.
|
||||
|
||||
### Validate
|
||||
After generating the script, call **editing agent** with `hilo_tools_validate_mv_storyboard`:
|
||||
- Pass segments, total_duration, scenes, and characters
|
||||
- Fix all errors and re-validate until PASSED
|
||||
|
||||
## STEP 3: GENERATE IMAGES
|
||||
|
||||
Call **image agent** to generate all character and scene images:
|
||||
|
||||
1. **Character images**: Use each character's `image_prompt`. Aspect ratio: `16:9`.
|
||||
2. **Scene images**: Use each scene's `image_prompt`. Aspect ratio: `16:9`.
|
||||
|
||||
Include ALL prompts in one task_description to the image agent for batch generation.
|
||||
|
||||
**IMPORTANT**: Do NOT specify filenames like "Save as: xxx.png" in the task_description — the image agent will try to rename files which breaks assets.json tracking. Just describe the images by their IDs (e.g. "Image 1 — char_singer_male") and use the returned paths directly.
|
||||
|
||||
## STEP 4: CONFIRM IMAGES WITH USER
|
||||
|
||||
Present all generated images to the user. Ask if any need adjustments. Regenerate as needed.
|
||||
|
||||
## STEP 5: GENERATE VIDEOS
|
||||
|
||||
For each script segment, the flow is: **reference images -> image agent generates first frame -> video agent generates video**.
|
||||
|
||||
Do NOT use character/scene reference images directly as first_frame_images for video generation! Reference images are portraits/environments — they need to be composited into a proper scene frame first.
|
||||
|
||||
**IMPORTANT — Batch first, then batch again**: Call image agent ONCE to batch-generate ALL first frames for ALL segments, then call video agent to batch-generate ALL videos. Do NOT interleave (generate one first frame -> one video -> next first frame -> next video).
|
||||
|
||||
### Step 5a: Generate ALL First Frame Images (image agent — one batch call)
|
||||
|
||||
Call **image agent** ONCE with ALL segment first frames in one batch:
|
||||
- For each segment, pass the scene image + character images as reference (`image_paths`)
|
||||
- Prompt should describe the exact scene composition: character placement, action pose, camera angle, lighting — matching the segment's `action`, `camera`, `lighting`, `visual_focus` fields
|
||||
- Aspect ratio: `16:9`
|
||||
- For performance segments: do NOT include microphones in the prompt — they interfere with lip-sync
|
||||
- **IMPORTANT — Face visibility**: ALL character first frames MUST show the character **facing the camera with a clear, frontal face**. Avoid side profiles, back views, or obscured faces. If the first frame doesn't show the character's real face clearly, Official video generation won't know what the face looks like and will hallucinate a different person's face in the video — destroying character consistency across the MV.
|
||||
|
||||
### Step 5b: Generate NARRATIVE Videos (video agent — Official)
|
||||
|
||||
After ALL first frames are generated, call video agent to generate videos for **narrative segments only**. Performance segments will be generated separately in Step 5d using Wan I2V with audio sync.
|
||||
|
||||
**IMPORTANT — Model selection**: Narrative video generation MUST use `hilo_tools_official_videos_generation` (Official Hilo) at default 768P resolution. Performance videos use Wan I2V. Note that the two models may output different resolutions/FPS — this will be normalized in Step 5e before assembly.
|
||||
|
||||
**IMPORTANT — No last_frame_images**: MV segments each have their own unique first frame — do NOT use `last_frame_images` for scene transitions. Explicitly tell the video agent: "Do NOT set last_frame_images, set all to null."
|
||||
|
||||
**IMPORTANT — Minimal motion prompts**: Video prompts should describe **subtle, gentle movements** — slight head turns, soft breathing, slow camera pan, gentle wind in hair. Do NOT describe large or fast actions (running, jumping, dramatic gestures, quick camera movement). Large motion causes the character's face to deform or morph during generation, which ruins lip-sync quality and visual consistency. The less the face moves, the better the final result.
|
||||
|
||||
**Duration strategy**: Official `duration: 6` actually produces ~5.88s, and `duration: 10` produces ~10.12s. Choose based on segment target duration:
|
||||
- Segment <= 5.8s -> generate `duration: 6` (actual ~5.88s), then trim
|
||||
- Segment > 5.8s -> generate `duration: 10` (actual ~10.12s), then trim
|
||||
- Segment > 10s -> generate `duration: 10`, then **extend** (see Step 5c)
|
||||
|
||||
Call video agent ONCE to generate ALL segment videos (narrative + performance together):
|
||||
- `first_frame_images`: ALL generated **narrative** first frames from Step 5a
|
||||
- `prompts`: ALL **narrative** video motion descriptions
|
||||
- `duration`: 6 or 10 per segment based on strategy above
|
||||
|
||||
### Step 5c: Adjust NARRATIVE Videos to Exact Segment Duration
|
||||
|
||||
**CRITICAL — Duration matching**: Official generates fixed 6s or 10s videos, but segments have varying durations. If not adjusted, the concatenated video will NOT match the audio, causing desync. This step MUST happen BEFORE Step 5d.
|
||||
|
||||
Call **editing agent** to adjust EVERY **narrative** video to its exact segment duration (performance videos are handled in Step 5d):
|
||||
|
||||
#### Case 1: Video longer than target -> Trim
|
||||
```bash
|
||||
ffmpeg -i <video> -t <target_duration> -an <output> -y
|
||||
```
|
||||
**Do NOT use `-c:v copy`** — stream copy can only cut at keyframe boundaries (~0.08-0.12s error per clip, accumulates across segments). Re-encoding is slower but gives frame-accurate cuts.
|
||||
|
||||
#### Case 2: Video shorter than target (segment > 10s) -> Extend then trim
|
||||
|
||||
**This case is NOT optional** — prelude, interlude, outro, and long verse segments commonly exceed 10s. Official max output is ~10s, so these clips WILL be too short. If you skip this, the final MV will be shorter than the audio and `hilo_tools_mv_final_assembly` will reject it.
|
||||
|
||||
**How to identify**: any segment where `target_duration > video_actual_duration`. Typical examples:
|
||||
- Prelude: 13-15s (generated video is ~10s, deficit ~3-5s)
|
||||
- Interlude: 10-30s (generated video is ~10s, deficit up to 20s)
|
||||
- Outro: 15-20s (generated video is ~10s, deficit ~5-10s)
|
||||
- Long verse segments: 11-15s
|
||||
|
||||
**How to extend**:
|
||||
|
||||
1. **Extract last frame** from the short video:
|
||||
```bash
|
||||
ffmpeg -sseof -0.1 -i <video> -vframes 1 -q:v 2 <last_frame.jpg> -y
|
||||
```
|
||||
|
||||
2. **Generate extension video** using the last frame as first_frame_image, with a prompt continuing the same scene/action. Choose duration based on how much more is needed:
|
||||
- Need <= 5.8s more -> generate `duration: 6`
|
||||
- Need > 5.8s more -> generate `duration: 10`
|
||||
- Need > 16s more -> repeat this process (chain multiple extensions)
|
||||
|
||||
3. **Concatenate** original + extension(s):
|
||||
```bash
|
||||
echo "file '<original.mp4>'" > concat.txt
|
||||
echo "file '<extension.mp4>'" >> concat.txt
|
||||
ffmpeg -f concat -safe 0 -i concat.txt -c copy <joined.mp4> -y
|
||||
```
|
||||
|
||||
4. **Trim** the joined result to exact target duration:
|
||||
```bash
|
||||
ffmpeg -i <joined.mp4> -t <target_duration> -an <output> -y
|
||||
```
|
||||
|
||||
**Batch all Case 2 segments together** — collect all segments that need extension, generate all extension videos in one batch call to the video agent, then concatenate and trim each one. Do not interleave with Case 1 processing.
|
||||
|
||||
#### General rules
|
||||
- Always use `-an` to strip audio (final assembly adds the original music)
|
||||
- **CRITICAL**: After adjusting all videos, **SUM all clip durations and compare to `total_duration`**. If the drift exceeds 0.5 seconds, `hilo_tools_mv_final_assembly` will **REJECT the assembly with a detailed error**. You MUST fix the short clips before retrying. Common causes: segments > 10s that were not extended (Case 2 above).
|
||||
- Process ALL segments (narrative + performance) — not just the long ones
|
||||
- **DO NOT SKIP Case 2 for segments > 10s** — prelude, interlude, outro, and long verse segments all need extension. Skipping this is the #1 cause of lip-sync desync.
|
||||
|
||||
### Step 5d: Generate Performance Videos with Audio Sync (Wan I2V)
|
||||
|
||||
**Do NOT use the Official + Kling lip-sync pipeline for performance segments.** Instead, use Wan I2V which generates video directly driven by audio — the mouth movements are naturally synced from the start, with no post-processing needed.
|
||||
|
||||
**Sub-step 1**: Call **editing agent** with `hilo_tools_audio_subclip_batch` to extract ALL performance audio segments in one batch call.
|
||||
|
||||
**Sub-step 2**: Call **video agent** with `hilo_tools_batch_wan_i2v_generation` to generate ALL performance videos in one call (internally limited to 3 concurrent to avoid API rate limits):
|
||||
- `image_paths`: ALL performance first-frame images from Step 5a
|
||||
- `audio_paths`: ALL corresponding subclipped audio segments from Sub-step 1
|
||||
- `prompts`: subtle motion prompts for each — the singer singing with gentle expression changes, soft breathing, slight head movement. **Keep motion minimal** to preserve face consistency.
|
||||
- `durations`: choose per clip based on audio length — 5 (for audio <= 5s) or 10 (for audio > 5s). Audio must be <= 10s (performance segments are capped at 10s by the storyboard validator).
|
||||
|
||||
**Sub-step 3 — Trim to exact duration**: Wan I2V outputs 5s or 10s video, but the performance segment may be e.g. 6.63s. Trim each output to the exact segment duration using ffmpeg:
|
||||
```bash
|
||||
ffmpeg -i <wan_output.mp4> -t <target_duration> -an <output> -y
|
||||
```
|
||||
The `-an` flag also strips the embedded audio (final assembly will use the original music track).
|
||||
|
||||
**Why Wan I2V instead of Official + Kling**: Kling lip-sync frequently fails with "face too large/small" errors and produces unnatural mouth movements. Wan I2V drives video generation directly from audio, producing naturally synced results without any post-processing step.
|
||||
|
||||
**Note**: Narrative segments still use Official (Step 5b) — only performance segments use Wan I2V.
|
||||
|
||||
### Step 5e: Normalize Clip Specs Before Assembly (MANDATORY)
|
||||
|
||||
Before final assembly, use ffprobe to check that ALL clips share the same resolution and FPS. If any mismatch is found (e.g. Wan I2V and Official Hilo clips differ), rescale the outliers to match the majority using ffmpeg, then re-verify.
|
||||
|
||||
## STEP 6: FINAL ASSEMBLY
|
||||
|
||||
Call **editing agent** with `hilo_tools_mv_final_assembly`:
|
||||
- `video_paths`: ALL generated video paths in segment order
|
||||
- `audio_path`: the ORIGINAL full audio file
|
||||
- `subtitle_segments`: Build from lyrics timestamps:
|
||||
```json
|
||||
[
|
||||
{"word": "complete lyric line", "time_begin": 10.5, "time_end": 15.2},
|
||||
{"word": "next line", "time_begin": 15.2, "time_end": 20.0}
|
||||
]
|
||||
```
|
||||
- Skip instrumental segments ([prelude], [interlude], [outro])
|
||||
- `time_begin` and `time_end` in seconds
|
||||
|
||||
## STEP 7: PRESENT RESULT
|
||||
|
||||
Show the final MV to the user with a summary:
|
||||
- Audio info (duration, language)
|
||||
- Script overview (characters, scenes, segment count)
|
||||
- Final MV path
|
||||
Reference in New Issue
Block a user