weitao
/
popiart-node-canvas
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Browse Source

docs(06): create phase plan for Video & Polish 

Phase 06: Video & Polish
- 4 plans created (06-01 through 06-04)
- 11 total tasks defined
- Addresses ISS-001 (model parameter adaptation)

Plans:
- 06-01: GenerateVideo node with video model selector
- 06-02: Video playback in output node
- 06-03: Custom model parameters from provider schemas
- 06-04: Edge case handling and final polish

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
handoff-20260429-1057
shrimbly 6 months ago
parent
32581ced85
commit
2c9874d3ca
4 changed files with 674 additions and 0 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 137
      .planning/phases/06-video-and-polish/06-01-PLAN.md
  2. 153
      .planning/phases/06-video-and-polish/06-02-PLAN.md
  3. 178
      .planning/phases/06-video-and-polish/06-03-PLAN.md
  4. 206
      .planning/phases/06-video-and-polish/06-04-PLAN.md

137
.planning/phases/06-video-and-polish/06-01-PLAN.md
View File

@ -0,0 +1,137 @@
---
phase: 06-video-and-polish
plan: 01
type: execute
---
<objective>
Create GenerateVideo node supporting video-capable models from Replicate and fal.ai.
Purpose: Enable video generation workflows using the same multi-provider infrastructure built for images.
Output: Working GenerateVideoNode with provider/model selection and video output display.
</objective>
<execution_context>
~/.claude/get-shit-done/workflows/execute-phase.md
~/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
# Prior phase context (established patterns):
@.planning/phases/03-generate-node-refactor/03-01-SUMMARY.md
@.planning/phases/03-generate-node-refactor/03-02-SUMMARY.md
# Key source files:
@src/components/nodes/GenerateImageNode.tsx
@src/app/api/generate/route.ts
@src/types/index.ts
@src/lib/providers/types.ts
@src/store/workflowStore.ts
**Tech stack available:** React, @xyflow/react, Zustand, Next.js API routes
**Established patterns:**
- Provider dropdown with conditional visibility based on API key
- Model selector fetching from /api/models with capabilities filter
- selectedModel object with {provider, modelId, displayName}
- Server-side provider dispatch in /api/generate
**Constraining decisions:**
- GenerateVideo as separate node type (not combined with GenerateImage)
- video capabilities: text-to-video, image-to-video
</context>
<tasks>
<task type="auto">
<name>Task 1: Create GenerateVideoNode component</name>
<files>
src/components/nodes/GenerateVideoNode.tsx,
src/components/nodes/index.ts,
src/types/index.ts,
src/store/workflowStore.ts,
src/components/WorkflowCanvas.tsx
</files>
<action>
Create GenerateVideoNode based on GenerateImageNode with these changes:
1. In types/index.ts:
- Add "generateVideo" to NodeType union
- Create GenerateVideoNodeData interface (copy NanoBananaNodeData, change outputImage to outputVideo: string | null, remove image-specific fields like aspectRatio, resolution, imageHistory)
- Add to WorkflowNodeData union
2. In GenerateVideoNode.tsx:
- Copy structure from GenerateImageNode
- Change VIDEO_CAPABILITIES to ["text-to-video", "image-to-video"]
- Fetch models with capabilities=text-to-video,image-to-video
- Remove Gemini provider option (Gemini doesn't do video)
- Show fal.ai always, Replicate when configured (like image node)
- Display video output using HTML5 video element with controls
- Remove aspect ratio/resolution selectors (video models have their own params)
3. In workflowStore.ts:
- Add createDefaultNodeData case for "generateVideo"
- Add defaultDimensions entry (same as nanoBanana: 200x280)
- Add to getConnectedInputs to extract video from GenerateVideo nodes (check for outputVideo)
4. In WorkflowCanvas.tsx:
- Add GenerateVideoNode to nodeTypes
- Add minimap color (purple: "#9333ea")
5. In nodes/index.ts:
- Export GenerateVideoNode
</action>
<verify>npm run build passes, no TypeScript errors</verify>
<done>GenerateVideoNode renders, shows provider/model dropdowns, can be added to canvas</done>
</task>
<task type="auto">
<name>Task 2: Add video generation to API route</name>
<files>
src/app/api/generate/route.ts,
src/types/index.ts
</files>
<action>
Extend /api/generate to handle video generation:
1. Update GenerateResponse in types/index.ts:
- Add optional video?: string field alongside image
- Add optional contentType?: "image" | "video" field
2. In route.ts generateWithReplicate:
- After fetching output, detect if output is video (check content-type header for video/*)
- If video, return { type: "video", data: dataUrl } in outputs array
- Video data URL format: data:video/mp4;base64,...
3. In route.ts generateWithFal:
- fal.ai video models return { video: { url: "..." } } not images array
- Check for result.video?.url in addition to result.images
- If video URL found, fetch and convert to base64 with video mime type
4. In main POST handler:
- Return video field in response when output type is video
- Set contentType: "video" when returning video
5. Handle large video files:
- Videos can be large, log warning if >10MB but still return
- Consider returning URL directly for very large videos (>20MB) with a videoUrl field
</action>
<verify>
Test with curl: Create a prediction for a video model and verify video data returns
</verify>
<done>API returns video base64 or URL for video-capable models, contentType field indicates output type</done>
</task>
</tasks>
<verification>
Before declaring plan complete:
- [ ] npm run build succeeds without errors
- [ ] GenerateVideoNode appears in node palette
- [ ] Video model dropdown populates from fal.ai/Replicate
- [ ] API route handles video outputs correctly
</verification>
<success_criteria>
- All tasks completed
- All verification checks pass
- No TypeScript errors
- GenerateVideoNode can be added to canvas and configured
- Video generation API path is ready (execution tested in 06-02 with playback)
</success_criteria>
<output>
After completion, create `.planning/phases/06-video-and-polish/06-01-SUMMARY.md`
</output>

153
.planning/phases/06-video-and-polish/06-02-PLAN.md
View File

@ -0,0 +1,153 @@
---
phase: 06-video-and-polish
plan: 02
type: execute
---
<objective>
Add video playback support to OutputNode and complete video generation workflow.
Purpose: Enable end-to-end video generation with playback and download.
Output: OutputNode plays video content, download works for video files.
</objective>
<execution_context>
~/.claude/get-shit-done/workflows/execute-phase.md
~/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
# Prior plan in this phase:
@.planning/phases/06-video-and-polish/06-01-SUMMARY.md
# Key source files:
@src/components/nodes/OutputNode.tsx
@src/components/nodes/GenerateVideoNode.tsx
@src/types/index.ts
@src/store/workflowStore.ts
**Established patterns:**
- OutputNode receives image via handle, displays in preview area
- Lightbox modal for full-size view
- Download button generates download link
**Constraining decisions:**
- Video displayed with HTML5 video element
- Support both base64 data URLs and HTTP URLs for video
</context>
<tasks>
<task type="auto">
<name>Task 1: Update OutputNode for video detection and playback</name>
<files>
src/components/nodes/OutputNode.tsx,
src/types/index.ts
</files>
<action>
Enhance OutputNode to handle both images and videos:
1. In types/index.ts, update OutputNodeData:
- Add video: string | null field
- Add contentType?: "image" | "video" field
2. In OutputNode.tsx:
- Detect content type from data URL prefix or contentType field
- isVideo check: data.video || data.contentType === "video" || (data.image?.startsWith("data:video/"))
- If video content:
- Replace img element with video element
- Add controls, loop, muted (autoplay with muted), playsinline attributes
- Style same as image: w-full h-full object-contain rounded
- If video in lightbox:
- Use video element in modal with same attributes
- Make it larger: max-h-[90vh] w-auto
- Update placeholder text: "Waiting for image or video"
3. Handle both base64 and URL sources:
- If value starts with "data:" use as src directly
- If value starts with "http" use as src directly
- Video element handles both formats natively
</action>
<verify>npm run build passes, OutputNode component renders without errors</verify>
<done>OutputNode displays video element when video content is connected, plays with controls</done>
</task>
<task type="auto">
<name>Task 2: Update video download and workflow execution</name>
<files>
src/components/nodes/OutputNode.tsx,
src/store/workflowStore.ts
</files>
<action>
1. In OutputNode.tsx handleDownload:
- Detect if content is video (same check as display)
- For video: set download filename to generated-{timestamp}.mp4
- For image: keep generated-{timestamp}.png
- Handle URL-based video: fetch URL, convert to blob, create blob URL for download
2. In workflowStore.ts getConnectedInputs:
- Add case for "generateVideo" node type
- Extract outputVideo field and add to images array (for img2vid workflows)
- Videos can be input to downstream nodes that accept video/image
3. In workflowStore.ts executeWorkflow:
- Add execution case for "generateVideo" node type
- Copy pattern from nanoBanana execution
- Call /api/generate with selectedModel containing video model
- Store result in outputVideo field (not outputImage)
- Update status appropriately
4. Update output node connection:
- In getConnectedInputs, when source is generateVideo, get outputVideo
- Pass to output node which stores in image field (reuse existing flow)
- The OutputNode detects video content via data URL prefix
</action>
<verify>
Create GenerateVideo -> Output workflow, run generation, verify video plays in output
</verify>
<done>Video downloads with .mp4 extension, workflow execution produces video in output node</done>
</task>
<task type="checkpoint:human-verify" gate="blocking">
<what-built>Video generation workflow: GenerateVideo node -> Output node with playback</what-built>
<how-to-verify>
1. Run: npm run dev
2. Add a GenerateVideo node to the canvas
3. Select fal.ai provider, pick a text-to-video model (e.g., any model with "video" in name)
4. Add a Prompt node, connect to GenerateVideo text input
5. Add an Output node, connect GenerateVideo output to it
6. Enter a prompt like "a cat walking in slow motion"
7. Run the workflow (Cmd+Enter)
8. Verify:
- GenerateVideo node shows loading state
- After completion, video appears in GenerateVideo preview
- Output node shows video with playback controls
- Video plays when clicking play button
- Download button saves .mp4 file
- Lightbox shows larger video player
</how-to-verify>
<resume-signal>Type "approved" to continue to next plan, or describe issues to fix</resume-signal>
</task>
</tasks>
<verification>
Before declaring plan complete:
- [ ] npm run build succeeds without errors
- [ ] OutputNode detects and displays video content
- [ ] Video playback works with controls
- [ ] Video download saves .mp4 file
- [ ] End-to-end workflow: Prompt -> GenerateVideo -> Output works
</verification>
<success_criteria>
- All tasks completed
- All verification checks pass
- Human verification approved
- No TypeScript errors
- Complete video generation pipeline working
</success_criteria>
<output>
After completion, create `.planning/phases/06-video-and-polish/06-02-SUMMARY.md`
</output>

178
.planning/phases/06-video-and-polish/06-03-PLAN.md
View File

@ -0,0 +1,178 @@
---
phase: 06-video-and-polish
plan: 03
type: execute
---
<objective>
Add dynamic model parameters fetched from provider schemas.
Purpose: Allow users to customize model-specific parameters (like seed, steps, guidance_scale) based on what each model supports. Addresses ISS-001.
Output: Generate nodes show dynamic parameter inputs based on selected model's schema.
</objective>
<execution_context>
~/.claude/get-shit-done/workflows/execute-phase.md
~/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/ISSUES.md
# Prior plans in this phase:
@.planning/phases/06-video-and-polish/06-01-SUMMARY.md
@.planning/phases/06-video-and-polish/06-02-SUMMARY.md
# Key source files:
@src/app/api/models/route.ts
@src/components/nodes/GenerateImageNode.tsx
@src/components/nodes/GenerateVideoNode.tsx
@src/types/index.ts
**Issue being addressed:** ISS-001 - Generate node should adapt to model requirements
**Established patterns:**
- Models have openapi_schema field from provider APIs (already fetched but not exposed)
- parameters?: Record<string, unknown> passed to generation API
**Constraining decisions:**
- Use provider schema, don't hardcode parameters
- Common parameters to expose: seed, num_inference_steps, guidance_scale, negative_prompt
</context>
<tasks>
<task type="auto">
<name>Task 1: Add schema endpoint and model schema fetching</name>
<files>
src/app/api/models/[modelId]/route.ts,
src/lib/providers/types.ts,
src/app/api/models/route.ts
</files>
<action>
Create endpoint to fetch individual model schema:
1. Create src/app/api/models/[modelId]/route.ts:
- GET /api/models/:modelId (modelId is URL-encoded, e.g., "fal-ai%2Fflux%2Fdev")
- Parse provider from query param: ?provider=fal
- For Replicate:
- Fetch model from https://api.replicate.com/v1/models/{owner}/{name}
- Extract latest_version.openapi_schema.components.schemas.Input
- Return simplified parameter schema
- For fal.ai:
- Fetch from https://fal.run/{modelId}/api (returns OpenAPI spec)
- Extract requestBody schema properties
- Return simplified parameter schema
2. In providers/types.ts, add ModelParameter interface:
```typescript
interface ModelParameter {
name: string;
type: "string" | "number" | "integer" | "boolean" | "array";
description?: string;
default?: unknown;
minimum?: number;
maximum?: number;
enum?: unknown[];
required?: boolean;
}
```
3. Schema simplification:
- Filter to user-relevant params: exclude internal params like "webhook", "sync_mode"
- Include: prompt (if not already handled), seed, num_inference_steps, guidance_scale, negative_prompt, width, height, num_outputs
- Return as ModelParameter[] array
4. Add caching for schema (same pattern as model list, 10 min TTL)
</action>
<verify>
curl "http://localhost:3000/api/models/fal-ai%2Fflux%2Fdev?provider=fal" returns parameter schema
</verify>
<done>API endpoint returns model parameter schema for both Replicate and fal.ai models</done>
</task>
<task type="auto">
<name>Task 2: Add parameter UI component</name>
<files>
src/components/nodes/ModelParameters.tsx,
src/components/nodes/GenerateImageNode.tsx,
src/components/nodes/GenerateVideoNode.tsx
</files>
<action>
Create reusable parameter input component:
1. Create src/components/nodes/ModelParameters.tsx:
- Props: modelId, provider, parameters, onParametersChange
- Fetch schema from /api/models/{modelId}?provider={provider} when modelId changes
- Render appropriate input for each parameter type:
- string: text input (or select if enum)
- number/integer: number input with min/max
- boolean: checkbox
- array: skip for now (complex)
- Show parameter name, description on hover
- Compact styling to fit in node (10px font, tight spacing)
- Collapsible section: "Parameters" header, collapsed by default
- Only show if model is selected (not Gemini)
2. Integrate into GenerateImageNode:
- Import ModelParameters component
- Add below model selector when external provider selected
- Store parameters in nodeData.parameters
- Pass to API call
3. Integrate into GenerateVideoNode:
- Same integration as GenerateImageNode
4. Update node data types:
- Add parameters?: Record<string, unknown> to NanoBananaNodeData
- Already exists in GenerateVideoNodeData if copied from 06-01
</action>
<verify>npm run build passes, parameter inputs appear when selecting external model</verify>
<done>Parameter inputs render based on model schema, values stored in node data</done>
</task>
<task type="auto">
<name>Task 3: Pass parameters through generation pipeline</name>
<files>
src/store/workflowStore.ts,
src/app/api/generate/route.ts
</files>
<action>
Ensure parameters flow through to providers:
1. In workflowStore.ts executeWorkflow:
- For nanoBanana node: include nodeData.parameters in API request body
- For generateVideo node: same inclusion
2. In API route.ts:
- Parameters already spread into predictionInput/requestBody via ...input.parameters
- Verify this is working correctly
- Add logging: log which parameters are being passed
3. Validate parameter types before sending:
- Coerce strings to numbers where schema expects number
- Filter out empty/undefined values
- Don't send default values (let model use its defaults)
4. Handle parameter errors:
- If provider rejects parameter, surface error clearly
- Show which parameter was invalid in error message
</action>
<verify>
Test generation with custom parameters (e.g., seed=12345), verify parameter appears in API logs
</verify>
<done>Custom parameters successfully passed to providers and affect generation output</done>
</task>
</tasks>
<verification>
Before declaring plan complete:
- [ ] npm run build succeeds without errors
- [ ] Schema endpoint returns parameters for Replicate and fal.ai models
- [ ] Parameter inputs appear in Generate nodes for external models
- [ ] Parameter values persist in node data
- [ ] Parameters are sent to providers during generation
- [ ] Custom seed produces reproducible results (if model supports it)
</verification>
<success_criteria>
- All tasks completed
- All verification checks pass
- No TypeScript errors
- ISS-001 addressed: nodes adapt inputs to model requirements
- Users can customize model-specific parameters
</success_criteria>
<output>
After completion, create `.planning/phases/06-video-and-polish/06-03-SUMMARY.md`
</output>

206
.planning/phases/06-video-and-polish/06-04-PLAN.md
View File

@ -0,0 +1,206 @@
---
phase: 06-video-and-polish
plan: 04
type: execute
---
<objective>
Handle edge cases, improve error states, and complete final polish for multi-provider support.
Purpose: Ensure robust, production-ready multi-provider experience with clear error messages and graceful degradation.
Output: Polished multi-provider integration ready for users.
</objective>
<execution_context>
~/.claude/get-shit-done/workflows/execute-phase.md
~/.claude/get-shit-done/templates/summary.md
~/.claude/get-shit-done/references/checkpoints.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/ISSUES.md
# Prior plans in this phase:
@.planning/phases/06-video-and-polish/06-01-SUMMARY.md
@.planning/phases/06-video-and-polish/06-02-SUMMARY.md
@.planning/phases/06-video-and-polish/06-03-SUMMARY.md
# Key source files:
@src/components/nodes/GenerateImageNode.tsx
@src/components/nodes/GenerateVideoNode.tsx
@src/components/nodes/OutputNode.tsx
@src/app/api/generate/route.ts
@src/store/workflowStore.ts
**Issues to address:**
- UAT-001: Provider icons should use real logos (cosmetic, if time permits)
**Edge cases to handle:**
- Model list fails to load
- Generation times out (video generation can be slow)
- Large video files
- Invalid parameters
- Provider rate limits
</context>
<tasks>
<task type="auto">
<name>Task 1: Improve error handling and loading states</name>
<files>
src/components/nodes/GenerateImageNode.tsx,
src/components/nodes/GenerateVideoNode.tsx,
src/components/nodes/ModelParameters.tsx,
src/app/api/generate/route.ts
</files>
<action>
Enhance error handling across generation nodes:
1. In GenerateImageNode/GenerateVideoNode:
- Show clear error when model list fails: "Failed to load models. Check API key."
- Add retry button for model list fetch
- Show loading spinner when fetching models
- Display model fetch error in a toast or inline message
2. In ModelParameters:
- Handle schema fetch failure gracefully: "Parameters unavailable"
- Show loading state while fetching schema
- Cache schema in component state to avoid re-fetching
3. In API route.ts:
- Improve error messages: include model name in errors
- Handle timeout explicitly: "Generation timed out after 5 minutes. Video models may take longer - try again."
- Detect rate limit errors (429) and provide helpful message
- For Replicate: if prediction fails, include failure reason from API
- For fal.ai: parse error response body for details
4. Video-specific handling:
- Warn if video file is very large (>50MB): "Video generated but file is large ({size}MB)"
- If video fetch fails, try to return URL directly as fallback
</action>
<verify>Simulate errors (bad API key, timeout) and verify clear error messages appear</verify>
<done>Clear error messages for all failure modes, retry capability, loading states visible</done>
</task>
<task type="auto">
<name>Task 2: Parameter validation and defaults</name>
<files>
src/components/nodes/ModelParameters.tsx,
src/store/workflowStore.ts
</files>
<action>
Add input validation for model parameters:
1. In ModelParameters:
- Validate number inputs against min/max from schema
- Show validation error inline: red border, error text below
- Prevent invalid values from being saved
- For enum parameters: use select dropdown with options
- Set default values from schema when parameter is first shown
- Reset parameters when model changes
2. In workflowStore executeWorkflow:
- Filter out parameters with invalid values before API call
- Log warning if parameters were filtered
3. Common parameter handling:
- seed: accept integers only, positive
- num_inference_steps: typically 1-100, integer
- guidance_scale: typically 0-20, float
- width/height: typically 256-2048, multiples of 8
- negative_prompt: string, no validation
4. Clear parameters when switching providers:
- Parameters from one provider likely don't apply to another
- Reset parameters state when provider changes
</action>
<verify>Enter invalid parameter values, verify validation errors appear and prevent submission</verify>
<done>Parameters validate against schema, invalid values prevented, defaults applied</done>
</task>
<task type="auto">
<name>Task 3: Final cleanup and testing</name>
<files>
src/store/workflowStore.ts,
src/components/FloatingActionBar.tsx,
src/components/modals/ModelSearchDialog.tsx
</files>
<action>
Final polish and integration testing:
1. Update MODEL.md search dialog:
- Add video capability badges display
- Ensure video models appear when searching
- Filter by capability when opened from GenerateVideo icon (if applicable)
2. Update FloatingActionBar:
- Consider adding video icon for quick GenerateVideo node creation
- Or reuse existing provider icons (clicking them for video)
3. Test complete workflows:
- Text -> GenerateImage -> Output (existing, verify still works)
- Text -> GenerateVideo -> Output (new)
- Image -> GenerateImage (img2img) -> Output
- Image -> GenerateVideo (img2vid) -> Output
4. Verify backward compatibility:
- Load existing workflow with old nanoBanana nodes
- Confirm they still execute correctly
5. Clean up any TODO comments or temporary code from phase development
6. Update types to mark deprecated aliases:
- Add @deprecated JSDoc to NanoBananaNode alias
</action>
<verify>
All four workflow patterns execute successfully:
- Text to image with Replicate
- Text to video with fal.ai
- Image to image with fal.ai
- Image to video with Replicate (if model available)
</verify>
<done>All workflows functional, backward compatible, code cleaned up</done>
</task>
<task type="checkpoint:human-verify" gate="blocking">
<what-built>Complete multi-provider video generation with custom parameters</what-built>
<how-to-verify>
1. Run: npm run dev
2. Test text-to-video workflow:
- Prompt node -> GenerateVideo (fal.ai, pick video model) -> Output
- Enter a prompt, run workflow
- Verify video plays in output node
3. Test custom parameters:
- Select external model in GenerateImage
- Expand "Parameters" section
- Modify a value (e.g., seed=42)
- Run generation, verify parameter affects output
4. Test error handling:
- Remove Replicate API key, try to select Replicate model
- Verify clear error message
5. Test backward compatibility:
- Load an existing workflow with NanoBanana nodes
- Run it, verify Gemini generation still works
6. Test video download:
- Generate a video, click download in output node
- Verify .mp4 file downloads correctly
</how-to-verify>
<resume-signal>Type "approved" to complete Phase 6, or describe issues</resume-signal>
</task>
</tasks>
<verification>
Before declaring plan complete:
- [ ] npm run build succeeds without errors
- [ ] All error states have clear messages
- [ ] Parameter validation works correctly
- [ ] All four workflow patterns tested
- [ ] Backward compatibility verified
- [ ] Human verification approved
</verification>
<success_criteria>
- All tasks completed
- All verification checks pass
- Human verification approved
- No TypeScript errors
- Phase 6 complete - Video & Polish milestone achieved
</success_criteria>
<output>
After completion, create `.planning/phases/06-video-and-polish/06-04-SUMMARY.md`
Additionally, update:
- .planning/STATE.md: Mark Phase 6 complete
- .planning/ROADMAP.md: Check off all Phase 6 plans
- .planning/ISSUES.md: Resolve ISS-001 as addressed in 06-03
</output>
Write Preview
Loading…
Cancel
Save