diff --git a/.planning/phases/06-video-and-polish/06-01-PLAN.md b/.planning/phases/06-video-and-polish/06-01-PLAN.md new file mode 100644 index 00000000..87ac3fe0 --- /dev/null +++ b/.planning/phases/06-video-and-polish/06-01-PLAN.md @@ -0,0 +1,137 @@ +--- +phase: 06-video-and-polish +plan: 01 +type: execute +--- + + +Create GenerateVideo node supporting video-capable models from Replicate and fal.ai. + +Purpose: Enable video generation workflows using the same multi-provider infrastructure built for images. +Output: Working GenerateVideoNode with provider/model selection and video output display. + + + +~/.claude/get-shit-done/workflows/execute-phase.md +~/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md + +# Prior phase context (established patterns): +@.planning/phases/03-generate-node-refactor/03-01-SUMMARY.md +@.planning/phases/03-generate-node-refactor/03-02-SUMMARY.md + +# Key source files: +@src/components/nodes/GenerateImageNode.tsx +@src/app/api/generate/route.ts +@src/types/index.ts +@src/lib/providers/types.ts +@src/store/workflowStore.ts + +**Tech stack available:** React, @xyflow/react, Zustand, Next.js API routes +**Established patterns:** +- Provider dropdown with conditional visibility based on API key +- Model selector fetching from /api/models with capabilities filter +- selectedModel object with {provider, modelId, displayName} +- Server-side provider dispatch in /api/generate +**Constraining decisions:** +- GenerateVideo as separate node type (not combined with GenerateImage) +- video capabilities: text-to-video, image-to-video + + + + + + Task 1: Create GenerateVideoNode component + + src/components/nodes/GenerateVideoNode.tsx, + src/components/nodes/index.ts, + src/types/index.ts, + src/store/workflowStore.ts, + src/components/WorkflowCanvas.tsx + + +Create GenerateVideoNode based on GenerateImageNode with these changes: +1. In types/index.ts: + - Add "generateVideo" to NodeType union + - Create GenerateVideoNodeData interface (copy NanoBananaNodeData, change outputImage to outputVideo: string | null, remove image-specific fields like aspectRatio, resolution, imageHistory) + - Add to WorkflowNodeData union +2. In GenerateVideoNode.tsx: + - Copy structure from GenerateImageNode + - Change VIDEO_CAPABILITIES to ["text-to-video", "image-to-video"] + - Fetch models with capabilities=text-to-video,image-to-video + - Remove Gemini provider option (Gemini doesn't do video) + - Show fal.ai always, Replicate when configured (like image node) + - Display video output using HTML5 video element with controls + - Remove aspect ratio/resolution selectors (video models have their own params) +3. In workflowStore.ts: + - Add createDefaultNodeData case for "generateVideo" + - Add defaultDimensions entry (same as nanoBanana: 200x280) + - Add to getConnectedInputs to extract video from GenerateVideo nodes (check for outputVideo) +4. In WorkflowCanvas.tsx: + - Add GenerateVideoNode to nodeTypes + - Add minimap color (purple: "#9333ea") +5. In nodes/index.ts: + - Export GenerateVideoNode + + npm run build passes, no TypeScript errors + GenerateVideoNode renders, shows provider/model dropdowns, can be added to canvas + + + + Task 2: Add video generation to API route + + src/app/api/generate/route.ts, + src/types/index.ts + + +Extend /api/generate to handle video generation: +1. Update GenerateResponse in types/index.ts: + - Add optional video?: string field alongside image + - Add optional contentType?: "image" | "video" field +2. In route.ts generateWithReplicate: + - After fetching output, detect if output is video (check content-type header for video/*) + - If video, return { type: "video", data: dataUrl } in outputs array + - Video data URL format: data:video/mp4;base64,... +3. In route.ts generateWithFal: + - fal.ai video models return { video: { url: "..." } } not images array + - Check for result.video?.url in addition to result.images + - If video URL found, fetch and convert to base64 with video mime type +4. In main POST handler: + - Return video field in response when output type is video + - Set contentType: "video" when returning video +5. Handle large video files: + - Videos can be large, log warning if >10MB but still return + - Consider returning URL directly for very large videos (>20MB) with a videoUrl field + + +Test with curl: Create a prediction for a video model and verify video data returns + + API returns video base64 or URL for video-capable models, contentType field indicates output type + + + + + +Before declaring plan complete: +- [ ] npm run build succeeds without errors +- [ ] GenerateVideoNode appears in node palette +- [ ] Video model dropdown populates from fal.ai/Replicate +- [ ] API route handles video outputs correctly + + + +- All tasks completed +- All verification checks pass +- No TypeScript errors +- GenerateVideoNode can be added to canvas and configured +- Video generation API path is ready (execution tested in 06-02 with playback) + + + +After completion, create `.planning/phases/06-video-and-polish/06-01-SUMMARY.md` + diff --git a/.planning/phases/06-video-and-polish/06-02-PLAN.md b/.planning/phases/06-video-and-polish/06-02-PLAN.md new file mode 100644 index 00000000..19553c51 --- /dev/null +++ b/.planning/phases/06-video-and-polish/06-02-PLAN.md @@ -0,0 +1,153 @@ +--- +phase: 06-video-and-polish +plan: 02 +type: execute +--- + + +Add video playback support to OutputNode and complete video generation workflow. + +Purpose: Enable end-to-end video generation with playback and download. +Output: OutputNode plays video content, download works for video files. + + + +~/.claude/get-shit-done/workflows/execute-phase.md +~/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md + +# Prior plan in this phase: +@.planning/phases/06-video-and-polish/06-01-SUMMARY.md + +# Key source files: +@src/components/nodes/OutputNode.tsx +@src/components/nodes/GenerateVideoNode.tsx +@src/types/index.ts +@src/store/workflowStore.ts + +**Established patterns:** +- OutputNode receives image via handle, displays in preview area +- Lightbox modal for full-size view +- Download button generates download link +**Constraining decisions:** +- Video displayed with HTML5 video element +- Support both base64 data URLs and HTTP URLs for video + + + + + + Task 1: Update OutputNode for video detection and playback + + src/components/nodes/OutputNode.tsx, + src/types/index.ts + + +Enhance OutputNode to handle both images and videos: +1. In types/index.ts, update OutputNodeData: + - Add video: string | null field + - Add contentType?: "image" | "video" field +2. In OutputNode.tsx: + - Detect content type from data URL prefix or contentType field + - isVideo check: data.video || data.contentType === "video" || (data.image?.startsWith("data:video/")) + - If video content: + - Replace img element with video element + - Add controls, loop, muted (autoplay with muted), playsinline attributes + - Style same as image: w-full h-full object-contain rounded + - If video in lightbox: + - Use video element in modal with same attributes + - Make it larger: max-h-[90vh] w-auto + - Update placeholder text: "Waiting for image or video" +3. Handle both base64 and URL sources: + - If value starts with "data:" use as src directly + - If value starts with "http" use as src directly + - Video element handles both formats natively + + npm run build passes, OutputNode component renders without errors + OutputNode displays video element when video content is connected, plays with controls + + + + Task 2: Update video download and workflow execution + + src/components/nodes/OutputNode.tsx, + src/store/workflowStore.ts + + +1. In OutputNode.tsx handleDownload: + - Detect if content is video (same check as display) + - For video: set download filename to generated-{timestamp}.mp4 + - For image: keep generated-{timestamp}.png + - Handle URL-based video: fetch URL, convert to blob, create blob URL for download + +2. In workflowStore.ts getConnectedInputs: + - Add case for "generateVideo" node type + - Extract outputVideo field and add to images array (for img2vid workflows) + - Videos can be input to downstream nodes that accept video/image + +3. In workflowStore.ts executeWorkflow: + - Add execution case for "generateVideo" node type + - Copy pattern from nanoBanana execution + - Call /api/generate with selectedModel containing video model + - Store result in outputVideo field (not outputImage) + - Update status appropriately + +4. Update output node connection: + - In getConnectedInputs, when source is generateVideo, get outputVideo + - Pass to output node which stores in image field (reuse existing flow) + - The OutputNode detects video content via data URL prefix + + +Create GenerateVideo -> Output workflow, run generation, verify video plays in output + + Video downloads with .mp4 extension, workflow execution produces video in output node + + + + Video generation workflow: GenerateVideo node -> Output node with playback + + 1. Run: npm run dev + 2. Add a GenerateVideo node to the canvas + 3. Select fal.ai provider, pick a text-to-video model (e.g., any model with "video" in name) + 4. Add a Prompt node, connect to GenerateVideo text input + 5. Add an Output node, connect GenerateVideo output to it + 6. Enter a prompt like "a cat walking in slow motion" + 7. Run the workflow (Cmd+Enter) + 8. Verify: + - GenerateVideo node shows loading state + - After completion, video appears in GenerateVideo preview + - Output node shows video with playback controls + - Video plays when clicking play button + - Download button saves .mp4 file + - Lightbox shows larger video player + + Type "approved" to continue to next plan, or describe issues to fix + + + + + +Before declaring plan complete: +- [ ] npm run build succeeds without errors +- [ ] OutputNode detects and displays video content +- [ ] Video playback works with controls +- [ ] Video download saves .mp4 file +- [ ] End-to-end workflow: Prompt -> GenerateVideo -> Output works + + + +- All tasks completed +- All verification checks pass +- Human verification approved +- No TypeScript errors +- Complete video generation pipeline working + + + +After completion, create `.planning/phases/06-video-and-polish/06-02-SUMMARY.md` + diff --git a/.planning/phases/06-video-and-polish/06-03-PLAN.md b/.planning/phases/06-video-and-polish/06-03-PLAN.md new file mode 100644 index 00000000..318db1af --- /dev/null +++ b/.planning/phases/06-video-and-polish/06-03-PLAN.md @@ -0,0 +1,178 @@ +--- +phase: 06-video-and-polish +plan: 03 +type: execute +--- + + +Add dynamic model parameters fetched from provider schemas. + +Purpose: Allow users to customize model-specific parameters (like seed, steps, guidance_scale) based on what each model supports. Addresses ISS-001. +Output: Generate nodes show dynamic parameter inputs based on selected model's schema. + + + +~/.claude/get-shit-done/workflows/execute-phase.md +~/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/ISSUES.md + +# Prior plans in this phase: +@.planning/phases/06-video-and-polish/06-01-SUMMARY.md +@.planning/phases/06-video-and-polish/06-02-SUMMARY.md + +# Key source files: +@src/app/api/models/route.ts +@src/components/nodes/GenerateImageNode.tsx +@src/components/nodes/GenerateVideoNode.tsx +@src/types/index.ts + +**Issue being addressed:** ISS-001 - Generate node should adapt to model requirements +**Established patterns:** +- Models have openapi_schema field from provider APIs (already fetched but not exposed) +- parameters?: Record passed to generation API +**Constraining decisions:** +- Use provider schema, don't hardcode parameters +- Common parameters to expose: seed, num_inference_steps, guidance_scale, negative_prompt + + + + + + Task 1: Add schema endpoint and model schema fetching + + src/app/api/models/[modelId]/route.ts, + src/lib/providers/types.ts, + src/app/api/models/route.ts + + +Create endpoint to fetch individual model schema: +1. Create src/app/api/models/[modelId]/route.ts: + - GET /api/models/:modelId (modelId is URL-encoded, e.g., "fal-ai%2Fflux%2Fdev") + - Parse provider from query param: ?provider=fal + - For Replicate: + - Fetch model from https://api.replicate.com/v1/models/{owner}/{name} + - Extract latest_version.openapi_schema.components.schemas.Input + - Return simplified parameter schema + - For fal.ai: + - Fetch from https://fal.run/{modelId}/api (returns OpenAPI spec) + - Extract requestBody schema properties + - Return simplified parameter schema +2. In providers/types.ts, add ModelParameter interface: + ```typescript + interface ModelParameter { + name: string; + type: "string" | "number" | "integer" | "boolean" | "array"; + description?: string; + default?: unknown; + minimum?: number; + maximum?: number; + enum?: unknown[]; + required?: boolean; + } + ``` +3. Schema simplification: + - Filter to user-relevant params: exclude internal params like "webhook", "sync_mode" + - Include: prompt (if not already handled), seed, num_inference_steps, guidance_scale, negative_prompt, width, height, num_outputs + - Return as ModelParameter[] array +4. Add caching for schema (same pattern as model list, 10 min TTL) + + +curl "http://localhost:3000/api/models/fal-ai%2Fflux%2Fdev?provider=fal" returns parameter schema + + API endpoint returns model parameter schema for both Replicate and fal.ai models + + + + Task 2: Add parameter UI component + + src/components/nodes/ModelParameters.tsx, + src/components/nodes/GenerateImageNode.tsx, + src/components/nodes/GenerateVideoNode.tsx + + +Create reusable parameter input component: +1. Create src/components/nodes/ModelParameters.tsx: + - Props: modelId, provider, parameters, onParametersChange + - Fetch schema from /api/models/{modelId}?provider={provider} when modelId changes + - Render appropriate input for each parameter type: + - string: text input (or select if enum) + - number/integer: number input with min/max + - boolean: checkbox + - array: skip for now (complex) + - Show parameter name, description on hover + - Compact styling to fit in node (10px font, tight spacing) + - Collapsible section: "Parameters" header, collapsed by default + - Only show if model is selected (not Gemini) +2. Integrate into GenerateImageNode: + - Import ModelParameters component + - Add below model selector when external provider selected + - Store parameters in nodeData.parameters + - Pass to API call +3. Integrate into GenerateVideoNode: + - Same integration as GenerateImageNode +4. Update node data types: + - Add parameters?: Record to NanoBananaNodeData + - Already exists in GenerateVideoNodeData if copied from 06-01 + + npm run build passes, parameter inputs appear when selecting external model + Parameter inputs render based on model schema, values stored in node data + + + + Task 3: Pass parameters through generation pipeline + + src/store/workflowStore.ts, + src/app/api/generate/route.ts + + +Ensure parameters flow through to providers: +1. In workflowStore.ts executeWorkflow: + - For nanoBanana node: include nodeData.parameters in API request body + - For generateVideo node: same inclusion +2. In API route.ts: + - Parameters already spread into predictionInput/requestBody via ...input.parameters + - Verify this is working correctly + - Add logging: log which parameters are being passed +3. Validate parameter types before sending: + - Coerce strings to numbers where schema expects number + - Filter out empty/undefined values + - Don't send default values (let model use its defaults) +4. Handle parameter errors: + - If provider rejects parameter, surface error clearly + - Show which parameter was invalid in error message + + +Test generation with custom parameters (e.g., seed=12345), verify parameter appears in API logs + + Custom parameters successfully passed to providers and affect generation output + + + + + +Before declaring plan complete: +- [ ] npm run build succeeds without errors +- [ ] Schema endpoint returns parameters for Replicate and fal.ai models +- [ ] Parameter inputs appear in Generate nodes for external models +- [ ] Parameter values persist in node data +- [ ] Parameters are sent to providers during generation +- [ ] Custom seed produces reproducible results (if model supports it) + + + +- All tasks completed +- All verification checks pass +- No TypeScript errors +- ISS-001 addressed: nodes adapt inputs to model requirements +- Users can customize model-specific parameters + + + +After completion, create `.planning/phases/06-video-and-polish/06-03-SUMMARY.md` + diff --git a/.planning/phases/06-video-and-polish/06-04-PLAN.md b/.planning/phases/06-video-and-polish/06-04-PLAN.md new file mode 100644 index 00000000..089ee955 --- /dev/null +++ b/.planning/phases/06-video-and-polish/06-04-PLAN.md @@ -0,0 +1,206 @@ +--- +phase: 06-video-and-polish +plan: 04 +type: execute +--- + + +Handle edge cases, improve error states, and complete final polish for multi-provider support. + +Purpose: Ensure robust, production-ready multi-provider experience with clear error messages and graceful degradation. +Output: Polished multi-provider integration ready for users. + + + +~/.claude/get-shit-done/workflows/execute-phase.md +~/.claude/get-shit-done/templates/summary.md +~/.claude/get-shit-done/references/checkpoints.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/ISSUES.md + +# Prior plans in this phase: +@.planning/phases/06-video-and-polish/06-01-SUMMARY.md +@.planning/phases/06-video-and-polish/06-02-SUMMARY.md +@.planning/phases/06-video-and-polish/06-03-SUMMARY.md + +# Key source files: +@src/components/nodes/GenerateImageNode.tsx +@src/components/nodes/GenerateVideoNode.tsx +@src/components/nodes/OutputNode.tsx +@src/app/api/generate/route.ts +@src/store/workflowStore.ts + +**Issues to address:** +- UAT-001: Provider icons should use real logos (cosmetic, if time permits) +**Edge cases to handle:** +- Model list fails to load +- Generation times out (video generation can be slow) +- Large video files +- Invalid parameters +- Provider rate limits + + + + + + Task 1: Improve error handling and loading states + + src/components/nodes/GenerateImageNode.tsx, + src/components/nodes/GenerateVideoNode.tsx, + src/components/nodes/ModelParameters.tsx, + src/app/api/generate/route.ts + + +Enhance error handling across generation nodes: +1. In GenerateImageNode/GenerateVideoNode: + - Show clear error when model list fails: "Failed to load models. Check API key." + - Add retry button for model list fetch + - Show loading spinner when fetching models + - Display model fetch error in a toast or inline message +2. In ModelParameters: + - Handle schema fetch failure gracefully: "Parameters unavailable" + - Show loading state while fetching schema + - Cache schema in component state to avoid re-fetching +3. In API route.ts: + - Improve error messages: include model name in errors + - Handle timeout explicitly: "Generation timed out after 5 minutes. Video models may take longer - try again." + - Detect rate limit errors (429) and provide helpful message + - For Replicate: if prediction fails, include failure reason from API + - For fal.ai: parse error response body for details +4. Video-specific handling: + - Warn if video file is very large (>50MB): "Video generated but file is large ({size}MB)" + - If video fetch fails, try to return URL directly as fallback + + Simulate errors (bad API key, timeout) and verify clear error messages appear + Clear error messages for all failure modes, retry capability, loading states visible + + + + Task 2: Parameter validation and defaults + + src/components/nodes/ModelParameters.tsx, + src/store/workflowStore.ts + + +Add input validation for model parameters: +1. In ModelParameters: + - Validate number inputs against min/max from schema + - Show validation error inline: red border, error text below + - Prevent invalid values from being saved + - For enum parameters: use select dropdown with options + - Set default values from schema when parameter is first shown + - Reset parameters when model changes +2. In workflowStore executeWorkflow: + - Filter out parameters with invalid values before API call + - Log warning if parameters were filtered +3. Common parameter handling: + - seed: accept integers only, positive + - num_inference_steps: typically 1-100, integer + - guidance_scale: typically 0-20, float + - width/height: typically 256-2048, multiples of 8 + - negative_prompt: string, no validation +4. Clear parameters when switching providers: + - Parameters from one provider likely don't apply to another + - Reset parameters state when provider changes + + Enter invalid parameter values, verify validation errors appear and prevent submission + Parameters validate against schema, invalid values prevented, defaults applied + + + + Task 3: Final cleanup and testing + + src/store/workflowStore.ts, + src/components/FloatingActionBar.tsx, + src/components/modals/ModelSearchDialog.tsx + + +Final polish and integration testing: +1. Update MODEL.md search dialog: + - Add video capability badges display + - Ensure video models appear when searching + - Filter by capability when opened from GenerateVideo icon (if applicable) +2. Update FloatingActionBar: + - Consider adding video icon for quick GenerateVideo node creation + - Or reuse existing provider icons (clicking them for video) +3. Test complete workflows: + - Text -> GenerateImage -> Output (existing, verify still works) + - Text -> GenerateVideo -> Output (new) + - Image -> GenerateImage (img2img) -> Output + - Image -> GenerateVideo (img2vid) -> Output +4. Verify backward compatibility: + - Load existing workflow with old nanoBanana nodes + - Confirm they still execute correctly +5. Clean up any TODO comments or temporary code from phase development +6. Update types to mark deprecated aliases: + - Add @deprecated JSDoc to NanoBananaNode alias + + +All four workflow patterns execute successfully: +- Text to image with Replicate +- Text to video with fal.ai +- Image to image with fal.ai +- Image to video with Replicate (if model available) + + All workflows functional, backward compatible, code cleaned up + + + + Complete multi-provider video generation with custom parameters + + 1. Run: npm run dev + 2. Test text-to-video workflow: + - Prompt node -> GenerateVideo (fal.ai, pick video model) -> Output + - Enter a prompt, run workflow + - Verify video plays in output node + 3. Test custom parameters: + - Select external model in GenerateImage + - Expand "Parameters" section + - Modify a value (e.g., seed=42) + - Run generation, verify parameter affects output + 4. Test error handling: + - Remove Replicate API key, try to select Replicate model + - Verify clear error message + 5. Test backward compatibility: + - Load an existing workflow with NanoBanana nodes + - Run it, verify Gemini generation still works + 6. Test video download: + - Generate a video, click download in output node + - Verify .mp4 file downloads correctly + + Type "approved" to complete Phase 6, or describe issues + + + + + +Before declaring plan complete: +- [ ] npm run build succeeds without errors +- [ ] All error states have clear messages +- [ ] Parameter validation works correctly +- [ ] All four workflow patterns tested +- [ ] Backward compatibility verified +- [ ] Human verification approved + + + +- All tasks completed +- All verification checks pass +- Human verification approved +- No TypeScript errors +- Phase 6 complete - Video & Polish milestone achieved + + + +After completion, create `.planning/phases/06-video-and-polish/06-04-SUMMARY.md` + +Additionally, update: +- .planning/STATE.md: Mark Phase 6 complete +- .planning/ROADMAP.md: Check off all Phase 6 plans +- .planning/ISSUES.md: Resolve ISS-001 as addressed in 06-03 +