diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md new file mode 100644 index 00000000..4d037f3a --- /dev/null +++ b/.planning/ROADMAP.md @@ -0,0 +1,660 @@ +# Roadmap: Node Banana Multi-Provider Support + +## Overview + +Transform Node Banana from a Gemini-only image generator into a multi-provider platform supporting Replicate and fal.ai. The journey builds provider infrastructure first, then adds dynamic model discovery, refactors the generate node for flexibility, creates a searchable model browser, adds local image serving for URL-based providers, and finishes with video support and polish. + +## Domain Expertise + +None + +## Milestones + +- ✅ **v1.0 Multi-Provider Support** - Phases 1-6 (shipped 2026-01-11) +- ✅ **v1.1 Improvements** - Phases 7-14 (shipped 2026-01-12) +- ✅ **v1.2 Improvements** - Phases 15-24 (shipped 2026-01-17) +- ✅ **v1.3 Improvements** - Phases 25-30 (shipped 2026-01-24) +- 🚧 **v1.4 Features** - Phases 31-35 (in progress) +- 📋 **v1.5 Store Refactoring** - Phases 36-39 (planned) + +## Phases + +**Phase Numbering:** +- Integer phases (1, 2, 3): Planned milestone work +- Decimal phases (2.1, 2.2): Urgent insertions (marked with INSERTED) + +
+✅ v1.0 Multi-Provider Support (Phases 1-6) - SHIPPED 2026-01-11 + +### Phase 1: Provider Infrastructure +**Goal**: Users can configure Replicate and fal.ai API keys in settings, with keys securely stored +**Depends on**: Nothing (first phase) +**Research**: Unlikely (internal UI patterns, Zustand state management) +**Plans**: 2 plans + +Plans: +- [x] 01-01: Provider settings UI in ProjectSetupModal +- [x] 01-02: Provider abstraction layer and types + +### Phase 2: Model Discovery +**Goal**: App can fetch and cache available models from enabled providers at runtime +**Depends on**: Phase 1 +**Research**: Likely (external APIs) +**Research topics**: Replicate model listing API endpoints, fal.ai model discovery endpoints, response schemas and pagination +**Plans**: 3 plans + +Plans: +- [x] 02-01: Replicate model fetching API route +- [x] 02-02: fal.ai model fetching API route +- [x] 02-03: Model caching and unified model interface + +### Phase 3: Generate Node Refactor +**Goal**: NanoBanana node becomes GenerateImage node supporting any provider's image models +**Depends on**: Phase 2 +**Research**: Unlikely (internal refactoring using existing patterns) +**Design decision**: Separate nodes for image vs video generation (GenerateVideo added in Phase 6) +**Plans**: 3 plans + +Plans: +- [x] 03-01: Rename NanoBanana to GenerateImage, add model selector (image models only) +- [x] 03-02: Provider-specific execution in generate API route +- [x] 03-03: Backward compatibility for existing workflows + +### Phase 4: Model Search Dialog +**Goal**: Users can browse models via floating action bar icons and searchable dialog +**Depends on**: Phase 3 +**Research**: Unlikely (internal UI using React patterns) +**Plans**: 2 plans + +Plans: +- [x] 04-01: Floating action bar with provider icons +- [x] 04-02: Model search dialog with filtering and selection + +### Phase 5: Image URL Server +**Goal**: Local API endpoint serves workflow images as URLs for providers requiring URL inputs +**Depends on**: Phase 3 +**Research**: Unlikely (Next.js API routes, existing patterns) +**Plans**: 2 plans + +Plans: +- [x] 05-01: Image serving endpoint and URL generation +- [x] 05-02: Integration with generate node for URL-based providers + +### Phase 6: Video & Polish +**Goal**: GenerateVideo node for video generation, video playback, custom parameters, edge cases +**Depends on**: Phase 5 +**Research**: Likely (video handling) +**Research topics**: HTML5 video element for base64/blob URLs, provider response formats for video content +**Design decision**: GenerateVideo as separate node type (not combined with GenerateImage) +**Plans**: 4 plans + +Plans: +- [x] 06-01: GenerateVideo node with video-capable model selector +- [x] 06-02: Video playback in output node +- [x] 06-03: Custom model parameters from provider schemas +- [x] 06-04: Edge case handling and final polish + +
+ +
+✅ v1.1 Improvements (Phases 7-14) - SHIPPED 2026-01-12 + +**Milestone Goal:** Fix connection issues, improve error visibility, add video history, auto-size nodes, and polish UI + +#### Phase 7: Video Connections ✅ + +**Goal**: Fix video handle connections to only allow valid targets (generateVideo, output) +**Depends on**: Phase 6 +**Research**: Unlikely (internal connection validation patterns) +**Plans**: 1 plan + +Plans: +- [x] 07-01: Add video handle type and connection validation + +#### Phase 8: Error Display ✅ + +**Goal**: Better error visibility with overlay display, not hidden by previous output +**Depends on**: Phase 7 +**Research**: Unlikely (internal UI patterns) +**Plans**: 1 plan + +Plans: +- [x] 08-01: Error notifications with persistent toast and node overlays + +#### Phase 9: Video History ✅ + +**Goal**: Add history carousel for generated videos matching image history pattern +**Depends on**: Phase 8 +**Research**: Unlikely (existing image history pattern) +**Plans**: 1 plan + +Plans: +- [x] 09-01: Video history types, store support, load API, and carousel UI + +#### Phase 10: Node Autosizing ✅ + +**Goal**: Auto-size nodes to output dimensions and aspect ratio +**Depends on**: Phase 9 +**Research**: Unlikely (React Flow node sizing) +**Plans**: 1 plan + +Plans: +- [x] 10-01: Node dimension utility and auto-resize on output + +#### Phase 11: UI Polish ✅ + +**Goal**: Flora UI alignment, header UI improvements, project settings, provider logos on nodes +**Depends on**: Phase 10 +**Research**: Unlikely (internal UI work) +**Plans**: 1 plan + +Plans: +- [x] 11-01: Provider badges on nodes and header UI streamlining + +#### Phase 12: Model Improvements ✅ + +**Goal**: Verify Replicate image models work, extend model cache TTL +**Depends on**: Phase 11 +**Research**: Likely (Replicate API behavior verification) +**Research topics**: Replicate image model endpoints, cache invalidation strategies +**Plans**: 1 plan + +Plans: +- [x] 12-01: Extended cache TTL, fixed isImageInput, fixed stale node data in execution + +#### Phase 13: Fix Duplicate Generations ✅ + +**Goal**: Fix image deduplication - generations folder has duplicate images due to hashing failure +**Depends on**: Phase 12 +**Research**: Likely (investigate current hashing implementation) +**Research topics**: Current save-generation hashing logic, why duplicates are being created +**Plans**: 1 plan + +Plans: +- [x] 13-01: Add MD5 content hashing and deduplication to save-generation API + +#### Phase 14: Fix Drag-Connect Node Creation Bugs ✅ + +**Goal**: Fix bugs with nodes created via drag-connect: (1) connection vanishes after selecting model from browser, (2) defaults to Gemini with missing model selector in header +**Depends on**: Phase 13 +**Research**: Unlikely (React Flow connection state, node creation flow) +**Research topics**: Connection state during node creation, model selection callbacks, createDefaultNodeData initialization +**Plans**: 1 plan + +Plans: +- [x] 14-01: Normalize dynamic handle IDs and fix connection persistence + +
+ +### 🚧 v1.2 Improvements (In Progress) + +**Milestone Goal:** Add automated testing across the application, modularize large monolithic files for better maintainability, and improve cost tracking for multi-provider support + +#### Phase 15: Test Infrastructure + +**Goal**: Set up testing framework with Vitest and React Testing Library for Next.js 16 +**Depends on**: Phase 14 +**Research**: Likely (Vitest + Next.js 16 App Router setup) +**Research topics**: Vitest configuration for Next.js 16, React Testing Library setup, test file organization +**Plans**: TBD + +Plans: +- [x] 15-01: React Testing Library setup and configuration + +#### Phase 16: Store Modularization ✅ + +**Goal**: Break up workflowStore.ts into focused modules (execution, nodes, edges, persistence) +**Depends on**: Phase 15 +**Research**: Unlikely (internal Zustand patterns) +**Plans**: 1 plan + +Plans: +- [x] 16-01: Extract localStorage helpers and node defaults to utility modules + +#### Phase 17: Component Tests + +**Goal**: Add tests for all 34 React components covering nodes, toolbars, modals, and edges +**Depends on**: Phase 16 +**Research**: Unlikely (established React Testing Library patterns) +**Plans**: 11 plans + +Plans: +- [x] 17-01: Core Nodes (BaseNode, PromptNode, ImageInputNode) +- [x] 17-02: Display Nodes (OutputNode, SplitGridNode, GroupNode) +- [x] 17-03: Generate Nodes (GenerateImageNode, GenerateVideoNode) +- [x] 17-04: Processing Nodes (LLMGenerateNode, AnnotationNode) +- [x] 17-05: Toolbars (Header, FloatingActionBar, MultiSelectToolbar) +- [x] 17-06: Canvas & Edges (WorkflowCanvas, EditableEdge, ReferenceEdge, EdgeToolbar) +- [x] 17-07: Menus & Notifications (ConnectionDropMenu, Toast, CostIndicator) +- [x] 17-08: Core Modals (ModelSearchDialog, ProjectSetupModal, CostDialog) +- [x] 17-09: Editor Modals (PromptEditorModal, SplitGridSettingsModal, AnnotationModal) +- [x] 17-10: Quickstart (WelcomeModal, QuickstartInitialView, QuickstartTemplatesView, PromptWorkflowView, QuickstartBackButton) +- [x] 17-11: Utilities (GlobalImageHistory, GroupsOverlay, ModelParameters) + +#### Phase 18: API Route Tests + +**Goal**: Add tests for generate, llm, models, and workflow API routes +**Depends on**: Phase 17 +**Research**: Unlikely (Next.js API route testing patterns) +**Plans**: 5 plans + +Plans: +- [x] 18-01: File I/O routes (workflow, save-generation) +- [x] 18-02: LLM route (Google, OpenAI providers) +- [x] 18-03: Generate route (Gemini provider) +- [x] 18-04: Generate route (Replicate, fal.ai providers) +- [x] 18-05: Models route (caching, aggregation) + +#### Phase 19: Type Refactoring + +**Goal**: Split types/index.ts into domain-specific type files (nodes, providers, workflow) +**Depends on**: Phase 18 +**Research**: Unlikely (internal refactoring) +**Plans**: 2 plans + +Plans: +- [x] 19-01: Extract node and annotation types +- [x] 19-02: Extract provider, workflow, API, and model types + +#### Phase 20: Integration Tests + +**Goal**: End-to-end workflow execution tests covering node connections and data flow +**Depends on**: Phase 19 +**Research**: Unlikely (internal testing patterns) +**Plans**: 2 plans + +Plans: +- [x] 20-01: Store integration tests (getConnectedInputs, validateWorkflow, topological sort) +- [x] 20-02: Workflow execution tests (data flow, error handling, connection validation) + +#### Phase 21: Fix Image Input & Deduplication Issues + +**Goal**: Fix Gemini nano-banana-pro model ignoring image inputs and resolve duplicate image saving across input/generated images +**Depends on**: Phase 20 +**Research**: Likely (Gemini API image input handling, current hashing implementation) +**Research topics**: Gemini 3 Pro image generation API requirements, why image inputs are ignored, current save logic for inputs vs generations +**Plans**: 1 plan + +**Issues:** +1. ~~nano-banana-pro model generates without considering image inputs~~ - RESOLVED (no longer an issue) +2. Input and generated images have duplicate files despite different hashes - need consistent hashing approach matching video saving +3. Generated images should be prepended with prompt details like generated videos + +Plans: +- [x] 21-01: Unify MD5 hashing for image deduplication + +#### Phase 22: Generate Node Dynamic Input Tests + +**Goal**: Test that generate nodes properly validate and render dynamic inputs from provider schemas, and that all inputs/parameters are correctly included in API calls +**Depends on**: Phase 21 +**Research**: Unlikely (existing test patterns from Phase 17) +**Plans**: 1 plan + +**Test coverage:** +1. Dynamic inputs from provider schemas render correctly as parameters and input handles on node creation +2. Validation of dynamic inputs (required fields, type checking, constraints) +3. Standard inputs (image, text) validate properly +4. On submission, all parameters and inputs are included in API call payload correctly (covered by Phase 18) +5. Coverage across all providers (Gemini, Replicate, fal.ai) + +Plans: +- [x] 22-01: ModelParameters tests, GenerateImageNode/GenerateVideoNode dynamic handle tests + +#### Phase 23: Model Browser Improvements + +**Goal**: Improve model browser UX with recently used models section, icon-based provider dropdown with Gemini, and include Gemini models in browse list +**Depends on**: Phase 22 +**Research**: Unlikely (existing UI patterns) +**Plans**: TBD + +**Features:** +1. Recently used models section at top of browse dialog showing last 4 models used +2. Provider dropdown uses icons instead of text, add Gemini to the provider list +3. Include Gemini models (nano-banana, nano-banana-pro) in the browsable model list alongside Replicate/fal.ai models + +Plans: +- [x] 23-01: Recently used models, icon-based provider filter, Gemini models in browse + +#### Phase 24: Improved Cost Summary + +**Goal**: Expand cost tracking to include fal.ai models (via their pricing API), video generation nodes, and graceful handling for Replicate (no pricing API available) +**Depends on**: Phase 23 +**Research**: Complete (see research notes below) +**Plans**: 3 plans + +**Research Notes:** +- fal.ai has a pricing API: `GET /v1/models/pricing?endpoint_id=model1,model2` returns `{ prices: [{ endpoint_id, unit_price, unit, currency }] }` +- Replicate does NOT expose pricing via API (open GitHub issue, unresolved). Prediction response has `metrics.predict_time` but no hardware/cost info. +- `ProviderModel` type already has `pricing?: { type, amount, currency }` field at `src/lib/providers/types.ts:69-74` + +**UX Decision: Provider-Grouped with Uncertainty Section** +- Two sections: "Known Costs" (Gemini + fal.ai) and "Pricing Unavailable" (Replicate) +- Known costs show total at section header, breakdown by provider with icons +- fal.ai items show billing unit from API (e.g., "per image", "per 5s video") +- Replicate section includes help text: "Pricing varies by hardware and runtime. Check replicate.com" +- Incurred cost only tracks Gemini & fal.ai (Replicate excluded with note) +- Empty provider sections hidden + +**Features:** +1. Fetch fal.ai model pricing via their pricing API and populate `ProviderModel.pricing` +2. Track generateVideo nodes in cost predictions (currently ignored) +3. For Replicate models: display "pricing unavailable" gracefully in dedicated section +4. Update CostDialog with two-section layout: Known Costs (Gemini, fal.ai) and Pricing Unavailable (Replicate) +5. Show billing units for fal.ai models (per image, per second of video, etc.) +6. Incurred cost excludes Replicate with explanatory note + +Plans: +- [x] 24-01: fal.ai pricing API integration and ProviderModel.pricing population +- [x] 24-02: Expand costCalculator to handle video nodes and external providers +- [x] 24-03: Update CostDialog UI for multi-provider breakdown + +
+✅ v1.3 Improvements (Phases 25-30) - SHIPPED 2026-01-24 + +**Milestone Goal:** Add rich template exploration with search/filters, node default preferences in settings, and canvas performance optimizations + +#### Phase 25: Template Explorer UI + +**Goal**: Full template exploration interface with cards layout, left-hand filter/search panel, supporting many templates +**Depends on**: Phase 24 +**Research**: Unlikely (internal UI patterns) +**Plans**: 2 plans + +**Features:** +- Cards layout for template browsing (scalable for many templates) +- Left-hand panel with search and category filters +- Template metadata display (name, description, node count, etc.) +- Replace current quickstart template selection + +Plans: +- [x] 25-01: Template types, TemplateCard component, TemplateExplorerView grid layout +- [x] 25-02: Sidebar filters (search, category, tags), WelcomeModal integration + +#### Phase 26: Template Preview Rendering ✅ + +**Goal**: Visual preview of workflow templates showing node layout and connections before loading +**Depends on**: Phase 25 +**Research**: Unlikely (React Flow rendering, existing patterns) +**Plans**: 1 plan + +**Final Implementation:** +- Horizontal card layout with thumbnail image, details, and "Use workflow" button +- Two-column grid layout for better scanning +- Conditional dialog width (6xl for explorer, 2xl for other views) +- Direct workflow loading without intermediate modal + +Plans: +- [x] 26-01: Horizontal cards with direct workflow loading, conditional dialog sizing + +#### Phase 27: Node Defaults Infrastructure ✅ + +**Goal**: Store default model and settings preferences per node type in localStorage +**Depends on**: Phase 26 +**Research**: Unlikely (localStorage, existing patterns) +**Plans**: 1 plan + +**Features:** +- Schema for node type defaults (model, parameters, settings) +- Apply defaults when creating nodes via keyboard shortcuts (Shift+G, etc.) +- Per-provider model defaults (e.g., default fal.ai model for GenerateImage) + +Plans: +- [x] 27-01: NodeDefaultsConfig types, localStorage helpers, createDefaultNodeData integration + +#### Phase 28: Node Defaults UI ✅ + +**Goal**: Settings panel for configuring default node preferences per node type +**Depends on**: Phase 27 +**Research**: Unlikely (internal UI patterns) +**Plans**: 1 plan + +**Features:** +- Section in settings for node defaults +- Select default model per node type (GenerateImage, GenerateVideo, LLM) +- Configure default parameters (e.g., seedream 4.5 with specific settings) +- Clear/reset to system defaults +- Shift+V keyboard shortcut for video nodes + +Plans: +- [x] 28-01: Node Defaults tab in ProjectSetupModal with model selection and LLM controls + +#### Phase 29: Canvas Performance + +**Goal**: Optimize canvas rendering for large workflows using virtualization and memoization +**Depends on**: Phase 28 +**Research**: Likely (React Flow virtualization, performance optimization) +**Research topics**: React Flow performance best practices, node virtualization, selective rendering +**Plans**: TBD + +Plans: +- [ ] 29-01: TBD (run /gsd:plan-phase 29 to break down) + +#### Phase 30: Small Fixes + +**Goal**: [To be planned] +**Depends on**: Phase 29 +**Research**: Unlikely +**Plans**: TBD + +Plans: +- [ ] 30-01: TBD (run /gsd:plan-phase 30 to break down) + +
+ +### 🚧 v1.4 Features (In Progress) + +**Milestone Goal:** Transform the Prompt-to-Workflow feature into a full agentic workflow builder with proposal dialogs, chat-based editing, and safe file handling for large workflows + +#### Phase 31: Workflow Proposal System + +**Goal**: Agent proposes workflow details before building with user feedback/approve flow +**Depends on**: Phase 30 +**Research**: Likely (LLM prompting for structured proposals) +**Research topics**: Structured output formats for workflow proposals, dialog UX patterns for agent communication +**Plans**: TBD + +Plans: +- [x] 31-01: Workflow proposal types, prompt, and API endpoint +- [x] 31-02: Chat infrastructure (Vercel AI SDK, streaming endpoint, types) + +#### Phase 32: Chat UI Foundation + +**Goal**: Floating chat window above minimap using Vercel AI SDK patterns +**Depends on**: Phase 31 +**Research**: Likely (Vercel AI SDK, chat UI components) +**Research topics**: Vercel AI SDK (ai package) integration, useChat hook, streaming responses, chat UI component patterns +**Plans**: TBD + +Plans: +- [ ] 32-01: TBD (run /gsd:plan-phase 32 to break down) + +#### Phase 33: Workflow Edit Safety + +**Goal**: Original file preservation before edits with accept/reject change flow +**Depends on**: Phase 32 +**Research**: Likely (versioning strategies, JSON diffing) +**Research topics**: Shadow copy vs diff-based vs undo stack approaches, JSON diff visualization +**Plans**: TBD + +Plans: +- [ ] 33-01: TBD (run /gsd:plan-phase 33 to break down) + +#### Phase 34: Agentic Workflow Editing + +**Goal**: Agent interprets edit requests and modifies workflow JSON with change narration +**Depends on**: Phase 33 +**Research**: Likely (system prompts for structured edits) +**Research topics**: System prompts for JSON file updates, structured output for workflow modifications, change explanation patterns +**Plans**: TBD + +Plans: +- [ ] 34-01: TBD (run /gsd:plan-phase 34 to break down) + +#### Phase 35: Large Workflow Handling + +**Goal**: Handle workflows with base64 images/videos efficiently, token management +**Depends on**: Phase 34 +**Research**: Likely (chunking, context window strategies) +**Research topics**: Token optimization for large payloads, base64 extraction/reinsertion, workflow summarization techniques +**Plans**: TBD + +Plans: +- [ ] 35-01: TBD (run /gsd:plan-phase 35 to break down) + +### 📋 v1.5 Store Refactoring (Planned) + +**Milestone Goal:** Major refactoring of workflowStore.ts (2,900+ lines) into modular, testable components. Extract execution engine, create Zustand slices, and improve code maintainability while maintaining full backward compatibility. + +**Problem Statement:** +- `workflowStore.ts` is 2,907 lines with 15+ mixed concerns +- `executeWorkflow` (~800 lines) and `regenerateNode` (~600 lines) contain node-specific logic for 8 node types inline +- Duplicated patterns (provider header building repeated 6+ times) +- Difficult to test individual concerns +- High cognitive load for any modifications + +**Target Architecture:** +``` +src/store/ +├── workflowStore.ts # ~100 lines - combines slices +├── slices/ # Zustand slice pattern +│ ├── coreSlice.ts # nodes, edges, clipboard +│ ├── groupsSlice.ts # group operations +│ ├── executionSlice.ts # execution state & actions +│ ├── persistenceSlice.ts # save/load, auto-save +│ ├── uiSlice.ts # modal state, quickstart +│ ├── costSlice.ts # cost tracking +│ ├── providersSlice.ts # provider settings +│ ├── historySlice.ts # global history, recent models +│ └── commentsSlice.ts # comment navigation +├── execution/ # Modular execution engine +│ ├── engine.ts # Core orchestration +│ ├── helpers.ts # Shared utilities +│ └── executors/ # Per-node-type executors +└── utils/ # Pure helper functions +``` + +#### Phase 36: Execution Engine Extraction + +**Goal**: Extract the massive executeWorkflow and regenerateNode functions into a modular execution engine with per-node-type executors +**Depends on**: Phase 35 +**Research**: Unlikely (internal Zustand patterns, existing codebase) +**Plans**: 3 plans + +**Scope:** +- Create `src/store/execution/` directory structure +- Extract execution types and interfaces +- Create node executor interface and registry +- Extract all 8 node-type executors (imageInput, annotation, prompt, nanoBanana, generateVideo, llmGenerate, splitGrid, output) +- Extract shared helpers (buildProviderHeaders, trackSaveGeneration, waitForPendingImageSyncs) +- Integrate engine back into workflowStore with same public API + +Plans: +- [ ] 36-01: Execution types, helpers, and engine orchestration +- [ ] 36-02: Node executors (all 8 types) with tests +- [ ] 36-03: Integrate engine with store, backward compatibility verification + +#### Phase 37: Pure Helpers Extraction + +**Goal**: Extract pure functions from the store into testable utility modules +**Depends on**: Phase 36 +**Research**: Unlikely (internal refactoring) +**Plans**: 2 plans + +**Scope:** +- Extract `getConnectedInputs` as pure function +- Extract `validateWorkflow` as pure function +- Extract `topologicalSort` logic +- Extract workflow migration logic (legacy nanoBanana nodes) +- Extract `clearNodeImageRefs` helper + +Plans: +- [ ] 37-01: getConnectedInputs, validateWorkflow, topologicalSort with tests +- [ ] 37-02: workflowMigration, clearNodeImageRefs with tests + +#### Phase 38: Zustand Slice Pattern + +**Goal**: Split monolithic store into composable Zustand slices for better separation of concerns +**Depends on**: Phase 37 +**Research**: Unlikely (Zustand slice patterns) +**Plans**: 3 plans + +**Scope:** +- Create slice pattern infrastructure +- Extract core slice (nodes, edges, edgeStyle, clipboard operations) +- Extract groups slice (all group operations) +- Extract execution slice (isRunning, currentNodeId, execute/regenerate actions) +- Extract persistence slice (save/load, auto-save, workflow metadata) +- Extract UI slice (modal count, quickstart, model search) +- Extract cost slice (incurredCost, cost tracking actions) +- Extract providers slice (providerSettings, API key management) +- Extract history slice (globalImageHistory, recentModels) +- Extract comments slice (comment navigation state) + +Plans: +- [ ] 38-01: Core slices (core, groups, execution) with tests +- [ ] 38-02: Feature slices (persistence, UI, cost, providers) with tests +- [ ] 38-03: Remaining slices (history, comments), final store composition + +#### Phase 39: Type Extraction & Final Integration + +**Goal**: Move store types to proper locations and verify full backward compatibility +**Depends on**: Phase 38 +**Research**: Unlikely (internal refactoring) +**Plans**: 1 plan + +**Scope:** +- Extract `WorkflowStore`, `WorkflowFile`, `ClipboardData`, `EdgeStyle` to `src/types/store.ts` +- Update all imports across codebase +- Verify all existing tests pass +- Verify all re-exports maintain backward compatibility +- Final cleanup and documentation + +Plans: +- [ ] 39-01: Extract store types, verify backward compatibility, final cleanup + +## Progress + +**Execution Order:** +Phases execute in numeric order: 1 → 2 → ... → 35 → 36 → 37 → 38 → 39 + +| Phase | Milestone | Plans Complete | Status | Completed | +|-------|-----------|----------------|--------|-----------| +| 1. Provider Infrastructure | v1.0 | 2/2 | Complete | 2026-01-09 | +| 2. Model Discovery | v1.0 | 3/3 | Complete | 2026-01-09 | +| 3. Generate Node Refactor | v1.0 | 3/3 | Complete | 2026-01-09 | +| 4. Model Search Dialog | v1.0 | 2/2 | Complete | 2026-01-09 | +| 5. Image URL Server | v1.0 | 2/2 | Complete | 2026-01-09 | +| 6. Video & Polish | v1.0 | 4/4 | Complete | 2026-01-11 | +| 7. Video Connections | v1.1 | 1/1 | Complete | 2026-01-12 | +| 8. Error Display | v1.1 | 1/1 | Complete | 2026-01-12 | +| 9. Video History | v1.1 | 1/1 | Complete | 2026-01-12 | +| 10. Node Autosizing | v1.1 | 1/1 | Complete | 2026-01-12 | +| 11. UI Polish | v1.1 | 1/1 | Complete | 2026-01-12 | +| 12. Model Improvements | v1.1 | 1/1 | Complete | 2026-01-12 | +| 13. Fix Duplicate Generations | v1.1 | 1/1 | Complete | 2026-01-12 | +| 14. Fix Drag-Connect Node Creation Bugs | v1.1 | 1/1 | Complete | 2026-01-12 | +| 15. Test Infrastructure | v1.2 | 1/1 | Complete | 2026-01-12 | +| 16. Store Modularization | v1.2 | 1/1 | Complete | 2026-01-12 | +| 17. Component Tests | v1.2 | 11/11 | Complete | 2026-01-13 | +| 18. API Route Tests | v1.2 | 5/5 | Complete | 2026-01-13 | +| 19. Type Refactoring | v1.2 | 2/2 | Complete | 2026-01-13 | +| 20. Integration Tests | v1.2 | 2/2 | Complete | 2026-01-13 | +| 21. Fix Image Input & Deduplication | v1.2 | 1/1 | Complete | 2026-01-13 | +| 22. Generate Node Dynamic Input Tests | v1.2 | 1/1 | Complete | 2026-01-13 | +| 23. Model Browser Improvements | v1.2 | 1/1 | Complete | 2026-01-13 | +| 24. Improved Cost Summary | v1.2 | 3/3 | Complete | 2026-01-17 | +| 25. Template Explorer UI | v1.3 | 2/2 | Complete | 2026-01-16 | +| 26. Template Preview Rendering | v1.3 | 1/1 | Complete | 2026-01-17 | +| 27. Node Defaults Infrastructure | v1.3 | 1/1 | Complete | 2026-01-17 | +| 28. Node Defaults UI | v1.3 | 1/1 | Complete | 2026-01-17 | +| 29. Canvas Performance | v1.3 | 0/0 | Deferred | - | +| 30. Small Fixes | v1.3 | 0/0 | Deferred | - | +| 31. Workflow Proposal System | v1.4 | 2/2 | Complete | 2026-01-26 | +| 32. Chat UI Foundation | v1.4 | 0/? | Not started | - | +| 33. Workflow Edit Safety | v1.4 | 0/? | Not started | - | +| 34. Agentic Workflow Editing | v1.4 | 0/? | Not started | - | +| 35. Large Workflow Handling | v1.4 | 0/? | Not started | - | +| 36. Execution Engine Extraction | v1.5 | 0/3 | Not started | - | +| 37. Pure Helpers Extraction | v1.5 | 0/2 | Not started | - | +| 38. Zustand Slice Pattern | v1.5 | 0/3 | Not started | - | +| 39. Type Extraction & Final Integration | v1.5 | 0/1 | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md new file mode 100644 index 00000000..7f01400d --- /dev/null +++ b/.planning/STATE.md @@ -0,0 +1,174 @@ +# Project State + +## Project Reference + +See: .planning/PROJECT.md (updated 2026-01-09) + +**Core value:** Provider infrastructure that dynamically discovers models from external APIs, enabling users to access hundreds of image/video generation models without hardcoding schemas. +**Current focus:** Agentic workflow builder (v1.4), then Store Refactoring (v1.5) + +## Current Position + +Phase: 31 of 35 (Workflow Proposal System) +Plan: 2 of 2 in current phase +Status: Phase complete +Last activity: 2026-01-26 - Completed 31-02-PLAN.md + +Progress: ░░░░░░░░░░ 6% + +## Performance Metrics + +**Velocity:** +- Total plans completed: 28 +- Average duration: 7 min +- Total execution time: 3.4 hours + +**By Phase:** + +| Phase | Plans | Total | Avg/Plan | +|-------|-------|-------|----------| +| 1. Provider Infrastructure | 2/2 | 14 min | 7 min | +| 2. Model Discovery | 3/3 | 14 min | 4.7 min | +| 3. Generate Node Refactor | 3/3 | 13 min | 4.3 min | +| 4. Model Search Dialog | 2/2 | 17 min | 8.5 min | +| 5. Image URL Server | 2/2 | 5 min | 2.5 min | +| 6. Video & Polish | 4/4 | 43 min | 14.3 min | +| 7. Video Connections | 1/1 | 4 min | 4 min | +| 8. Error Display | 1/1 | 14 min | 14 min | +| 9. Video History | 1/1 | 12 min | 12 min | +| 10. Node Autosizing | 1/1 | 2 min | 2 min | +| 11. UI Polish | 1/1 | 8 min | 8 min | +| 12. Model Improvements | 1/1 | - | - | +| 13. Fix Duplicate Generations | 1/1 | 1 min | 1 min | +| 14. Fix Drag-Connect Bugs | 1/1 | 7 min | 7 min | +| 15. Test Infrastructure | 1/1 | 3 min | 3 min | +| 16. Store Modularization | 1/1 | 22 min | 22 min | +| 19. Type Refactoring | 2/2 | 16 min | 8 min | +| 22. Generate Node Dynamic Input Tests | 1/1 | 20 min | 20 min | +| 23. Model Browser Improvements | 1/1 | 5 min | 5 min | +| 20. Integration Tests | 2/2 | 11 min | 5.5 min | +| 24. Improved Cost Summary | 3/3 | 25 min | 8 min | +| 25. Template Explorer UI | 2/2 | 23 min | 11.5 min | +| 26. Template Preview Rendering | 1/1 | 25 min | 25 min | +| 27. Node Defaults Infrastructure | 1/1 | 15 min | 15 min | +| 28. Node Defaults UI | 1/1 | 32 min | 32 min | +| 31. Workflow Proposal System | 2/2 | 6 min | 3 min | + +**Recent Trend:** +- Last 5 plans: 25 min, 15 min, 32 min, 4 min, 2 min +- Trend: Infrastructure plans (types, API) completing quickly + +## Accumulated Context + +### Decisions + +Decisions are logged in PROJECT.md Key Decisions table. +Recent decisions affecting current work: + +- Gemini always enabled via env var (GEMINI_API_KEY), Replicate/fal.ai optional +- API keys stored in localStorage under node-banana-provider-settings key +- Local state in modal to avoid saving on every keystroke +- Provider config pattern: {id, name, enabled, apiKey, apiKeyEnvVar?} +- Provider registry uses self-registration pattern via registerProvider() +- Gemini remains special-cased in /api/generate for now, not yet migrated +- Capability inference from model name/description keywords +- fal.ai API key optional (works without but rate limited) +- fal.ai auth header: "Key {apiKey}" format (not Bearer) +- fal.ai category maps directly to ModelCapability (no inference) +- 1-hour cache TTL for model lists (extended from 10 min) +- Unified API at /api/models with header-based auth +- Provider dropdown shows Gemini always, others only if API key configured +- Aspect ratio/resolution controls shown only for Gemini provider +- Backward compatibility via aliases: NanoBananaNode, saveNanoBananaDefaults +- Server-side provider execution in API route (not client-side) +- Header-based API key passing: X-Replicate-API-Key, X-Fal-API-Key +- fal.ai sync API (fal.run) instead of queue-based async +- Dual migration approach: loadWorkflow migrates + UI effect for runtime +- fal.ai icon always visible in action bar (works without key but rate limited) +- Replicate icon only visible when API key is configured +- Client-side search filtering for Replicate (their search API unreliable) +- Show all capability badges to differentiate similar models +- Extract variant suffix from fal.ai model IDs for display name +- No TTL for image store - explicit cleanup pattern (callers delete after use) +- 256KB threshold for shouldUseImageUrl (Replicate recommendation) +- Gemini excluded from video node (doesn't support video generation) +- Large videos (>20MB) return URL instead of base64 to avoid memory issues +- Fetch schema from provider API at model selection time with 10-min cache +- Filter internal params, prioritize user-relevant ones (seed, steps, guidance) +- Collapsible parameters section to keep node UI compact +- Node autosizing constraints: 200-500px width, 200-600px height, ~100px chrome +- Provider badges prepend node title (left side) with w-4 h-4 icons +- Node titles show only model name (no "Generate Image/Video" prefix) +- BaseNode supports titlePrefix prop for icon prepending +- Header aligned for saved/unsaved states with same icon layout +- isImageInput() uses word-boundary checks (not substring) to avoid matching num_images +- Workflow execution gets fresh node data from store (not stale captured array) +- regenerateNode includes parameters in request body +- MD5 content hashing for generation deduplication (fast, collision resistance not critical) +- Hash prefix in filename for O(1) duplicate lookup +- Normalized handle IDs (image, text, image-0) for connection stability across model changes +- Handle-to-schema mapping built at execution time from inputSchema +- Placeholder handles (dimmed 30%) for input types not used by model, preserving connections +- ReactFlowProvider wrapper for component tests using @xyflow/react hooks +- Zustand store mocking with vi.mock pattern returning mocked functions +- Vitest jsdom environment for React component tests +- Store utilities extracted to src/store/utils/ with re-exports for backward compatibility +- Consolidated defaultNodeDimensions (was duplicated in addNode and createGroup) +- localStorage mocking pattern for testing utility modules +- Type domain files live in src/types/*.ts with re-exports from index.ts +- BaseNodeData in annotation.ts to avoid circular imports (nodes.ts imports annotation.ts) +- 7 type domain files: annotation, nodes, providers, models, workflow, api, quickstart +- index.ts is pure re-export hub with no type definitions +- Max 8 recent models stored in localStorage, 4 displayed in UI +- Gemini models hardcoded in /api/models (not fetched from external API) +- Green color theme for Gemini provider (bg-green-500/20 text-green-300) +- Cost dialog: Gemini Cost section (with prices) + External Providers section (with model links) +- External provider pricing removed (fal.ai 429 errors, Replicate no API) - show model links instead +- Incurred cost only tracks Gemini generations +- Template category colors: blue=product, purple=style, green=composition, amber=community +- Template grid layout: 2 cols mobile, 3 cols lg+ +- Template node count calculated at runtime from workflow.nodes.length +- onWheelCapture pattern for isolating modal scroll from React Flow canvas +- min-h-0 on flexbox containers enables overflow scrolling +- overflow-clip instead of overflow-hidden when child needs scroll +- Node defaults UI uses local state pattern (load on open, save on button click) +- ModelSearchDialog reused with onModelSelected callback for defaults UI +- LLM_PROVIDERS/LLM_MODELS duplicated from LLMGenerateNode (not exported) +- Shift+V keyboard shortcut for video node creation +- Proposal types focus on purpose/description, not internal state or positions +- Reuse parseJSONFromResponse from validation.ts for consistent JSON parsing +- Validate proposal shape before returning to catch LLM errors early +- Use createGoogleGenerativeAI factory for custom API key injection (AI SDK) +- toTextStreamResponse for AI SDK v6 streaming (not toDataStreamResponse) + +### Deferred Issues + +- UAT-001: Resolved - Provider icons now use real Replicate/fal.ai logos +- ISS-001: Resolved - Generate nodes now adapt to model requirements via dynamic parameters + +### Blockers/Concerns + +- Pre-existing lint configuration issue (ESLint not configured). Not blocking development. + +### Roadmap Evolution + +- v1.0 Multi-Provider Support shipped: 6 phases (Phase 1-6), 15 plans +- Milestone v1.1 Improvements created: 6 phases (Phase 7-12), improvements and polish +- Phase 13 added: Fix duplicate generations (hashing failure investigation) +- Phase 14 added: Fix drag-connect node creation bugs (consolidated from two phases) +- Milestone v1.2 Improvements created: 6 phases (Phase 15-20), testing and modularization +- Phase 21 added: Fix Gemini Pro image input handling and image deduplication issues +- Phase 22 added: Generate node dynamic input validation and API call tests +- Phase 23 added: Model browser improvements (recently used, icon dropdown, Gemini models in list) +- Phase 24 added: Improved cost summary (fal.ai pricing API, video node tracking, multi-provider support) +- Milestone v1.3 Improvements created: 5 phases (Phase 25-29), template explorer, node defaults, canvas performance +- Phase 30 added: Small fixes +- Milestone v1.4 Features created: 5 phases (Phase 31-35), agentic workflow builder with proposal dialogs and chat-based editing +- Milestone v1.5 Store Refactoring created: 4 phases (Phase 36-39), major refactor of workflowStore.ts into modular execution engine and Zustand slices + +## Session Continuity + +Last session: 2026-01-26 +Stopped at: Completed 31-02-PLAN.md +Resume file: None +Next action: Plan Phase 32 (Chat UI Integration) diff --git a/.planning/phases/31-workflow-proposal-system/31-02-SUMMARY.md b/.planning/phases/31-workflow-proposal-system/31-02-SUMMARY.md new file mode 100644 index 00000000..8c63ec50 --- /dev/null +++ b/.planning/phases/31-workflow-proposal-system/31-02-SUMMARY.md @@ -0,0 +1,118 @@ +--- +phase: 31-workflow-proposal-system +plan: 02 +subsystem: api +tags: [vercel-ai-sdk, streaming, gemini, chat, ai-sdk-google] + +# Dependency graph +requires: + - phase: 31-01 + provides: WorkflowProposal types, domain prompts +provides: + - Streaming /api/chat endpoint + - Chat types (ChatMessage, ChatRole, ConversationState) + - Vercel AI SDK integration with Gemini +affects: [31-03-chat-ui, workflow-planning] + +# Tech tracking +tech-stack: + added: [ai@6.0.49, @ai-sdk/google@3.0.13] + patterns: [createGoogleGenerativeAI, streamText, toTextStreamResponse] + +key-files: + created: + - src/types/chat.ts + - src/app/api/chat/route.ts + modified: + - src/types/index.ts + - package.json + +key-decisions: + - "Use createGoogleGenerativeAI factory for custom API key injection" + - "toTextStreamResponse for AI SDK v6 streaming (not toDataStreamResponse)" + - "System prompt contains full Node Banana domain expertise for conversational planning" + +patterns-established: + - "Vercel AI SDK streaming pattern: createGoogleGenerativeAI → streamText → toTextStreamResponse" + +issues-created: [] + +# Metrics +duration: 2min +completed: 2026-01-26 +--- + +# Phase 31 Plan 02: Chat Infrastructure Summary + +**Streaming chat endpoint with Vercel AI SDK, Gemini integration, and conversational workflow planning assistant** + +## Performance + +- **Duration:** 2 min +- **Started:** 2026-01-26T09:37:57Z +- **Completed:** 2026-01-26T09:40:07Z +- **Tasks:** 4 +- **Files modified:** 4 + +## Accomplishments +- Installed Vercel AI SDK packages (ai, @ai-sdk/google) +- Created chat types for conversation state management +- Built streaming /api/chat endpoint with domain expertise +- Exported chat types from central type index + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Install Vercel AI SDK packages** - `744e0a7` (chore) +2. **Task 2: Create chat types** - `b263c03` (feat) +3. **Task 3: Create streaming chat endpoint** - `727e54d` (feat) +4. **Task 4: Update type exports** - `e558777` (feat) + +## Files Created/Modified +- `src/types/chat.ts` - ChatMessage, ChatRole, ConversationState, ChatRequest types +- `src/app/api/chat/route.ts` - Streaming POST endpoint with Node Banana system prompt +- `src/types/index.ts` - Added chat types re-export +- `package.json` - Added ai and @ai-sdk/google dependencies + +## Decisions Made +- Used `createGoogleGenerativeAI` factory instead of default `google` export to inject API key from GEMINI_API_KEY env var +- Used `toTextStreamResponse()` which is the correct method in AI SDK v6 (not toDataStreamResponse) +- System prompt includes comprehensive Node Banana domain knowledge for conversational workflow planning + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 3 - Blocking] Fixed google() API usage** +- **Found during:** Task 3 (Create streaming chat endpoint) +- **Issue:** Plan specified `google('gemini-2.5-flash', { apiKey })` but @ai-sdk/google doesn't accept second argument +- **Fix:** Used `createGoogleGenerativeAI({ apiKey })` factory to create provider with custom API key +- **Files modified:** src/app/api/chat/route.ts +- **Verification:** Build succeeds +- **Committed in:** 727e54d (Task 3 commit) + +**2. [Rule 3 - Blocking] Fixed streaming response method** +- **Found during:** Task 3 (Create streaming chat endpoint) +- **Issue:** `toDataStreamResponse()` doesn't exist in AI SDK v6 +- **Fix:** Changed to `toTextStreamResponse()` which is the correct method +- **Files modified:** src/app/api/chat/route.ts +- **Verification:** Build succeeds +- **Committed in:** 727e54d (Task 3 commit) + +--- + +**Total deviations:** 2 auto-fixed (2 blocking) +**Impact on plan:** Both fixes required for build to succeed. API signature differences between plan and actual AI SDK v6. + +## Issues Encountered +None beyond the API fixes above. + +## Next Phase Readiness +- Chat streaming infrastructure complete +- Ready for Chat UI integration in WelcomeModal (31-03) +- Vercel AI SDK useChat hook can connect to /api/chat endpoint + +--- +*Phase: 31-workflow-proposal-system* +*Completed: 2026-01-26*