popiart-node-canvas/.planning/phases/13-fix-duplicate-generations/13-01-PLAN.md

phase	plan	type
13-fix-duplicate-generations	1	execute

Add content-based deduplication to the auto-save generations flow.

Purpose: Prevent duplicate images/videos in the generations folder when the same content is generated multiple times. Currently every generation is saved regardless of whether identical content already exists.

Output: Modified save-generation API that checks content hash before writing, returns existing file if duplicate found.

<execution_context> @~/.claude/get-shit-done/workflows/execute-phase.md @~/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md

Key source files: @src/app/api/save-generation/route.ts - Current save logic (no deduplication) @src/utils/imageStorage.ts - Existing hash logic for workflow images (lines 206-211)

Current behavior:

• workflowStore.ts calls /api/save-generation with imageId = Date.now() for every generation
• /api/save-generation saves to {directoryPath}/{imageId}.{ext} unconditionally
• No content hash check - identical images get duplicate files

Target behavior:

• Hash image/video content on server before saving
• Check if file with same hash already exists in generations folder
• If duplicate: return existing file path, don't create new file
• If unique: save file with hash-based filename, return new path

Prior art:

• imageStorage.ts:saveImageAndGetId uses client-side hash of substring samples (lines 206-211)
• Server-side crypto hash is more reliable and works for URL-fetched content

Task 1: Add content hashing to save-generation API src/app/api/save-generation/route.ts Add server-side content hashing using Node.js crypto:

1. Import `crypto` module at top
2. After buffer is created (line ~95-108), compute content hash:
   ```typescript
   const contentHash = crypto.createHash('md5').update(buffer).digest('hex');
   ```
3. Check if file with this hash already exists in directory:
   - List files in directoryPath
   - Check if any filename starts with the hash
   - If found, return early with existing file info
4. Use hash-prefixed filename instead of timestamp-based:
   - Change filename format to: `{hash}_{promptSnippet}.{extension}`
   - Keep prompt snippet for human readability
   - Hash ensures uniqueness without timestamp

Why MD5: Fast for binary data, collision resistance not critical for dedup (worst case: rare false duplicate). SHA256 is overkill for local file dedup.

Why hash in filename: Enables O(1) duplicate check via glob/fs.existsSync pattern match instead of reading all files.

Build passes: `npm run build` succeeds without errors. Manual test: Generate same image twice, second generation should return existing file path without creating duplicate. - MD5 hash computed from buffer content - Filename uses hash prefix: `{hash}_{prompt}.{ext}` - Duplicate check finds existing files by hash prefix - Returns existing file info if duplicate found - New files saved with hash-prefixed name Task 2: Add duplicate detection response handling src/app/api/save-generation/route.ts Enhance response to indicate whether file was deduplicated:

1. Add `isDuplicate: boolean` field to response JSON
2. When returning existing file (duplicate found):
   - Set `isDuplicate: true`
   - Return existing `filePath` and `filename`
   - Log deduplication event for observability
3. When saving new file:
   - Set `isDuplicate: false`
   - Return new `filePath` and `filename` as before

This allows callers (workflowStore) to know if deduplication occurred, useful for debugging and future UI feedback.

Test duplicate scenario: Generate same content twice, second call returns `{ success: true, isDuplicate: true, filePath: existing }`. Test unique scenario: Generate new content, returns `{ success: true, isDuplicate: false, filePath: new }`. - Response includes `isDuplicate` boolean field - Duplicate files logged: "Generation deduplicated: existing file found" - Unique files logged: "Generation saved" (existing log) Before declaring phase complete: - [ ] `npm run build` succeeds without errors - [ ] Generate same prompt twice → second returns existing file (no new file created) - [ ] Generate different prompt → new file created with hash prefix - [ ] Response includes `isDuplicate` field - [ ] Existing generations folder functionality preserved

<success_criteria>

• All tasks completed
• All verification checks pass
• No duplicate files created for identical content
• Backward compatible: existing generations still accessible
• Hash-based filenames enable fast duplicate lookup </success_criteria>

After completion, create `.planning/phases/13-fix-duplicate-generations/13-01-SUMMARY.md`