diff --git a/.planning/phases/13-fix-duplicate-generations/13-01-PLAN.md b/.planning/phases/13-fix-duplicate-generations/13-01-PLAN.md new file mode 100644 index 00000000..edab715c --- /dev/null +++ b/.planning/phases/13-fix-duplicate-generations/13-01-PLAN.md @@ -0,0 +1,133 @@ +--- +phase: 13-fix-duplicate-generations +plan: 01 +type: execute +--- + + +Add content-based deduplication to the auto-save generations flow. + +Purpose: Prevent duplicate images/videos in the generations folder when the same content is generated multiple times. Currently every generation is saved regardless of whether identical content already exists. + +Output: Modified save-generation API that checks content hash before writing, returns existing file if duplicate found. + + + +@~/.claude/get-shit-done/workflows/execute-phase.md +@~/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md + +**Key source files:** +@src/app/api/save-generation/route.ts - Current save logic (no deduplication) +@src/utils/imageStorage.ts - Existing hash logic for workflow images (lines 206-211) + +**Current behavior:** +- `workflowStore.ts` calls `/api/save-generation` with `imageId = Date.now()` for every generation +- `/api/save-generation` saves to `{directoryPath}/{imageId}.{ext}` unconditionally +- No content hash check - identical images get duplicate files + +**Target behavior:** +- Hash image/video content on server before saving +- Check if file with same hash already exists in generations folder +- If duplicate: return existing file path, don't create new file +- If unique: save file with hash-based filename, return new path + +**Prior art:** +- `imageStorage.ts:saveImageAndGetId` uses client-side hash of substring samples (lines 206-211) +- Server-side crypto hash is more reliable and works for URL-fetched content + + + + + + Task 1: Add content hashing to save-generation API + src/app/api/save-generation/route.ts + + Add server-side content hashing using Node.js crypto: + + 1. Import `crypto` module at top + 2. After buffer is created (line ~95-108), compute content hash: + ```typescript + const contentHash = crypto.createHash('md5').update(buffer).digest('hex'); + ``` + 3. Check if file with this hash already exists in directory: + - List files in directoryPath + - Check if any filename starts with the hash + - If found, return early with existing file info + 4. Use hash-prefixed filename instead of timestamp-based: + - Change filename format to: `{hash}_{promptSnippet}.{extension}` + - Keep prompt snippet for human readability + - Hash ensures uniqueness without timestamp + + Why MD5: Fast for binary data, collision resistance not critical for dedup (worst case: rare false duplicate). SHA256 is overkill for local file dedup. + + Why hash in filename: Enables O(1) duplicate check via glob/fs.existsSync pattern match instead of reading all files. + + + Build passes: `npm run build` succeeds without errors. + Manual test: Generate same image twice, second generation should return existing file path without creating duplicate. + + + - MD5 hash computed from buffer content + - Filename uses hash prefix: `{hash}_{prompt}.{ext}` + - Duplicate check finds existing files by hash prefix + - Returns existing file info if duplicate found + - New files saved with hash-prefixed name + + + + + Task 2: Add duplicate detection response handling + src/app/api/save-generation/route.ts + + Enhance response to indicate whether file was deduplicated: + + 1. Add `isDuplicate: boolean` field to response JSON + 2. When returning existing file (duplicate found): + - Set `isDuplicate: true` + - Return existing `filePath` and `filename` + - Log deduplication event for observability + 3. When saving new file: + - Set `isDuplicate: false` + - Return new `filePath` and `filename` as before + + This allows callers (workflowStore) to know if deduplication occurred, useful for debugging and future UI feedback. + + + Test duplicate scenario: Generate same content twice, second call returns `{ success: true, isDuplicate: true, filePath: existing }`. + Test unique scenario: Generate new content, returns `{ success: true, isDuplicate: false, filePath: new }`. + + + - Response includes `isDuplicate` boolean field + - Duplicate files logged: "Generation deduplicated: existing file found" + - Unique files logged: "Generation saved" (existing log) + + + + + + +Before declaring phase complete: +- [ ] `npm run build` succeeds without errors +- [ ] Generate same prompt twice → second returns existing file (no new file created) +- [ ] Generate different prompt → new file created with hash prefix +- [ ] Response includes `isDuplicate` field +- [ ] Existing generations folder functionality preserved + + + +- All tasks completed +- All verification checks pass +- No duplicate files created for identical content +- Backward compatible: existing generations still accessible +- Hash-based filenames enable fast duplicate lookup + + + +After completion, create `.planning/phases/13-fix-duplicate-generations/13-01-SUMMARY.md` +