Upload pipeline: staging, MD5 auto-match, IQDB, three-column board

Backend:
- TempUpload model: staged files (pending / visual_match / completed /
  error) with resolution, e621 payload, custom metadata and IQDB data
- Files land in a temp folder and only move into the watched library
  folder once resolved; duplicates resolve immediately without a copy
- Endpoints: stage (multipart), list, retrieve, temp file, IQDB save,
  resolve (link to post or custom metadata), discard/dismiss
- cleanup_temp_uploads command for old staged files
- Replaces the old direct-to-library upload endpoint

Frontend:
- Upload page is now a three-column board (Pending & Unmatched /
  Visual Similarity Detected / Auto-uploaded & Indexed)
- After upload: MD5s are batch-checked against e621 and matches
  auto-complete with full post metadata; remaining files run through
  IQDB and move to the similarity column when candidates exist
- Metadata modal with IQDB candidates, post-ID linking and custom
  tags/rating/notes; discard and dismiss actions
- e621 client gains fetchPostsByMd5 and iqdbSearch helpers

Roadmap updated with the completed upload items.
This commit is contained in:
2026-09-17 11:12:03 -05:00
parent 60e1113231
commit d0e2901c92
13 changed files with 1189 additions and 118 deletions
+22 -23
View File
@@ -37,32 +37,31 @@ they land.
- [ ] "Followed" nav entry
- [ ] `F` shortcut to favorite a post, tag finder in the Ctrl+K palette
## 3. Uploads & ingestion pipeline (major rework)
## 3. Uploads & ingestion pipeline
Current upload writes straight into the watched folder. The target flow matches
the original app:
Files now stage first and are resolved before entering the library.
- [ ] **Backend staging storage**
- [ ] `TempUpload` model: temp_id, user, md5, filename, temp path, status
(`pending` / `visual_match` / `completed` / `error`), timestamps
- [ ] Files land in a temp folder first; only "completed" moves them into the
watched library folder
- [ ] Pending uploads list endpoint, delete/discard endpoint
- [ ] **Auto-upload / auto-match**
- [ ] Compute MD5 on upload; if it already exists in the library, mark as
duplicate and resolve automatically (no new file)
- [ ] If the MD5 matches an e621 post, auto-complete: fetch and store the post
metadata, seed rating/tags, mark as indexed
- [ ] Batch MD5 checks against e621 for staged uploads
- [ ] **IQDB / visual similarity on upload**
- [ ] Run IQDB on staged uploads and surface candidate posts
- [ ] Perceptual-hash check against the library → "Visual Similarity Detected"
state with side-by-side comparison
- [ ] **Upload UI**
- [ ] Three-column board: Pending & Unmatched / Visual Similarity Detected /
- [x] **Backend staging storage**
- [x] `TempUpload` model: user, md5, filename, temp path, status
(`pending` / `visual_match` / `completed` / `error`), resolution
- [x] Files land in a temp folder first; only completed uploads move into
the watched library folder
- [x] Pending/upload list endpoint, file serving, discard endpoint
- [x] `cleanup_temp_uploads` command for old staged files
- [x] **Auto-upload / auto-match**
- [x] MD5 computed on staging; exact duplicates resolve immediately
- [x] MD5 batch-checked against e621; matches auto-complete with post
metadata stored and rating seeded
- [x] **IQDB similarity on upload** (SPA-driven)
- [x] Automatic + manual IQDB checks with candidate posts
- [x] "Visual Similarity Detected" state with candidate picker
- [ ] Perceptual-hash comparison against the library (shared with the
duplicates engine, still pending)
- [x] **Upload UI**
- [x] Three-column board: Pending & Unmatched / Visual Similarity Detected /
Auto-uploaded & Indexed
- [ ] Metadata modal (link to e621 post via search + custom tags/rating/notes)
- [ ] Per-file progress plus batch summary
- [x] Metadata modal (link to e621 post, IQDB candidates, custom metadata)
- [x] Per-file progress plus batch processing indicator
## 4. Staff tools