JakeBreath
cd490b0a23
Backend: - Perceptual hashes (aHash/dHash/pHash/wHash via imagehash, no imgdd) stored on items, computed on upload/download and by the new compute_visual_hashes command - Duplicates API: exact duplicates (multi-location items), visual matches for one item, union-find similarity groups with pagination - Delete API with ownership/staff checks, per-item and per-copy deletion, watched-folder path validation; storage overview and temp cleanup; file list accepts j_ids batches - Staged uploads are flagged visual_match with their library matches (threshold via VISUAL_MATCH_THRESHOLD) - Staff users API: list with upload counts, set role and avatar by J-ID; User.avatar FK with signed avatar URLs - Download threads close their DB connection and stale tasks are reaped, keeping behaviour Gunicorn-friendly Frontend: - /duplicates: exact duplicate groups with per-copy delete, visual similarity controls, search similar to a J-ID, paginated groups with selection, bulk delete and dismiss - /delete: storage cards, delete by J-ID with preview grid, temp cleanup - /users: staff directory with role selects and avatar J-ID inputs - Nav + command palette entries; top-bar avatar; upload cards and the metadata modal show library visual matches
106 lines
4.8 KiB
Markdown
106 lines
4.8 KiB
Markdown
# J621 Roadmap / To-Dos
|
|
|
|
Working list of what's still missing, roughly in priority order. Check items off as
|
|
they land.
|
|
|
|
## 1. Library management
|
|
|
|
- [x] **Duplicates engine**
|
|
- [x] Exact MD5 duplicate detection with grouped results
|
|
- [x] Perceptual similarity (aHash, dHash, pHash, wHash) with threshold slider
|
|
and algorithm toggles (`imagehash` server-side)
|
|
- [x] Visual similarity groups: pagination, selection, delete and dismiss
|
|
- [x] **Delete & storage page**
|
|
- [x] Storage overview (watched folder, media folder, temp)
|
|
- [x] Delete by J-ID with preview grid and bulk selection (plus per-copy
|
|
deletion of duplicate locations)
|
|
- [x] Temp folder cleanup
|
|
- [ ] **Library search upgrades**
|
|
- [ ] Search by tags (custom + e621 tags), not just filename
|
|
- [ ] Tag cloud from the library
|
|
- [ ] Status filter (matched / not found / deleted / unknown / custom)
|
|
|
|
## 2. e621 integration
|
|
|
|
- [x] **Download progress bar on the detail view**
|
|
- [x] Backend download task (status, progress %, downloaded/total bytes, speed)
|
|
- [x] Progress endpoint the SPA polls; cancel support
|
|
- [x] Frontend progress bar with speed and cancel on `/detail/<post>`
|
|
- [x] The status footer's "Active Workers" now counts running download tasks
|
|
- [x] **Download to client** — streams the e621 original straight to the
|
|
browser (works for guests; host-restricted proxy, no library write)
|
|
- [ ] **Match local files to e621**
|
|
- [ ] Match by MD5 from the library detail (manual post ID entry)
|
|
- [ ] Batch cache status for the whole library (`matched` / `not_found` / `deleted`)
|
|
- [ ] Show match status + e621 metadata in the library (model already supports it)
|
|
- [ ] **IQDB reverse search**
|
|
- [ ] Search from a local file (`POST /iqdb_queries.json`)
|
|
- [ ] Show candidate posts to link
|
|
- [ ] **Follows**
|
|
- [ ] Follow tags/pools, Followed screens with unseen badges
|
|
- [ ] "Followed" nav entry
|
|
- [ ] `F` shortcut to favorite a post, tag finder in the Ctrl+K palette
|
|
|
|
## 3. Uploads & ingestion pipeline
|
|
|
|
Files now stage first and are resolved before entering the library.
|
|
|
|
- [x] **Backend staging storage**
|
|
- [x] `TempUpload` model: user, md5, filename, temp path, status
|
|
(`pending` / `visual_match` / `completed` / `error`), resolution
|
|
- [x] Files land in a temp folder first; only completed uploads move into
|
|
the watched library folder
|
|
- [x] Pending/upload list endpoint, file serving, discard endpoint
|
|
- [x] `cleanup_temp_uploads` command for old staged files
|
|
- [x] **Auto-upload / auto-match**
|
|
- [x] MD5 computed on staging; exact duplicates resolve immediately
|
|
- [x] MD5 batch-checked against e621; matches auto-complete with post
|
|
metadata stored and rating seeded
|
|
- [x] **IQDB similarity on upload** (SPA-driven)
|
|
- [x] Automatic + manual IQDB checks with candidate posts
|
|
- [x] "Visual Similarity Detected" state with candidate picker
|
|
- [x] Perceptual-hash comparison against the library (staged uploads are
|
|
flagged with their library matches as soon as they land)
|
|
- [x] **Upload UI**
|
|
- [x] Three-column board: Pending & Unmatched / Visual Similarity Detected /
|
|
Auto-uploaded & Indexed
|
|
- [x] Metadata modal (link to e621 post, IQDB candidates, custom metadata)
|
|
- [x] Per-file progress plus batch processing indicator
|
|
|
|
## 4. Staff tools
|
|
|
|
- [ ] **Optimization modal** (image quality, GIF/APNG resolution+FPS+compression,
|
|
video bitrate/presets/two-pass/hw-accel) with original vs processed preview
|
|
and override
|
|
- [ ] **Stats dashboard**
|
|
- [ ] CPU / RAM / GPU / disk usage
|
|
- [ ] Active jobs list (this is what makes the footer "Active Workers" real)
|
|
- [ ] Live log tail
|
|
|
|
## 5. Shell & polish
|
|
|
|
- [ ] Toasts instead of inline messages / confirm dialogs
|
|
- [ ] Mobile drawer polish for metadata panels (design spec §layout)
|
|
- [x] Profile pictures: staff Users page sets avatars from library J-IDs
|
|
(self-service picker in Account still pending)
|
|
- [ ] Profile extras (per-user browse preferences)
|
|
- [ ] Command palette: tag finder, recent searches
|
|
|
|
## 6. Infrastructure
|
|
|
|
- [ ] Guest blacklist refresh on a timer (`refresh_guest_blacklist` via cron/systemd)
|
|
- [ ] Production setup: build the SPA, serve via Nginx (static + `/media` +
|
|
`/library`), systemd unit for Waitress
|
|
- [ ] Automated tests (backend API + frontend components)
|
|
- [ ] Backfill e621 metadata for items downloaded before metadata was stored
|
|
(re-download or a "match" action)
|
|
|
|
## Dependencies / notes
|
|
|
|
- Duplicates and upload visual-similarity share the perceptual hashing layer
|
|
(`imagehash`/`imgdd` server-side + a hash cache table).
|
|
- Download progress, optimization jobs and the stats "Active Workers" count all
|
|
want the same background-task/progress primitive — design it once.
|
|
- IQDB and e621 matching depend on e621 credentials being configured; the SPA
|
|
already talks to e621 directly where possible.
|