JakeBreath
cd490b0a23
Backend: - Perceptual hashes (aHash/dHash/pHash/wHash via imagehash, no imgdd) stored on items, computed on upload/download and by the new compute_visual_hashes command - Duplicates API: exact duplicates (multi-location items), visual matches for one item, union-find similarity groups with pagination - Delete API with ownership/staff checks, per-item and per-copy deletion, watched-folder path validation; storage overview and temp cleanup; file list accepts j_ids batches - Staged uploads are flagged visual_match with their library matches (threshold via VISUAL_MATCH_THRESHOLD) - Staff users API: list with upload counts, set role and avatar by J-ID; User.avatar FK with signed avatar URLs - Download threads close their DB connection and stale tasks are reaped, keeping behaviour Gunicorn-friendly Frontend: - /duplicates: exact duplicate groups with per-copy delete, visual similarity controls, search similar to a J-ID, paginated groups with selection, bulk delete and dismiss - /delete: storage cards, delete by J-ID with preview grid, temp cleanup - /users: staff directory with role selects and avatar J-ID inputs - Nav + command palette entries; top-bar avatar; upload cards and the metadata modal show library visual matches
4.8 KiB
4.8 KiB
J621 Roadmap / To-Dos
Working list of what's still missing, roughly in priority order. Check items off as they land.
1. Library management
- Duplicates engine
- Exact MD5 duplicate detection with grouped results
- Perceptual similarity (aHash, dHash, pHash, wHash) with threshold slider
and algorithm toggles (
imagehashserver-side) - Visual similarity groups: pagination, selection, delete and dismiss
- Delete & storage page
- Storage overview (watched folder, media folder, temp)
- Delete by J-ID with preview grid and bulk selection (plus per-copy deletion of duplicate locations)
- Temp folder cleanup
- Library search upgrades
- Search by tags (custom + e621 tags), not just filename
- Tag cloud from the library
- Status filter (matched / not found / deleted / unknown / custom)
2. e621 integration
- Download progress bar on the detail view
- Backend download task (status, progress %, downloaded/total bytes, speed)
- Progress endpoint the SPA polls; cancel support
- Frontend progress bar with speed and cancel on
/detail/<post> - The status footer's "Active Workers" now counts running download tasks
- Download to client — streams the e621 original straight to the browser (works for guests; host-restricted proxy, no library write)
- Match local files to e621
- Match by MD5 from the library detail (manual post ID entry)
- Batch cache status for the whole library (
matched/not_found/deleted) - Show match status + e621 metadata in the library (model already supports it)
- IQDB reverse search
- Search from a local file (
POST /iqdb_queries.json) - Show candidate posts to link
- Search from a local file (
- Follows
- Follow tags/pools, Followed screens with unseen badges
- "Followed" nav entry
Fshortcut to favorite a post, tag finder in the Ctrl+K palette
3. Uploads & ingestion pipeline
Files now stage first and are resolved before entering the library.
- Backend staging storage
TempUploadmodel: user, md5, filename, temp path, status (pending/visual_match/completed/error), resolution- Files land in a temp folder first; only completed uploads move into the watched library folder
- Pending/upload list endpoint, file serving, discard endpoint
cleanup_temp_uploadscommand for old staged files
- Auto-upload / auto-match
- MD5 computed on staging; exact duplicates resolve immediately
- MD5 batch-checked against e621; matches auto-complete with post metadata stored and rating seeded
- IQDB similarity on upload (SPA-driven)
- Automatic + manual IQDB checks with candidate posts
- "Visual Similarity Detected" state with candidate picker
- Perceptual-hash comparison against the library (staged uploads are flagged with their library matches as soon as they land)
- Upload UI
- Three-column board: Pending & Unmatched / Visual Similarity Detected / Auto-uploaded & Indexed
- Metadata modal (link to e621 post, IQDB candidates, custom metadata)
- Per-file progress plus batch processing indicator
4. Staff tools
- Optimization modal (image quality, GIF/APNG resolution+FPS+compression, video bitrate/presets/two-pass/hw-accel) with original vs processed preview and override
- Stats dashboard
- CPU / RAM / GPU / disk usage
- Active jobs list (this is what makes the footer "Active Workers" real)
- Live log tail
5. Shell & polish
- Toasts instead of inline messages / confirm dialogs
- Mobile drawer polish for metadata panels (design spec §layout)
- Profile pictures: staff Users page sets avatars from library J-IDs (self-service picker in Account still pending)
- Profile extras (per-user browse preferences)
- Command palette: tag finder, recent searches
6. Infrastructure
- Guest blacklist refresh on a timer (
refresh_guest_blacklistvia cron/systemd) - Production setup: build the SPA, serve via Nginx (static +
/media+/library), systemd unit for Waitress - Automated tests (backend API + frontend components)
- Backfill e621 metadata for items downloaded before metadata was stored (re-download or a "match" action)
Dependencies / notes
- Duplicates and upload visual-similarity share the perceptual hashing layer
(
imagehash/imgddserver-side + a hash cache table). - Download progress, optimization jobs and the stats "Active Workers" count all want the same background-task/progress primitive — design it once.
- IQDB and e621 matching depend on e621 credentials being configured; the SPA already talks to e621 directly where possible.