J621/ROADMAP.md

# J621 Roadmap / To-Dos

Working list of what's still missing, roughly in priority order. Check items off as
they land.

## 1. Library management

- [x] **Duplicates engine**
  - [x] Exact MD5 duplicate detection with grouped results
  - [x] Perceptual similarity (aHash, dHash, pHash, wHash) with threshold slider
        and algorithm toggles (`imagehash` server-side)
  - [x] Visual similarity groups: pagination, selection, delete and dismiss
- [x] **Delete & storage page**
  - [x] Storage overview (watched folder, media folder, temp)
  - [x] Delete by J-ID with preview grid and bulk selection (plus per-copy
        deletion of duplicate locations)
  - [x] Temp folder cleanup
- [x] **Library search upgrades**
  - [x] Search by tags (custom + e621 tags) with a Filename / Tags / Both selector
  - [x] Tag cloud in the sidebar (click to search, hidden from guests for
        blacklisted items)
  - [x] Status filter (matched / custom / unknown — not_found/deleted arrive with
        the e621 match cache)

## 2. e621 integration

- [x] **Download progress bar on the detail view**
  - [x] Backend download task (status, progress %, downloaded/total bytes, speed)
  - [x] Progress endpoint the SPA polls; cancel support
  - [x] Frontend progress bar with speed and cancel on `/detail/<post>`
  - [x] The status footer's "Active Workers" now counts running download tasks
- [x] **Download to client** — streams the e621 original straight to the
      browser (works for guests; host-restricted proxy, no library write)
- [x] **Match local files to e621**
  - [x] Match by MD5 from the library detail, plus manual post ID linking
        (with an MD5-mismatch warning) and unlink
  - [x] Batch cache status for the whole library (`matched` / `not_found` /
        `deleted`) via background scan tasks (`/api/matches/`) and
        `manage.py match_e621`
  - [x] Show match status + e621 metadata in the library: status badge and
        match card on the detail, and `not_found` / `deleted` status filters
- [ ] **IQDB reverse search**
  - [ ] Search from a local file (`POST /iqdb_queries.json`)
  - [ ] Show candidate posts to link
- [ ] **Follows**
  - [ ] Follow tags/pools, Followed screens with unseen badges
  - [ ] "Followed" nav entry
- [ ] `F` shortcut to favorite a post, tag finder in the Ctrl+K palette

## 3. Uploads & ingestion pipeline

Files now stage first and are resolved before entering the library.

- [x] **Backend staging storage**
  - [x] `TempUpload` model: user, md5, filename, temp path, status
        (`pending` / `visual_match` / `completed` / `error`), resolution
  - [x] Files land in a temp folder first; only completed uploads move into
        the watched library folder
  - [x] Pending/upload list endpoint, file serving, discard endpoint
  - [x] `cleanup_temp_uploads` command for old staged files
- [x] **Auto-upload / auto-match**
  - [x] MD5 computed on staging; exact duplicates resolve immediately
  - [x] MD5 batch-checked against e621; matches auto-complete with post
        metadata stored and rating seeded
- [x] **IQDB similarity on upload** (SPA-driven)
  - [x] Automatic + manual IQDB checks with candidate posts
  - [x] "Visual Similarity Detected" state with candidate picker
  - [x] Perceptual-hash comparison against the library (staged uploads are
        flagged with their library matches as soon as they land)
- [x] **Upload UI**
  - [x] Three-column board: Pending & Unmatched / Visual Similarity Detected /
        Auto-uploaded & Indexed
  - [x] Metadata modal (link to e621 post, IQDB candidates, custom metadata)
  - [x] Per-file progress plus batch processing indicator

## 4. Staff tools

- [ ] **Optimization modal** (image quality, GIF/APNG resolution+FPS+compression,
      video bitrate/presets/two-pass/hw-accel) with original vs processed preview
      and override
- [ ] **Stats dashboard**
  - [ ] CPU / RAM / GPU / disk usage
  - [ ] Active jobs list (this is what makes the footer "Active Workers" real)
  - [ ] Live log tail

## 5. Shell & polish

- [ ] Toasts instead of inline messages / confirm dialogs
- [ ] Mobile drawer polish for metadata panels (design spec §layout)
- [x] Profile pictures: staff Users page sets avatars from library J-IDs
      (self-service picker in Account still pending)
- [ ] Profile extras (per-user browse preferences)
- [ ] Command palette: tag finder, recent searches

## 6. Infrastructure

- [ ] Guest blacklist refresh on a timer (`refresh_guest_blacklist` via cron/systemd)
- [ ] Production setup: build the SPA, serve via Nginx (static + `/media` +
      `/library`), systemd unit for Waitress
- [ ] Automated tests (backend API + frontend components)
- [ ] Backfill e621 metadata for items downloaded before metadata was stored
      (re-download or a "match" action)

## Dependencies / notes

- Duplicates and upload visual-similarity share the perceptual hashing layer
  (`imagehash`/`imgdd` server-side + a hash cache table).
- Download progress, optimization jobs and the stats "Active Workers" count all
  want the same background-task/progress primitive — design it once.
- IQDB and e621 matching depend on e621 credentials being configured; the SPA
  talks to e621 directly for browsing, while the backend e621 client
  (`apps/library/e621.py`) handles metadata matching and batch scans.