- Active jobs on /stats get a cancel button wired to the existing
download/match cancel endpoints, showing "cancelling..." and an inline
error when the task already finished.
- Cancelling now sets the status immediately, so a task whose runner died
in a restart stops showing as "downloading".
- Download streams use a bounded read timeout (10 s connect / 60 s read):
a stalled socket fails within a minute (previously it could block
forever), and a task cancelled while stalled is marked cancelled rather
than error.
- The stats job list reaps stale download/match tasks, so phantom jobs
never appear on the dashboard.
run_download_task sets e621_match_status from MediaItem, but the module
only imported DownloadTask — every download that carried e621 metadata
failed right after indexing, leaving the file in the library unlinked.
Verified the runner end-to-end with a stubbed fetch.
- MediaItem gains e621_match_status (unknown/matched/not_found/deleted)
and e621_checked_at, backfilled for existing matched items.
- Server-side e621 client (apps/library/e621.py) using the user's stored
credentials, throttled to 2 req/s, with typed errors.
- Matching service: MD5 lookup, manual post linking (flags MD5
mismatches), unlink, metadata refresh, deleted-post detection.
- Detail actions POST /api/files/J-x/match/ and /unlink/ (uploader or
staff only).
- Background library scans: MatchTask + /api/matches/ with missing/all
scopes, progress polling, cancel and stale-task reaping; the scan
counts toward the footer's Active Workers. Same pass available as
manage.py match_e621 for cron.
- Library gains not_found/deleted status filters; the detail page adds
an e621 match card (check / link by post ID / unlink) and the metadata
card warns when a post was deleted on e621.
Backend:
- Perceptual hashes (aHash/dHash/pHash/wHash via imagehash, no imgdd)
stored on items, computed on upload/download and by the new
compute_visual_hashes command
- Duplicates API: exact duplicates (multi-location items), visual matches
for one item, union-find similarity groups with pagination
- Delete API with ownership/staff checks, per-item and per-copy deletion,
watched-folder path validation; storage overview and temp cleanup;
file list accepts j_ids batches
- Staged uploads are flagged visual_match with their library matches
(threshold via VISUAL_MATCH_THRESHOLD)
- Staff users API: list with upload counts, set role and avatar by J-ID;
User.avatar FK with signed avatar URLs
- Download threads close their DB connection and stale tasks are reaped,
keeping behaviour Gunicorn-friendly
Frontend:
- /duplicates: exact duplicate groups with per-copy delete, visual
similarity controls, search similar to a J-ID, paginated groups with
selection, bulk delete and dismiss
- /delete: storage cards, delete by J-ID with preview grid, temp cleanup
- /users: staff directory with role selects and avatar J-ID inputs
- Nav + command palette entries; top-bar avatar; upload cards and the
metadata modal show library visual matches
Backend:
- DownloadTask model + background thread runner: streams the file with
progress (%, bytes, speed) and a cancel flag, then indexes it, names it
J-<id>.<ext> and applies the e621 metadata
- DownloadTaskViewSet (create/retrieve/cancel) replaces the synchronous
endpoint; the status footer's worker counts now reflect download jobs
- Client download proxy (/api/online/file/) streams an e621 original to
the browser with Content-Disposition: attachment, restricted to the
configured e621 CDN hosts so it cannot be used as an open proxy
Frontend:
- Online detail: progress bar with percentage, transferred size, speed
and cancel while downloading; success links to the new J-ID
- New 'Download to client' button available to everyone (guests too)
JakeBreath