JakeBreath/J621
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
27cbfd882c67c9bed3d30bee6555bd7a660ad083
J621/ROADMAP.md
T
JakeBreath b024fc52d7 Add the staff stats dashboard 
Backend: GET /api/stats/ (staff only) gathers psutil CPU/memory counters,
nvidia-smi GPU stats, the cached disk numbers and the running/finished
download + match jobs. Root logging now also writes a rotating file
(backend/logs/j621.log) so the dashboard can tail it, and psutil joins the
requirements. The storage payload computation is shared with the existing
storage endpoint.

Frontend: a /stats route + Stats nav entry for staff, polling every 2 s —
per-core CPU bars, memory and swap, GPUs (utilization, VRAM, temperature),
disk with the media/temp breakdown, active jobs with progress bars,
recently finished jobs with summaries, and the log tail with level colours
and an auto-scroll toggle. Section 4 of the roadmap is complete.
2026-09-17 21:36:33 -05:00

8.3 KiB
Raw Blame History

J621 Roadmap / To-Dos

Working list of what's still missing, roughly in priority order. Check items off as they land.

1. Library management

  • Duplicates engine

    • Exact MD5 duplicate detection with grouped results
    • Perceptual similarity (aHash, dHash, pHash, wHash) with threshold slider and algorithm toggles (imagehash server-side)
    • Visual similarity groups: pagination, selection, delete and dismiss

  • Delete & storage page

    • Storage overview (watched folder, media folder, temp)
    • Delete by J-ID with preview grid and bulk selection (plus per-copy deletion of duplicate locations)
    • Temp folder cleanup

  • Library search upgrades

    • Search by tags (custom + e621 tags) with a Filename / Tags / Both selector
    • Tag cloud in the sidebar (click to search, hidden from guests for blacklisted items)
    • Status filter (matched / not_found / deleted / custom / unknown)

  • Ephemeral similarity check (/similar)

    • Drop a file: exact MD5 match, perceptual matches against the library, and e621 IQDB candidates (auto-run for images)
    • Nothing enters the library: temp files are wiped on startup, after SIMILARITY_TTL_MINUTES (default 30), on demand and by manage.py cleanup_similarity

2. e621 integration

  • Download progress bar on the detail view
    • Backend download task (status, progress %, downloaded/total bytes, speed)
    • Progress endpoint the SPA polls; cancel support
    • Frontend progress bar with speed and cancel on /detail/<post>
    • The status footer's "Active Workers" now counts running download tasks
  • Download to client — streams the e621 original straight to the browser (works for guests; host-restricted proxy, no library write)
  • Match local files to e621
    • Match by MD5 from the library detail, plus manual post ID linking (with an MD5-mismatch warning) and unlink
    • Batch cache status for the whole library (matched / not_found / deleted) via background scan tasks (/api/matches/) and manage.py match_e621
    • Show match status + e621 metadata in the library: status badge and match card on the detail, and not_found / deleted status filters
  • IQDB reverse search
    • Search from a local file (POST /iqdb_queries.json from the library detail, using the signed raw file)
    • Show candidate posts to link (thumbnail, rating, score, favs, tags; select then confirm — same flow as uploads, exact MD5 marked)
  • Follows
    • Follow tags from the + on tag chips (online + library detail) and pools from the Follow button on a pool page (validated against e621, first feed fetched immediately)
    • Followed screens: cover cards (latest image), unseen badges and Mark seen per follow
    • Merged newest-first feed with per-follow filter and unseen-only toggle
    • Blacklisted-tag cloud built in a daemon thread, then polled (10 min TTL; the user's e621 blacklist, guest default as fallback)
    • "Followed" nav entry
    • Periodic sync commands: sync_followed_tags and sync_followed_pools (one e621 search per unique tag/pool)
  • Pools browser (/pools + /pools/<id>)
    • Index search by name; category, active/deleted and sort filters; pagination according to the e621 OpenAPI spec
    • Cover thumbnails from each pool's first post (one batched post call, blacklist-aware) and deleted-pool markers
    • Pool detail: DText description, posts kept in pool order with chunked loading and in-library badges, blacklist reveal toggle, Follow button
  • F shortcut to favorite a post (online detail), tag finder in the Ctrl+K palette (e621 tag suggestions with follow toggles, recent searches, and "search e621 for …")

3. Uploads & ingestion pipeline

Files now stage first and are resolved before entering the library.

  • Backend staging storage
    • TempUpload model: user, md5, filename, temp path, status (pending / visual_match / completed / error), resolution
    • Files land in a temp folder first; only completed uploads move into the watched library folder
    • Pending/upload list endpoint, file serving, discard endpoint
    • cleanup_temp_uploads command for old staged files
  • Auto-upload / auto-match
    • MD5 computed on staging; exact duplicates resolve immediately
    • MD5 batch-checked against e621; matches auto-complete with post metadata stored and rating seeded
  • IQDB similarity on upload (SPA-driven)
    • Automatic + manual IQDB checks with candidate posts
    • "Visual Similarity Detected" state with candidate picker
    • Perceptual-hash comparison against the library (staged uploads are flagged with their library matches as soon as they land)
  • Upload UI
    • Three-column board: Pending & Unmatched / Visual Similarity Detected / Auto-uploaded & Indexed
    • Metadata modal (link to e621 post, IQDB candidates, custom metadata)
    • Per-file progress plus batch processing indicator

4. Staff tools

  • Optimization modal (client-side, no server processing)
    • Browser pipeline in a Web Worker: Mediabunny/WebCodecs for video (hardware-accelerated where available), jSquash (MozJPEG/OxiPNG/libwebp) for images, gifenc/gifuct-js + UPNG for GIF/APNG
    • Options per file type: image quality/resolution; animation scale/fps/palette; video quality/resolution/codec/container/hardware preference (codec support detected per browser)
    • Original vs processed previews with sizes and savings, progress bar with ETA and a "Processing the file…" state
    • Apply overwrites the same J-ID: POST /api/files/J-x/optimize/ replaces the file(s) and recomputes MD5/size/perceptual hashes (409 when the result matches another item, 400 when it is identical)
  • Stats dashboard (/stats, staff)
    • CPU / RAM / GPU / disk usage (psutil + nvidia-smi; capacity bars follow the design system's 80% / 95% colour thresholds)
    • Active jobs list (downloads + match scans with progress) plus the recently finished ones
    • Live log tail with level highlighting and an auto-scroll toggle (root logging now also writes a rotating backend/logs/j621.log)

5. Shell & polish

  • 18+ entry screen: an age check gates the app before anything renders (remembered per browser in localStorage)
  • Toasts instead of inline messages / confirm dialogs
  • Mobile drawer polish for metadata panels (design spec §layout)
  • Profile pictures: staff Users page sets avatars from library J-IDs (self-service picker in Account still pending)
  • Profile extras (per-user browse preferences)
  • Command palette: tag finder, recent searches, navigation commands (Ctrl+K / ⌘K, reachable from anywhere in the shell)

6. Infrastructure

  • Guest blacklist refresh on a timer (refresh_guest_blacklist via cron/systemd)
  • Follow sync on a timer (sync_followed_tags + sync_followed_pools via cron/systemd, e.g. every 30 minutes)
  • Similarity temp cleanup on a timer (cleanup_similarity via cron; the TTL also cleans lazily when new checks are created)
  • Production setup: build the SPA, serve via Nginx (static + /media + /library), systemd unit for Waitress
  • Automated tests (backend API + frontend components)
  • Backfill e621 metadata for items downloaded before metadata was stored — covered by the match scan (/api/matches/ scope all, or per-item Recheck) and manage.py match_e621 --scope all

Dependencies / notes

  • Duplicates and upload visual-similarity share the perceptual hashing layer (imagehash server-side; no external hashing service).
  • Download progress, optimization and the stats "Active Workers" count share the same task style; optimization itself runs in the browser (the home server is too weak for ffmpeg) and only the result is uploaded.
  • IQDB and e621 matching depend on e621 credentials being configured; the SPA talks to e621 directly for browsing, while the backend e621 client (apps/library/e621.py) handles metadata matching and batch scans.
Reference in New Issue View Git Blame Copy Permalink