Something on the network polls /health (and /v1/models) every 30 seconds;
the former now answers 200 with a database check instead of a 404, and
django.request is limited to ERROR in the logging config so scanner 404s
stop filling backend/logs/j621.log. Real server errors still log, and the
dev server keeps printing every request to its console.
Jobs already run on the server — leaving the page or closing the tab does
not stop them — but the SPA lost its link to them because the task id
lived in component state. The online detail page now looks up the newest
task for the post: an active one resumes the progress bar and cancel
button, and a finished one shows "your last download for this post
finished — J-xx". The downloads list accepts a post_id filter for that
lookup.
The footer's Active Workers count is also a link to the staff stats
dashboard, which is the global view of running jobs.
- Active jobs on /stats get a cancel button wired to the existing
download/match cancel endpoints, showing "cancelling..." and an inline
error when the task already finished.
- Cancelling now sets the status immediately, so a task whose runner died
in a restart stops showing as "downloading".
- Download streams use a bounded read timeout (10 s connect / 60 s read):
a stalled socket fails within a minute (previously it could block
forever), and a task cancelled while stalled is marked cancelled rather
than error.
- The stats job list reaps stale download/match tasks, so phantom jobs
never appear on the dashboard.
Backend: GET /api/stats/ (staff only) gathers psutil CPU/memory counters,
nvidia-smi GPU stats, the cached disk numbers and the running/finished
download + match jobs. Root logging now also writes a rotating file
(backend/logs/j621.log) so the dashboard can tail it, and psutil joins the
requirements. The storage payload computation is shared with the existing
storage endpoint.
Frontend: a /stats route + Stats nav entry for staff, polling every 2 s —
per-core CPU bars, memory and swap, GPUs (utilization, VRAM, temperature),
disk with the media/temp breakdown, active jobs with progress bars,
recently finished jobs with summaries, and the log tail with level colours
and an auto-scroll toggle. Section 4 of the roadmap is complete.
The Online sidebar's "Your blacklist" section is editable now: typing a
tag appends it and each entry gets an x to remove it. Changes are written
straight to the e621 account (PATCH /users/{id}.json with
user[blacklisted_tags]); the store keeps the fetched profile for the id,
updates the list optimistically and reloads it from e621. The Followed
page's blacklist cloud is rebuilt afterwards via the new
/api/follows/cloud/?refresh=1 force flag.
Verified against the live API with a reversible add/verify/restore test.
Backend:
- POST /api/files/J-x/optimize/ applies a browser-processed file: replaces
every copy (renaming when the extension changes), recomputes MD5, size
and perceptual hashes, seeds guest visibility; 400 when identical,
409 when the result matches another item, owner/staff only.
Frontend (no server-side processing by design):
- optimize.worker.ts + pipelines: Mediabunny/WebCodecs for video with a
prefer-hardware hint and per-browser codec detection; MozJPEG/OxiPNG/
libwebp (jSquash) for images; gifuct-js+gifenc and UPNG for GIF/APNG.
- OptimizeModal: per-file-type options, original vs processed previews
with sizes/savings, progress bar with ETA, then Apply (overwrite).
- Optimize button on the library detail for the uploader/staff.
run_download_task sets e621_match_status from MediaItem, but the module
only imported DownloadTask — every download that carried e621 metadata
failed right after indexing, leaving the file in the library unlinked.
Verified the runner end-to-end with a stubbed fetch.
- /similar (nav: Similar): drop a file to get the exact MD5 match, the
perceptual matches against the library, and e621 IQDB candidates
(auto-run for images when credentials are configured). Read-only —
nothing enters the library.
- SimilarityCheck model + /api/similarity/ (create/list/retrieve/delete)
with signed preview URLs and an expires_at timestamp.
- Temp files are wiped on startup (AppConfig.ready, file-only so no
database access during initialization), lazily past
SIMILARITY_TTL_MINUTES (default 30, env-overridable), on delete, and
by manage.py cleanup_similarity.
- uploadFile() takes a target path; .env.example documents the TTL.
Backend (new apps.follows):
- FollowedTag/FollowedPool/FollowedPost models; per-user follows with
unseen tracking, plus FollowCloud for the cached blacklist cloud.
- Two periodic commands sharing one fetch path: sync_followed_tags and
sync_followed_pools fetch each followed tag/pool's newest posts (one
e621 search per unique follow), store unseen feed rows, refresh covers
and pool metadata; both fall back to anonymous e621 access.
- API: /api/follows/tags|pools (follow, unfollow, mark seen), a merged
feed with per-follow filtering, and /api/follows/cloud/ which rebuilds
the blacklisted-tag cloud in a daemon thread when its 10 min cache is
stale (polling returns building/ready).
- e621 client now supports anonymous reads; trimmed posts carry preview
URLs for covers and feed tiles.
Frontend:
- /followed page: follow forms, cover cards with unseen badges and
Mark seen, merged feed with filter/unseen toggle, and a blacklist
cloud panel that polls while building. Followed nav entry added.
- MediaItem gains e621_match_status (unknown/matched/not_found/deleted)
and e621_checked_at, backfilled for existing matched items.
- Server-side e621 client (apps/library/e621.py) using the user's stored
credentials, throttled to 2 req/s, with typed errors.
- Matching service: MD5 lookup, manual post linking (flags MD5
mismatches), unlink, metadata refresh, deleted-post detection.
- Detail actions POST /api/files/J-x/match/ and /unlink/ (uploader or
staff only).
- Background library scans: MatchTask + /api/matches/ with missing/all
scopes, progress polling, cancel and stale-task reaping; the scan
counts toward the footer's Active Workers. Same pass available as
manage.py match_e621 for cron.
- Library gains not_found/deleted status filters; the detail page adds
an e621 match card (check / link by post ID / unlink) and the metadata
card warns when a post was deleted on e621.
Items flagged hidden_from_guests (blacklisted tags) returned 404 for
<img> requests since tags cannot send the auth header. The API now
exposes signed raw_url/thumbnail_url fields (mirroring upload previews
and avatars), and the SPA uses them in the gallery, detail view,
duplicates and delete screens, and upload visual matches.
Backend:
- MediaItem gains search_tags (custom + e621 tags, lowercase) and
has_custom_data, maintained on save with a data migration backfill
- File list search accepts search_type=filename|tags|both (tag search is
word-AND across the flattened tag text) and status=matched|custom|
unknown filters
- New /api/tags/cloud/ endpoint (cached 2 min per guest/auth, invalidated
on item changes and deletions) returning the most-used tags, honouring
guest visibility
Frontend:
- Library sidebar: Filename/Tags/Both selector, status pill toggles
(persisted), and a clickable tag cloud that runs a tag search
- Roadmap updated
Backend:
- Perceptual hashes (aHash/dHash/pHash/wHash via imagehash, no imgdd)
stored on items, computed on upload/download and by the new
compute_visual_hashes command
- Duplicates API: exact duplicates (multi-location items), visual matches
for one item, union-find similarity groups with pagination
- Delete API with ownership/staff checks, per-item and per-copy deletion,
watched-folder path validation; storage overview and temp cleanup;
file list accepts j_ids batches
- Staged uploads are flagged visual_match with their library matches
(threshold via VISUAL_MATCH_THRESHOLD)
- Staff users API: list with upload counts, set role and avatar by J-ID;
User.avatar FK with signed avatar URLs
- Download threads close their DB connection and stale tasks are reaped,
keeping behaviour Gunicorn-friendly
Frontend:
- /duplicates: exact duplicate groups with per-copy delete, visual
similarity controls, search similar to a J-ID, paginated groups with
selection, bulk delete and dismiss
- /delete: storage cards, delete by J-ID with preview grid, temp cleanup
- /users: staff directory with role selects and avatar J-ID inputs
- Nav + command palette entries; top-bar avatar; upload cards and the
metadata modal show library visual matches
Backend:
- DownloadTask model + background thread runner: streams the file with
progress (%, bytes, speed) and a cancel flag, then indexes it, names it
J-<id>.<ext> and applies the e621 metadata
- DownloadTaskViewSet (create/retrieve/cancel) replaces the synchronous
endpoint; the status footer's worker counts now reflect download jobs
- Client download proxy (/api/online/file/) streams an e621 original to
the browser with Content-Disposition: attachment, restricted to the
configured e621 CDN hosts so it cannot be used as an open proxy
Frontend:
- Online detail: progress bar with percentage, transferred size, speed
and cancel while downloading; success links to the new J-ID
- New 'Download to client' button available to everyone (guests too)
Files added through the upload pipeline and Download to Library are
renamed to their J-ID right after indexing, so every new library file is
traceable by its identifier (scanned files keep their existing names).
index_file now returns the created location so callers can rename it;
the location record is updated to the new path.
- 'dismiss all' clears every indexed record at once
- Indexed cards show the actual file preview: completed records now get a
signed library media URL (raw for images, thumbnail for videos) so
<img>/<video> tags can load it, including items hidden from guests
- Media raw/thumbnail endpoints accept the signature for anonymous
requests and fall back to the normal guest-filtered path otherwise
- Guest blacklist keeps a persistent Redis mirror: an expired TTL or an
unreachable e621 keeps the last successful list instead of falling
back to the small local list
- IQDB responses carry no preview/file data, so candidates only showed an
ID; the SPA now enriches them with one batched posts lookup (preview,
rating, score, favourites, dimensions, tag preview)
- Candidate tiles are selectable instead of instantly resolving: picking
one shows its info and an explicit 'Link selected post' button
- Linking a post now fetches the e621 original into the library and
drops the staged upload; when the staged file's MD5 already equals the
post's file, the staged copy is moved instead (identical bytes)
- Keep the file URL in stored e621 metadata; sanitize the new candidate
fields server-side
- Staged files are now served through a signed URL (Django signing, 24h)
so <img>/<video> tags can load previews without an Authorization
header; the file endpoint accepts header auth or a valid signature,
rejects tampered signatures, and still scopes access to the owner
- Serializer responses now carry the request context so URLs are signed
per user
- Add .webp to the allowed extensions (backend + upload hint)
Backend:
- TempUpload model: staged files (pending / visual_match / completed /
error) with resolution, e621 payload, custom metadata and IQDB data
- Files land in a temp folder and only move into the watched library
folder once resolved; duplicates resolve immediately without a copy
- Endpoints: stage (multipart), list, retrieve, temp file, IQDB save,
resolve (link to post or custom metadata), discard/dismiss
- cleanup_temp_uploads command for old staged files
- Replaces the old direct-to-library upload endpoint
Frontend:
- Upload page is now a three-column board (Pending & Unmatched /
Visual Similarity Detected / Auto-uploaded & Indexed)
- After upload: MD5s are batch-checked against e621 and matches
auto-complete with full post metadata; remaining files run through
IQDB and move to the similarity column when candidates exist
- Metadata modal with IQDB candidates, post-ID linking and custom
tags/rating/notes; discard and dismiss actions
- e621 client gains fetchPostsByMd5 and iqdbSearch helpers
Roadmap updated with the completed upload items.
Roles:
- JakeBreathild is now staff + superuser (the real account); the 'jake'
smoke-test account was demoted to a regular user
- /me exposes is_superuser and the account page shows an admin badge
e621 metadata:
- MediaItem gains e621_post_id and e621_data (trimmed post payload:
tags by category, rating, score, favourites, comments, sources,
description, pools, relationships, file info, uploader)
- Download to Library accepts the post payload from the SPA and stores
it; the item's custom rating is seeded from the e621 rating when empty
- Library detail shows an e621 metadata card: link to the in-app post,
score/favourites/comments, taxonomy-coloured tags, DText description,
sources and pools; grid cards get an e621 badge and fall back to the
e621 rating for their colour (display_rating)
- Guest visibility now also considers e621 tags, so downloaded explicit
content is hidden from anonymous visitors
Backend:
- User.role (user/uploader/staff) with can_upload; uploads and downloads
gated to uploader+; owners and staff can edit their items
- MediaItem.uploaded_by plus J-<id> identity (serializer, admin,
scan_files --user, first superuser as default owner)
- API resolves J-<id>, bare numeric ids and MD5s; neighbors and lookup
return j_ids
- Guest safety: mirror e621's anonymous default blacklist into Redis
(parses comments, negations and wildcards), flag hidden_from_guests
and filter lists, details and lookups for anonymous users
- POST /api/online/downloads/ writes an e621 file into the watched
folder and indexes it for the uploader
- MariaDB + Redis via docker compose (host ports 3307/6380), PyMySQL
driver shim, Redis cache replacing the file cache; SQLite data
dumped and loaded into MariaDB
Frontend:
- Single /detail/:itemId route with an adaptive shell: J-<id> renders
the library item, bare numbers render the e621 post
- Legacy /view/<md5> and /online/view/<id> redirect to canonical URLs
- Cards expose J-IDs; library custom-data editor is read-only for
non-owners
- Role gating: Upload hidden/blocked for regular users, account shows
the role, guest hint on the library
Backend:
- POST /api/files/lookup/ reports which MD5s are already in the library
Frontend:
- e621 client extended with post/tag/favorite types and helpers: post
search, post detail, batch posts by id, tag autocomplete, toggle
favorite
- /online: tag search with autocomplete, post grid with rating colors
and in-library badges, numbered pagination, page tag cloud, blacklist
panel and filtered counts from /users/me.json, anonymous hint
- /online/view/🆔 media viewer (sample or original, video support),
taxonomy-colored tags by category, specs sheet, favorite/unfavorite,
description, sources, pools and parent/children thumbnails
- Shared CollapsibleSidebar extracted from the library page; Online
added to the nav, command palette and sidebar toggle
Backend:
- User model gains e621_username / e621_api_key / e621_base_url
- GET/PUT /api/auth/e621/ for the owner's credentials; /me exposes only
the username and a configured flag, never the key
Frontend:
- e621 client core: Basic auth, _client param (browsers cannot set a
User-Agent), serialized queue throttled to 1 request/second, readable
error mapping
- Account screen (/account): username, API key with reveal toggle,
base URL (e621 / e926 / custom), Save + Test connection
- Credentials are fetched from the backend and held in memory only,
cleared on logout
Backend:
- GET /api/files/{md5}/neighbors/ returns previous/next items in the
current ordering (name, size, created_at) for keyboard navigation
Frontend:
- / focuses the library search input
- D downloads the file on the detail page
- [ and ] navigate to the previous/next item; hint shown on the page
- Ctrl/Cmd+K opens a command palette (navigation, toggle filters,
focus search, log in/out); Escape closes it
- Sort order now persists alongside the other library filters so
prev/next stays consistent
- OS name comes from /etc/os-release (e.g. 'CachyOS Linux'), mirroring
the original app's os_utils; macOS/Windows get proper names too
- Status pill shows the name as-is instead of uppercasing it
- Removed the watched-folder path next to the logo; it now appears only
in the footer strip (spec updated to match)
Backend:
- apps.core with GET /api/status/ (env, git hash, OS, watched folder,
worker counts) and a TimingMiddleware adding X-Server-Time-Ms
- status endpoint reports its own server-side assembly time
Frontend:
- top bar shows the storage line (watched folder) and a status pill
with ENV, git hash, host OS, server time (e621 time once it exists)
- fixed 32px footer strip: storage path, active workers, build version
- collapsible filter sidebar: closed by default, Ctrl/Cmd+B toggle,
overlay drawer with backdrop blur under 1024px
- rating filters, per-page and sidebar state persist in localStorage
Backend (Django 6.1 + DRF):
- Token auth with a custom User model (register/login/logout/me)
- Library models (MediaItem, MediaLocation) and REST endpoints
- File list/detail with search, rating filter, sorting, pagination
- Multipart upload with optional rating/tags/notes
- Range-aware media serving (video seeking) and ffmpeg thumbnails
- scan_files management command for the watched folder
Frontend (React 19 + Vite + TypeScript):
- Catppuccin Mocha design tokens from the design docs
- App shell, token persistence, protected routes
- Library grid with filters, file detail with custom data editor
- Upload page with per-file progress via XHR
- Dev proxy to the Django API
JakeBreath