Signed media URLs embedded the current second (TimestampSigner), so every
API response re-minted every raw/thumbnail/staged URL and the browser
re-downloaded each file on every poll or navigation. Responses also carried
no cache headers at all.
- sign with a plain Signer plus a bucket-quantized exp (7d TTL, 24h bucket),
so a URL is byte-identical across responses and rotates once a day; legacy
TimestampSigner URLs stay accepted for one release
- add a v=<md5> version parameter to library media URLs so replacing a file
under the same J-ID (the optimize flow) busts caches exactly when needed
- serve_file now sends ETag/Last-Modified and a private Cache-Control and
answers conditional requests with 304; library media gets max-age 6d +
immutable, staged/similarity files 1h
- build cached 480px JPEG thumbnails for images (Pillow, keyed by MD5 under
MEDIA_ROOT/thumbs) instead of serving full-size originals through the
thumbnail endpoint; the library grid uses thumbnail_url for images too
Signed media URLs are fetched by <img>/<video> tags without an
Authorization header, so they were charged to the anonymous 120/min
bucket: past that, galleries and the fish-greeting download got 429 JSON
instead of image bytes. The raw/thumbnail/staged-file/similarity-file
actions are now exempt, and THROTTLE_ENABLED=false removes the general
anon+user limits for private/tailnet deployments (login/register/proxy
guards stay).
The SPA's e621 client also stops self-throttling so hard: 1s gap between
browsing calls (2.5s for the stricter IQDB endpoint) and a 15s cooldown
instead of 60s when e621 answers 429.
Redis backs the DRF throttles, and the stock RedisCache raises inside the
throttle check when Redis is unreachable or refusing writes (a failed RDB
snapshot disables writes by default) — turning a cache problem into a
blanket 500, which is exactly what took prod down. ResilientRedisCache
treats backend failures as cache misses, logs the first one per worker, and
lets rate limits degrade until Redis is back.
Backend: GET /api/random/ (aliases /random and /random/) returns a random
library image with:
- rating=s,q,e filtering (comma separated, default any);
- fastfetch mode (?fastfetch=1 or any User-Agent containing "fastfetch")
that only considers png/jpg/gif - what terminal viewers can show;
- JSON with j_id, filename, extension, rating, size, e621 id plus absolute
url/download_url/thumbnail_url. Authenticated callers get signed URLs so
fastfetch and image viewers can load them without headers; guests get
unsigned URLs and never receive hidden_from_guests items.
Tests: apps/library/tests/test_random.py (8 tests) covering the response
contract, guest signatures, image-only default, the fastfetch format
restriction (flag and User-Agent), rating filters, guest visibility and the
short alias.
Frontend: /random page with rating pills, R to roll, Open/Download and a
library link, plus navigation and command palette entries; needs a backend,
hidden in local mode.
nginx: /random negotiates on Accept so browsers keep getting the SPA while
scripts get the JSON (verified with the proxy and frontend containers).
Also fixes a regression from the SSRF change: the guest download proxy
still referenced the removed 'parsed' variable on its success path, so
every proxied download would have 500'd. Redirect hops are now covered by
tests with a mocked requests.get.
Test suite (the manual audit harness, now a real test):
- backend/apps/core/tests/test_security.py: 20 transactional tests across
guest visibility, object ownership, staged-upload/similarity privacy,
staff role boundaries, deletion rules, encrypted credentials, throttling
and the download allowlist. Uses temp media folders and clears cache.
Needs a one-time GRANT on test_j621 (documented in the module + README).
Images (deploy/J621-Frontend, deploy/J621-Backend, repo root as context):
- Frontend: node build -> static nginx with SPA fallback, asset caching and
an internal health endpoint.
- Backend: gunicorn + whitenoise (admin static collected at build), ffmpeg
for video thumbnails, migrations applied on start, GIT_HASH build arg so
the shell's version pill shows the commit.
Composes (distinct project names so they coexist with the dev stack):
- compose.yml (both), compose.frontend.yml, compose.backend.yml.
- A shared nginx proxy service is the only entry point (no host nginx, no
published host ports): /api,/admin,/static,/health -> backend, everything
else -> SPA; both upstreams resolve at request time so one config serves
all variants.
- A Tailscale sidecar per compose shares the nginx network namespace;
serve.default/frontend/backend.json use funnel ports 443 and 8443 only
(10000 is the remaining allowance) with ${TS_CERT_DOMAIN} substitution.
Registry: push_frontend.sh / push_backend.sh / push_all.sh build multi-arch
images and push :latest + :<sha> to the Gitea registry, following the
existing Packs-site pattern.
Docs: deploy/README.md + .env.example, ROADMAP section 6 updated,
AGENTS.md deployment and test notes.
Verified: all three composes validate, both nginx configs pass nginx -t,
both images build, the combined stack boots against real MariaDB/Redis
(migrations applied, /health ok, whitenoise serving admin static, env=prod,
git hash baked in), the proxy serves the SPA and routes /api, and
manage.py test apps.core.tests passes 20/20.
Footer:
- Left is now 'Backend Storage:' with a capacity bar (blue, peach at 80%,
red at 95% per DESIGN.md) and a used/total/free tooltip; the watched
folder path is no longer printed. /api/status/ returns a compact storage
summary instead of the path (the full storage page still shows paths to
authenticated users).
- Centre shows the backend API origin (empty = same origin). Staff get a
link to /setup to point the browser elsewhere; everyone else sees it as
plain text. The Account 'Backend connection' card is gone — this is
installation plumbing, not a per-user setting.
- Design spec updated to match.
Staff role:
- The custom role did nothing on several endpoints that only accepted
Django's is_staff/is_superuser. One canonical check now exists:
User.is_app_staff (superuser, Django staff, or the staff role), used by
the stats/users APIs, item object permissions, can_delete, upload/
similarity/download/match querysets, and the management commands
(which also pick staff-role accounts for e621 sync/match and file
ownership).
Verified with a role-only staff account (is_staff/is_superuser false):
stats/users 200, all 32 downloads + 2 scans visible, others' items
editable; the same account as role=user gets 403 for all of those.
Something on the network polls /health (and /v1/models) every 30 seconds;
the former now answers 200 with a database check instead of a 404, and
django.request is limited to ERROR in the logging config so scanner 404s
stop filling backend/logs/j621.log. Real server errors still log, and the
dev server keeps printing every request to its console.
- Active jobs on /stats get a cancel button wired to the existing
download/match cancel endpoints, showing "cancelling..." and an inline
error when the task already finished.
- Cancelling now sets the status immediately, so a task whose runner died
in a restart stops showing as "downloading".
- Download streams use a bounded read timeout (10 s connect / 60 s read):
a stalled socket fails within a minute (previously it could block
forever), and a task cancelled while stalled is marked cancelled rather
than error.
- The stats job list reaps stale download/match tasks, so phantom jobs
never appear on the dashboard.
Backend: GET /api/stats/ (staff only) gathers psutil CPU/memory counters,
nvidia-smi GPU stats, the cached disk numbers and the running/finished
download + match jobs. Root logging now also writes a rotating file
(backend/logs/j621.log) so the dashboard can tail it, and psutil joins the
requirements. The storage payload computation is shared with the existing
storage endpoint.
Frontend: a /stats route + Stats nav entry for staff, polling every 2 s —
per-core CPU bars, memory and swap, GPUs (utilization, VRAM, temperature),
disk with the media/temp breakdown, active jobs with progress bars,
recently finished jobs with summaries, and the log tail with level colours
and an auto-scroll toggle. Section 4 of the roadmap is complete.
- MediaItem gains e621_match_status (unknown/matched/not_found/deleted)
and e621_checked_at, backfilled for existing matched items.
- Server-side e621 client (apps/library/e621.py) using the user's stored
credentials, throttled to 2 req/s, with typed errors.
- Matching service: MD5 lookup, manual post linking (flags MD5
mismatches), unlink, metadata refresh, deleted-post detection.
- Detail actions POST /api/files/J-x/match/ and /unlink/ (uploader or
staff only).
- Background library scans: MatchTask + /api/matches/ with missing/all
scopes, progress polling, cancel and stale-task reaping; the scan
counts toward the footer's Active Workers. Same pass available as
manage.py match_e621 for cron.
- Library gains not_found/deleted status filters; the detail page adds
an e621 match card (check / link by post ID / unlink) and the metadata
card warns when a post was deleted on e621.
Backend:
- DownloadTask model + background thread runner: streams the file with
progress (%, bytes, speed) and a cancel flag, then indexes it, names it
J-<id>.<ext> and applies the e621 metadata
- DownloadTaskViewSet (create/retrieve/cancel) replaces the synchronous
endpoint; the status footer's worker counts now reflect download jobs
- Client download proxy (/api/online/file/) streams an e621 original to
the browser with Content-Disposition: attachment, restricted to the
configured e621 CDN hosts so it cannot be used as an open proxy
Frontend:
- Online detail: progress bar with percentage, transferred size, speed
and cancel while downloading; success links to the new J-ID
- New 'Download to client' button available to everyone (guests too)
- OS name comes from /etc/os-release (e.g. 'CachyOS Linux'), mirroring
the original app's os_utils; macOS/Windows get proper names too
- Status pill shows the name as-is instead of uppercasing it
- Removed the watched-folder path next to the logo; it now appears only
in the footer strip (spec updated to match)
Backend:
- apps.core with GET /api/status/ (env, git hash, OS, watched folder,
worker counts) and a TimingMiddleware adding X-Server-Time-Ms
- status endpoint reports its own server-side assembly time
Frontend:
- top bar shows the storage line (watched folder) and a status pill
with ENV, git hash, host OS, server time (e621 time once it exists)
- fixed 32px footer strip: storage path, active workers, build version
- collapsible filter sidebar: closed by default, Ctrl/Cmd+B toggle,
overlay drawer with backdrop blur under 1024px
- rating filters, per-page and sidebar state persist in localStorage
JakeBreath