curl exit 3 (malformed URL) on 'J621 Setup 0.1.1.exe': percent-encode the
asset name in the query string.
Also replace assets instead of skipping them on re-runs: NSIS builds are
not bit-reproducible, so latest.yml/latest-linux.yml must reference the
installers produced by the same run. Existing assets are deleted by id
before the fresh upload.
The Windows NSIS step failed under wine for two reasons: no X display
(nodrv_CreateWindow) and missing 32-bit libraries (failed to load
syswow64\ntdll.dll). The job now installs xvfb + wine32:i386 and runs the
desktop build under xvfb-run, with Gecko/Mono lookups disabled.
Also add `images` and `desktop` dispatch inputs so either half of the CD
can be skipped (e.g. desktop-only or images-only releases).
Gitea's container registry rejects the automatic job token
(go-gitea/gitea#23642 is still open), so the image push keeps a PAT with
only the write:package scope; a preflight step fails clearly when the
REGISTRY_USER/REGISTRY_TOKEN secrets are missing. Release creation needs
no PAT: the desktop job asks for contents: write on the job token.
The automatic job token creates the desktop release, so the CD desktop job
asks for contents: write; the images job keeps contents: read and asks for
packages: write so the job token can stand in for the scoped registry PAT.
CI stays read-only. The registry token itself remains a write:package-only
PAT (verified login + pull).
One dispatch builds and pushes both images and builds the desktop packages
into a Gitea release (desktop-v<version>, installers + latest*.yml attached,
idempotent on re-run). The live update feed stays a deploy-host operation:
CI has no SSH key for jakerasp, so push_desktop.sh --no-build remains the
way to publish it.
Repo secrets REGISTRY_USER/REGISTRY_TOKEN are set, so the image push uses
the Gitea registry credentials directly.
- ci.yml: connect to the test MariaDB as root so Django creates the test
database itself (no client install/grant step), and drop actions/cache
(cache: pip/npm): Gitea's cache service hangs the job on restore/save.
- publish.yml: prefer the REGISTRY_USER/REGISTRY_TOKEN secrets (as on other
repos) and fall back to the automatic Actions token.
- AGENTS.md: note the CI layout, the runner labels and the cache caveat.
- .gitea/workflows/ci.yml: on every push/PR, run Django checks + the full
backend suite against MariaDB/Redis services and the frontend
lint/type-check/build. Runs on the nitro-ci runner (ubuntu-latest).
- .gitea/workflows/publish.yml: manual dispatch; multi-arch build+push of
both images as :latest and :<short-sha> with GIT_HASH baked in.
- push_*.sh: non-interactive registry login for CI (REGISTRY_USER/
REGISTRY_TOKEN) and a PLATFORMS override.
Signed media URLs are fetched by <img>/<video> tags without an
Authorization header, so they were charged to the anonymous 120/min
bucket: past that, galleries and the fish-greeting download got 429 JSON
instead of image bytes. The raw/thumbnail/staged-file/similarity-file
actions are now exempt, and THROTTLE_ENABLED=false removes the general
anon+user limits for private/tailnet deployments (login/register/proxy
guards stay).
The SPA's e621 client also stops self-throttling so hard: 1s gap between
browsing calls (2.5s for the stricter IQDB endpoint) and a 15s cooldown
instead of 60s when e621 answers 429.
rsync without --delete left every previous version on the server (the
screenshot showed 0.1.0 and 0.1.1 side by side). The push now removes
non-current installers over ssh first, so the remote feed mirrors the local
one whether the transfer uses rsync or tar.
build_desktop.sh now reads the version first and removes anything in
desktop/release/ that is not that version (plus the regenerated unpacked
trees), so a version bump never leaves old installers lying around — the
same rule push_desktop.sh applies to the feed.
Both scripts now read the version from desktop/package.json: the build
report and checksums only cover the current version's artifacts, the feed
copy ignores older files, and publishing prunes previous installers from
the feed (latest*.yml only ever points at the current one).
The icon set, e621 referrer fix and setup-screen corrections shipped after
0.1.0, and the updater compares versions, so installed 0.1.0 builds would
never have seen them.
e621's CDN answers cross-site image loads that carry no Referer with a 403
(Chromium sends none from a custom-scheme page, then blocks the response as
ORB), so images never appeared in the desktop app. The main process now
attaches an e621 referrer to requests for its hosts.
The shell also answers /api, /admin, /static and /health with a 404 JSON
instead of the SPA fallback — that fallback made the setup screen's empty-URL
connection test report "Connected" against the shell itself. The setup screen
is now desktop-aware (no same-origin option, no "Use this server", clearer
copy), and the smoke test runs against a throwaway profile and covers both
regressions.
Redis backs the DRF throttles, and the stock RedisCache raises inside the
throttle check when Redis is unreachable or refusing writes (a failed RDB
snapshot disables writes by default) — turning a cache problem into a
blanket 500, which is exactly what took prod down. ResilientRedisCache
treats backend failures as cache misses, logs the first one per worker, and
lets rate limits degrade until Redis is back.
The remote feed copy now happens by default (overridable with
J621_DESKTOP_FEED_HOST or --host, skippable with --local), so a release is
one command on the build machine.
--host user@server:/path syncs deploy/data/desktop to the same path on the
server with rsync, falling back to tar over ssh when the server has no
rsync. A failed transfer now exits non-zero instead of claiming the feed is
live.
The backend only allows app://j621 through CORS_ALLOWED_ORIGINS, so
gen_env.sh now always writes that origin (plus the split-deploy frontend
when one is given) and --update keeps hand-added origins instead of
overwriting the list.
The Linux packages only installed a single 1024x1024 hicolor PNG, and
Plasma's icon lookup returns nothing for a lone oversized icon — confirmed
with kiconfinder6 under Papirus-Dark. Generate 16-1024 px PNGs from the
favicon and point electron-builder at the directory so every standard
hicolor size is installed.
deploy/build_desktop.sh builds the Arch, Debian and Windows packages into
desktop/release/ without publishing anything, lists what it produced with
sizes and SHA-256 sums for release notes, and prints the suggested Gitea
tag. Installing locally, handing the files out and attaching them to a
Gitea release all stay manual; push_desktop.sh remains the update-feed
publisher.
electron-builder now publishes latest-linux.yml / latest.yml and embeds
app-update.yml plus package-type, so electron-updater runs pacman -U or
dpkg -i through pkexec for packages and updates the per-user NSIS install
without elevation. The app checks only when asked (menu item), asks before
downloading and before installing, and J621_UPDATE_URL overrides the feed
for tests or forks.
deploy/push_desktop.sh builds and publishes the artifacts to
deploy/data/desktop, which the frontend nginx mounts read-only at
/desktop/. Verified detection and up-to-date handling against a local feed
with the packaged Arch build.
electron-builder produces j621-desktop_*_amd64.deb,
j621-desktop-*.pkg.tar.zst (zstd, lean Arch dependencies instead of the
Electron 2 era default set) and a per-user NSIS installer cross-built with
wine. The launcher entry and Electron's desktopName now agree on
io.j621.desktop so window association works, and package metadata points
at the repository homepage and the non-commercial licence.
desktop/ serves the normal frontend build over a privileged app://j621
scheme, so localStorage, OPFS, Web Workers, WebCodecs and history routing
behave exactly like Chrome. Development points at the Vite dev server;
`npm run smoke` runs headless Electron and checks the bundled app.
External links open in the system browser, and download navigations
(?download=1 or media URLs) are rerouted through webContents.downloadURL,
since preventing them cancels the download. The setup screen names the
shell's origin when the connection test fails, and the deploy docs list
app://j621 for CORS_ALLOWED_ORIGINS.
Batching must not start while files are still being uploaded, and the MD5
phase must move a whole chunk at once instead of one resolve per file:
- the upload queue drains completely first (failed uploads included) before
any matching starts;
- phase 1 asks e621 for every md5 (75 per posts.json request), builds the
md5 -> post map from the response, and sends the matches to the new
POST /api/uploads/link-bulk/ action, so a whole 75-file chunk moves into
Indexed in a single board update;
- link-bulk indexes the staged file directly when the post's MD5 matches
(identical bytes), so there is no per-file download round trip;
- phase 2 runs local visual similarity for whatever stayed pending, phase 3
the IQDB queue.
Verified end to end with real e621 files: one md5 query for the batch, one
link-bulk call, both matching files flipped to Indexed together, then the
visual and IQDB phases. 23 library tests green (link-bulk, visual phase,
deferred visual matching).
Uploads were doing md5 + local visual matching inside the upload request
(backend create) while the frontend later ran its own e621 MD5 pass, so the
pipeline looked interleaved per file. Now every step is a phase applied to
the whole batch in order:
1. upload (fast: md5 + exact-duplicate check only),
2. e621 MD5 lookup, 75 md5: metatags per posts.json request,
3. local visual similarity, one file at a time via the new
POST /api/uploads/<id>/visual-match/ action,
4. IQDB through the existing serial queue.
The board shows the active phase with its own progress bar (e621 MD5 in
peach, visual in lavender, IQDB in teal) and every step updates the staged
list as it lands. Verified from a headless run: one batched posts.json
request for 10 files, then 10 visual-match calls, then IQDB.
- MD5 auto-match now sends 75 md5: metatags per posts.json query, the same
batch size the original J621-Django app used (was 20); the limit cap no
longer truncates batches.
- every settled upload is upserted into the staged list right away, so the
Pending / Visual Similarity / Auto-uploaded columns move as files land
instead of waiting for the whole batch (auto-matched resolves and IQDB
results use the same path).
- finished upload tiles fade out and remove themselves ~2s after completing;
failures stay until cleared. Batch counters are tracked separately from the
visible tiles so the header and progress bar stay accurate as tiles vanish.
- AGENTS.md now points at the original Django app for reference behavior.
Verified live with a headless upload run: the columns showed the new files
immediately and the 8 tiles were gone ~3s after finishing.
e621 intermittently answers 429 to the IQDB endpoint; browsers hide that
status behind CORS ('Access-Control-Allow-Origin missing'), so the SPA
cannot read it. Treat every network-level failure as a possible rate limit
and pause all e621 traffic for a minute. The cooldown is shared through
localStorage so extra tabs respect it, requests are spaced 1.5s apart
instead of 1s, user-cancelled requests do not trigger a cooldown, and the
upload queue waits the cooldown out with a countdown instead of looking
stuck.
Server side: the per-process e621 gap goes from 0.5s to 1s so two gunicorn
workers cannot together exceed e621's 2/s hard limit.
The dev Vite proxy rewrites the request Host to 127.0.0.1:8000, so the
backend's absolute signed file URLs pointed at a different origin than the
SPA (localhost:5173). Images tolerated it, but the auth'd fetch that reads
the staging blob for IQDB was blocked ('Cross-Origin Request Blocked') and
every similarity check died before reaching e621.
apiUrl() now keeps API-built absolute URLs on the page's origin whenever the
SPA is in same-origin mode (dev proxy, deploy nginx) and leaves them
absolute when an explicit backend URL is configured. All consumers use it:
staging previews and the bulk modal, library cards, optimizer (range sniff +
worker), IQDB card, delete page, similar page.
e621 identification now follows the documented 'App/version (developer)'
form: server-side requests send 'J621/<hash> (JakeBreath)' and the browser
_client gets the same string, with the hash baked into the frontend image
(GIT_HASH build arg; guarded at runtime so the dev server still works).
IQDB stalls: requests now time out after 20s (a hung fetch used to block the
serialized e621 queue forever), and all checks run through one serial drain
so repeated 'Check similarity' clicks can no longer start overlapping runs
that re-download the same staging blobs. Auth/rate-limit/timeout/network
failures stop the queue with the reason and a retry button instead of
grinding through the rest.
Verified live: staged file URL is same-origin through the proxy and fetches
200 through it.
The upload grid was its own scroll container (max-height + overflow on the
same element). Firefox sizes auto grid rows to min-content in that setup,
so every tile collapsed to its footer height and the image was clipped to a
wide strip; Chromium sizes them to max-content, which is why this only
showed up in the user's browser. auto-rows-max pins rows to max-content in
both; verified with headless Firefox screenshots and Chromium measurements
(60 tiles render 152x194 each, 1845px of content in a 639px scroller).
The box is now 70vh so it behaves as a proper fixed gallery area with its
own scrollbar instead of shrinking with the item count.
IQDB progress: the page header now shows a live 'Checking IQDB — x/y'
counter with a bar while background checks run, so the queue is visible
without opening the per-file modal (which keeps its spinner, candidates
and per-file errors).
The modal held a snapshot of the staged upload, so IQDB results that landed
from the background check queue never appeared until it was closed and
reopened — the only hint a check was running was the e621 request history.
It now follows the live uploads query, so candidates, progress and errors
show up in place.
Related gaps fixed along the way:
- files flagged by the local visual-similarity check were skipped by the
IQDB pass entirely (only 'pending' files were checked), so their modal
could only ever show 'already in your library'; unresolved files of both
statuses are now checked, and the check button shows on visual-match
cards too;
- a check with no candidates posted nothing, leaving 'never checked' and
'checked, no match' indistinguishable; results are stored even when
empty and the modal now says which one it is;
- per-file failures surface in the modal instead of being swallowed, the
modal shows a spinner while the query runs and a check now/re-check
button, and auto-runs skip files already checked (and videos, since IQDB
is image-only).
Backend production code unchanged; tests pin the empty-result recording
(18 library tests, full suite 56 green).
A 200-file bulk move is one long server-side copy+index chain per file, so
the old single request sat on a spinner the whole time (and got uncomfortably
close to the 120s proxy timeout). The modal now resolves the selection in
chunks of 8:
- footer switches to a progress bar with 'n moved / done / total / percent'
while running, and the header explains that files are being indexed;
- the rating pills, selection actions, grid and close button are locked
while it runs so progress can't be lost by accident;
- failures are collected with their filenames, the modal stays open for a
summary, and 'Retry failed' re-selects only the files that are still
pending; a clean run still auto-closes with a toast.
Upload board:
- the tile grid no longer re-sorts itself as files finish (that reshuffled
the list under the cursor); it keeps insertion order, uses auto-fill tiles
of ~150px so they hold a readable size, scrolls inside a 60vh area and no
longer chains the page scroll (overscroll-contain);
- the files currently in flight are pinned in a small live strip above the
grid (name, percent, bar) so progress stays visible while the grid is
scrolled with hundreds of tiles.
Bulk rating: a 'bulk rate' button in the Pending & Unmatched header opens a
large modal with Safe/Questionable/Explicit pills, a tickable thumbnail grid
(Select all / Clear) and one confirm that moves every selected upload into
the library with that rating. Backed by POST /api/uploads/resolve-bulk/
(temp_ids + rating, own rows only): each staged file is resolved as a custom
entry (keeps its staged tags/notes), and already-completed or foreign ids are
reported per entry instead of failing the whole batch. Built for the
358-file backlog.
Tests: 4 bulk-resolve tests (resolution with the rating, input validation,
foreign ids untouched, mixed completed+pending) — full backend suite 53
green. Verified live end to end: staged a file, bulk-resolved it as 'q', saw
J-96 created with that rating, then removed the item, temp row and test
token.
The upload board partitions /api/uploads/ into Pending / Visual similarity /
Auto-uploaded, but the endpoint was paginated at 48 — a 69-file batch
silently lost 21 entries, and the similarity sweep (which reads the same
list back after uploading) only ever saw the first page. The staged-upload
list is now unpaginated: it is a transient per-user set, still limited to
the caller's rows and the uploader role. The page takes a plain array.
Watching progress with dozens of files was also poor:
- the queue uploads three files at a time instead of strictly one at a time;
- the Uploads section now shows a batch bar and 'n/m uploaded · x%' next to
the count, so the overall progress never scrolls out of sight;
- entries are ordered active-first (uploading, queued, failed, done) so the
file being uploaded is always at the top of the grid;
- tiles are larger (4 columns at lg instead of 5);
- the header reads 'Uploading n/m…' and 'Checking n file(s) against IQDB…'
instead of a bare spinner.
Tests: staged-upload list unpaginated past 48, per-user, uploader-only
(3 new; full suite 49 green). Live-checked the bare-array response.
Follow lists were paginated at the API default of 48, but the SPA treats
them as complete sets: the tag/pool toggles read their state from page one
(so the 49th follow looked unfollowed and its spinner waited for a page that
could never contain it) and the Followed page rendered only 48 cards while
showing that as the count. Both follow endpoints are now unpaginated — they
are per-user sets and still restricted to the caller's rows — and the three
consumers take plain arrays.
Post visibility: the old J621-Django online view fetched limit=320 (e621's
maximum) while ours hard-coded 48, and fetchPostsByIds capped id batches at
100. The Online browser now has a 'Posts per page' setting (48/100/200/320)
in its sidebar, mirrored in Account -> Browsing preferences, stored per user
as e621_per_page and also used for pool loading; the id-batch cap is raised
to 320.
Tests: follow list shape/isolation (4) and preference validation/merge (3)
added; the full backend suite is 46 green. Live-checked the array response
shape and the preference bounds (200 accepted, 500 rejected).
Two stale-state bugs came from react-router reusing the detail component
between posts (parent/child links hit the same /detail/<id> route):
- the previously viewed post's download panel kept rendering, so a freshly
opened post could claim 'Downloaded to the library J-xx'. The task view is
now gated on the task's post_id as well, and DetailPage keys the detail
views per item — component state (download panel, delete confirmation,
optimizer modal) cannot survive a post change any more.
- 'Your last download for this post finished' appeared on every revisit. It
now only shows when this visit actually saw the download running (derived
state, set during render), which still reports a re-attached download
finishing while staying quiet on later visits; the 'In library' button
remains the persistent indicator.
Library detail gets the same per-item key, so its delete confirmation and
optimizer modal reset between items too.
tsc, oxlint and the build are clean (the derived-state pattern was chosen
over a ref read in render / setState-in-effect, both flagged by the linter).
The build context is the repository root and there was no .dockerignore, so
'COPY backend/ ./' swept backend/venv (327 MB), backend/media (the actual
library, 224 MB), backend/logs, backend/staticfiles and backend/.env
(SECRET_KEY plus the database password) into the backend image: 1.43 GB per
architecture, including secrets headed for the registry. The frontend build
stage also copied the host's node_modules over the fresh install.
- Root .dockerignore excludes .git, virtualenvs, __pycache__, db.sqlite3,
logs/staticfiles, .env files, media/, node_modules, dist and the deploy
runtime state (data/, tailscale-state/).
- The backend Dockerfile now asserts the context is clean (.env, venv,
media/library, db.sqlite3 all absent) before collectstatic, so a missing
ignore file fails the build instead of leaking.
- Rebuilt: backend 1.43 GB -> 876 MB ('COPY backend/' is now 268 kB),
frontend stays at 65 MB. Verified by booting the compose stack with the
new image: migrations applied, /health ok, no .env or venv inside, and
/app/media is the mounted (empty) volume; the scheduler runs too.
- Removed the stale local images that still contained the library and the
dev .env.
On CachyOS (and OMF-style setups) config.fish sources a distro file that
defines fish_greeting while the shell starts. A function defined that way
beats autoloading from functions/, so the installed greeting never ran.
install.fish now appends a guarded block to ~/.config/fish/config.fish that
sources the greeting after everything else (idempotent via a marker, skipped
with --no-config), and the README documents the symptom, the check
(functions --details fish_greeting) and the manual one-liner.
Verified in a sandbox HOME that reproduces the CachyOS setup: before the
install fish resolves the distro file and prints its message, after it
resolves ~/.config/fish/functions/fish_greeting.fish and runs ours; a second
install leaves a single block. Applied to this machine's real config as
well, where fish_greeting now resolves to the user function and the
__j621_fetch_random helper is loaded.
It has a '#!/usr/bin/env fish' shebang but was committed as mode 644, so
./install.fish answered 'Permission denied' on a fresh clone. Mode is now
100755 like the deploy scripts; 'fish install.fish' worked either way.
Backend: a GreetingToken model stores only a SHA-256 hash of a j621r_…
key (shown once at creation) plus label, prefix, created/last-used. A
dedicated GreetingTokenAuthentication understands the usual
'Authorization: Token …' header but is registered only on RandomItemView
(alongside the normal token auth), so a greeting token authenticates
/api/random/ and is rejected with 401 everywhere else — exactly the scope
shell greetings need. Endpoints: GET/POST /api/auth/greeting-tokens/ and
DELETE /api/auth/greeting-tokens/{id}/ (own tokens only; the list never
returns keys or hashes).
Frontend: /tokens page (Account → Shell tokens card, command palette entry)
lists tokens with label, prefix, created/last-used and revoke (shared
confirm dialog). Creating one shows the key with Copy and 'Copy for fish'
buttons plus a pointer to extras/fish_greeting.
Tests: apps/accounts/tests/test_greeting_tokens.py — 9 tests covering
create-once semantics and hashing, hidden keys in listings, the scope
guarantee (random 200 with a signed URL; 401 on files, storage, me, tags
cloud, delete and the token list itself), unknown/revoked keys, cross-user
revocation, last-used tracking and label limits.
Verified live: created a token, rolled /random (signed URL), got 401 from
four other endpoints, saw the list omit secrets, revoked it (204) and the
same key then 401'd on /random. Full suite: 39 tests green.
install.fish now:
- asks for the API origin (default https://j621.rainbow-herring.ts.net) and
an optional token when run interactively, or takes --url/--token;
- writes ~/.config/j621Greeting/config.fish with mode 600 (it may hold a
token), keeps an existing config unless --force/--url is given, and
--no-prompt runs fully unattended;
- verifies the setup: /health must answer 'ok' and a fastfetch random roll is
attempted, reporting the API's own message when it finds nothing; exits
non-zero when the backend is unreachable so scripts notice.
README documents the prompts/flags, the chmod 600 config, and that the
greeting is designed to run without a token (guest mode: unsigned links,
guest-visible items only) with a token adding signed links and hidden items.
Tested with fish 4.9.3 in throwaway HOME dirs: flag-driven install with a
token, interactive install with a piped origin and no token, re-run keeping
the existing config, and an unreachable backend exiting 1.
The original script (J621-Django/extras/fish_greeting system) downloaded
image bytes from /random/?rating=X and read the X-File-* headers. The new
API answers with JSON and a signed link, so the greeting now:
- asks /api/random/?fastfetch=1&rating=<mode> for JSON;
- parses url/j_id/filename/md5 with jq, python3, or a grep/sed fallback;
- downloads the signed link and keeps the original display path (fastfetch
kitty/kitty-icat logos, gifsicle preprocessing for GIFs, recursion guard,
logging with rotation, greeting_mode 0/1/2 = NSFW/SFW/Questionable);
- is configured through ~/.config/j621Greeting/config.fish or universal
variables (J621_BASE, J621_WEB, J621_TOKEN, J621_FASTFETCH_CONFIG) instead
of a hardcoded host, and prints the /detail/<J-ID> link on J621_WEB;
- reports the API's own 404 message when a rating has no images, and keeps
curl quiet so failures do not spill into the greeting.
extras/fish_greeting/ contains the function, an install.fish (copies it into
the fish functions dir, creates the config once, checks dependencies) and a
README with the old-vs-new table and troubleshooting. The existing
gm-switch helper and .desktop launchers keep working (same mode file).
Tested with fish 4.9.3: syntax check on every file, guest and token runs
against the dev API (unsigned vs signed URLs), the empty-rating and
unreachable-API paths, and the grep/sed fallback with jq and python3
unavailable.
Backend: GET /api/random/ (aliases /random and /random/) returns a random
library image with:
- rating=s,q,e filtering (comma separated, default any);
- fastfetch mode (?fastfetch=1 or any User-Agent containing "fastfetch")
that only considers png/jpg/gif - what terminal viewers can show;
- JSON with j_id, filename, extension, rating, size, e621 id plus absolute
url/download_url/thumbnail_url. Authenticated callers get signed URLs so
fastfetch and image viewers can load them without headers; guests get
unsigned URLs and never receive hidden_from_guests items.
Tests: apps/library/tests/test_random.py (8 tests) covering the response
contract, guest signatures, image-only default, the fastfetch format
restriction (flag and User-Agent), rating filters, guest visibility and the
short alias.
Frontend: /random page with rating pills, R to roll, Open/Download and a
library link, plus navigation and command palette entries; needs a backend,
hidden in local mode.
nginx: /random negotiates on Accept so browsers keep getting the SPA while
scripts get the JSON (verified with the proxy and frontend containers).
Also fixes a regression from the SSRF change: the guest download proxy
still referenced the removed 'parsed' variable on its success path, so
every proxied download would have 500'd. Redirect hops are now covered by
tests with a mocked requests.get.
Adds deploy/scheduler-entrypoint.sh to the backend image (entrypoint
j621-scheduler) and a scheduler service to the four composes that have a
backend. It waits until the database and migrations are ready, runs every
job once, then keeps to the intervals:
sync_followed_tags + sync_followed_pools every 30 min (J621_SYNC_EVERY)
cleanup_similarity hourly (J621_CLEAN_EVERY)
refresh_guest_blacklist daily (J621_BLACKLIST_EVERY)
It shares the backend image, media volume and env file, so commands see
the same library and database; failures are logged and retried next
interval. Output goes to docker compose logs scheduler; the frontend-only
composes have no backend and therefore no scheduler.
Verified against a real stack: the scheduler waited for migrations, ran all
four commands on start (follow syncs as anonymous, similarity cleanup, and
a guest blacklist refresh that pulled the real 14-tag list from e621), and
kept looping. All six composes still validate.
Builds deploy/.env from .env.example, generating SECRET_KEY and both
database passwords with openssl (base64/hex only, so nothing needs quoting
in the env file or the compose parser). Derives TS_HOSTNAME and
ALLOWED_HOSTS from the tailnet hostname, optionally sets
CORS_ALLOWED_ORIGINS/CSRF_TRUSTED_ORIGINS for split deployments, forces
DEBUG=False and writes the file with mode 600.
Modes: --no-prompt (defaults only), --update (refresh hostnames/auth key
while keeping the existing secrets, reading the stored FQDN from
ALLOWED_HOSTS), --force (rotate everything, with the SECRET_KEY warning in
the docs). Refuses to overwrite an existing file otherwise.
Verified: all modes, updated FQDN preservation, mode 600, and
docker compose config accepting the generated file.
Three more composes — compose.tailnet.yml, compose.tailnet.frontend.yml,
compose.tailnet.backend.yml — mirror the funnel set exactly but mount
serve.default/frontend/backend.tailnet.json, which drop AllowFunnel. The
sidecar still registers and serves HTTPS with a tailnet certificate, but
nothing is exposed publicly; Serve also needs no ACL change.
Project names carry a -tailnet suffix so both sets can coexist, and the
README explains that each set needs its own data directory (or host), plus
how to switch a host between funnel and tailnet by starting the other file
with the same .env.
Verified: all six composes validate with docker compose config, and the
six serve configs split cleanly into funnel (AllowFunnel present) and
tailnet-only (absent).
Test suite (the manual audit harness, now a real test):
- backend/apps/core/tests/test_security.py: 20 transactional tests across
guest visibility, object ownership, staged-upload/similarity privacy,
staff role boundaries, deletion rules, encrypted credentials, throttling
and the download allowlist. Uses temp media folders and clears cache.
Needs a one-time GRANT on test_j621 (documented in the module + README).
Images (deploy/J621-Frontend, deploy/J621-Backend, repo root as context):
- Frontend: node build -> static nginx with SPA fallback, asset caching and
an internal health endpoint.
- Backend: gunicorn + whitenoise (admin static collected at build), ffmpeg
for video thumbnails, migrations applied on start, GIT_HASH build arg so
the shell's version pill shows the commit.
Composes (distinct project names so they coexist with the dev stack):
- compose.yml (both), compose.frontend.yml, compose.backend.yml.
- A shared nginx proxy service is the only entry point (no host nginx, no
published host ports): /api,/admin,/static,/health -> backend, everything
else -> SPA; both upstreams resolve at request time so one config serves
all variants.
- A Tailscale sidecar per compose shares the nginx network namespace;
serve.default/frontend/backend.json use funnel ports 443 and 8443 only
(10000 is the remaining allowance) with ${TS_CERT_DOMAIN} substitution.
Registry: push_frontend.sh / push_backend.sh / push_all.sh build multi-arch
images and push :latest + :<sha> to the Gitea registry, following the
existing Packs-site pattern.
Docs: deploy/README.md + .env.example, ROADMAP section 6 updated,
AGENTS.md deployment and test notes.
Verified: all three composes validate, both nginx configs pass nginx -t,
both images build, the combined stack boots against real MariaDB/Redis
(migrations applied, /health ok, whitenoise serving admin static, env=prod,
git hash baked in), the proxy serves the SPA and routes /api, and
manage.py test apps.core.tests passes 20/20.
Findings from the audit (50-check harness across guest/user/uploader/staff/
admin) and their fixes:
- SSRF: 'Download to Library' and the staged-upload resolve path fetched
any http(s) URL. services.validate_remote_url now enforces the e621
media allowlist and open_remote re-validates every redirect hop; the
download-task create endpoint and the guest proxy use them, so internal
addresses (127.0.0.1, LAN, metadata) are rejected with 400.
- Privilege escalation: staff could promote users to staff and demote
other staff. Role changes across the staff boundary now require an
admin, matching the account-deletion rules; the Users page hides what
the backend would refuse.
- SPA-only gating: /api/storage/ and /api/duplicates/* were readable by
any authenticated account (absolute paths, duplicate groups) while the
SPA only shows them to uploaders. They now require CanUpload.
- Throttling (REST_FRAMEWORK, env-overridable, counted in Redis):
anon 120/min, user 600/min, login 5/min, register 20/hour, guest e621
proxy 60/hour. Login now goes through a throttled view.
- e621 API keys are encrypted at rest with a Fernet key derived from
SECRET_KEY (apps/accounts/crypto.py); a data migration encrypts existing
rows and the column widens first. Reads decrypt transparently, legacy
plaintext still works, and a changed SECRET_KEY reads as 'not
configured' instead of leaking. Rotating SECRET_KEY now invalidates
stored keys as well as signed media URLs.
- Hardening: the server refuses to start with DEBUG=False while SECRET_KEY
is still the development default.
Verified: corrected harness 50/50 (guest visibility, IDOR, signed-URL
tamper/expiry, staged-upload/similarity privacy, role matrix, SSRF),
login throttles at the 6th attempt with 429, anon polling unaffected, the
guest proxy still reaches allowlisted hosts, live e621 auth works with the
decrypted key, and DB rows hold only ciphertext.
The app can now operate backend-agnostically: a production build still
asks on first start, but /setup also offers 'Continue without a backend'
(stored as the sentinel 'none'), and the shell adapts:
- Local mode shows only the e621-facing pages: Online (search, post view,
favorites, blacklist editor, direct downloads) and Pools. Library,
uploads, duplicates, stats, users, follows and similarity are hidden
from the nav and palette and render a 'backend needed' state when
reached directly; /detail/<e621 id> still works while /detail/J-x asks
for a backend.
- e621 credentials are stored in this browser (j621.e621) and the
Account page becomes a credentials-only screen; the store reads/writes
locally instead of /api/auth/e621/.
- The header replaces the status pill and login/user area with an e621
credentials button and a 'Setup Backend' button; the footer shows
'Local mode — e621 features only' with the same entry point.
- In-library lookups (badges/browse markers) are skipped without a
backend; 'Download to client' links straight to the e621 file instead
of the backend proxy; follow buttons and palette follow toggles are
hidden; api() fails fast with a clear message if something slips
through.
Mode logic lives in lib/backend.ts (URL / '' same-origin / 'none') with
its matrix verified in Node; tsc, oxlint and the build are clean.
Two bugs made the toggle look broken even though the preference was
stored correctly (JakeBreathild had online_hot_default false):
- The Account card never seeded online_hot_default into its form state,
so the checkbox always rendered checked via the '?? true' fallback.
It now starts from the saved value.
- Turning the toggle off did not change Online when the URL still
carried tags=order:hot (e.g. Ctrl+Shift+R reloading the old URL).
order:hot on its own is the default, not a deliberate search, so it is
now removed when the account has the toggle off; 'order:hot canine' or
any other search is still left untouched. The decision moved into
hotDefault.ts (set-hot / clear-hot / keep) with the matrix verified in
Node.
New online_hot_default preference (on by default, so guests and accounts
that never saved preferences get it): opening /online without a search
replaces the URL with ?tags=order:hot — e621's metatag for the order the
Hot page uses — so it is visible in the search field and shareable.
Existing searches are never touched: they live in the URL, so a refresh
or back/forward keeps them, while a fresh visit (nav pill, first time,
after a long time away) gets the hot default again. Turning the toggle
off opens Online on the site-wide newest posts as before.
The toggle sits in Account -> Browsing preferences and saves with the
rest of the settings; the backend validates the new boolean
(400 for non-boolean input).
DELETE /api/users/{id}/ with guards: nobody deletes the account they are
signed in as (400); staff can delete regular/uploader accounts only,
while admins can also delete staff and admins (403 for staff targets
otherwise, and the last admin can never be deleted). Deleting a user
removes their follows, tokens and staged uploads — including the staged
files on disk — while library items survive and simply lose their owner
(uploaded_by is SET_NULL), as does download/match/similarity history.
The Users page gets a per-row delete button behind the shared confirm
dialog, hidden wherever the backend would refuse (own row, or a
staff/admin target when the actor is not an admin).
Verified against the dev server: staff 204 for a regular account, 400
for self, 403 for an admin; admin 204; a plain account gets 403. After
deleting a user that owned J-81 and had a staged file, the file was gone
and J-81 survived with a null owner.
Footer:
- Left is now 'Backend Storage:' with a capacity bar (blue, peach at 80%,
red at 95% per DESIGN.md) and a used/total/free tooltip; the watched
folder path is no longer printed. /api/status/ returns a compact storage
summary instead of the path (the full storage page still shows paths to
authenticated users).
- Centre shows the backend API origin (empty = same origin). Staff get a
link to /setup to point the browser elsewhere; everyone else sees it as
plain text. The Account 'Backend connection' card is gone — this is
installation plumbing, not a per-user setting.
- Design spec updated to match.
Staff role:
- The custom role did nothing on several endpoints that only accepted
Django's is_staff/is_superuser. One canonical check now exists:
User.is_app_staff (superuser, Django staff, or the staff role), used by
the stats/users APIs, item object permissions, can_delete, upload/
similarity/download/match querysets, and the management commands
(which also pick staff-role accounts for e621 sync/match and file
ownership).
Verified with a role-only staff account (is_staff/is_superuser false):
stats/users 200, all 32 downloads + 2 scans visible, others' items
editable; the same account as role=user gets 403 for all of those.
Replaces the build-time VITE_API_BASE knob with a runtime setup screen so
one build works same-origin and cross-origin:
- frontend/src/lib/backend.ts stores the API origin in localStorage
(empty = same origin). DEFAULT_BACKEND_URL is the clearly marked,
easily edited prefilled default — the matrix.org equivalent; set it to
your public API origin.
- Production builds show /setup before anything else on first start,
with a connection test against /health (or leave it blank for this
server). The route stays reachable from Account -> Backend connection;
switching backends clears the previous backend's token and reloads.
- Input normalisation: scheme defaulted (https, http for localhost),
trailing slashes trimmed; a failed cross-origin test points at
CORS_ALLOWED_ORIGINS.
- Dev keeps defaulting to the same-origin Vite proxy; /setup can be
visited manually.
Verified: normalisation cases in Node, /health returns CORS headers for
an allowed origin, tsc/oxlint/build clean.
- django-cors-headers with env-driven CORS_ALLOWED_ORIGINS,
CORS_ALLOW_ALL_ORIGINS, CORS_ALLOW_CREDENTIALS and CSRF_TRUSTED_ORIGINS;
same-origin traffic is unaffected and a disallowed origin gets no CORS
headers. Token auth needs no cookies, so credentials stay off by default.
- TRUST_PROXY_HEADERS=true lets a TLS-terminating proxy supply
X-Forwarded-Proto/Host for correct absolute URLs.
- API media URLs (raw/thumbnail/upload/similarity/staged previews) are now
absolute, built from the request host, so <img>/<video>/fetch() keep
working when the SPA is served from another origin. Signed URLs are still
per-user; nothing is stored in the DB.
- The SPA gains VITE_API_BASE (build-time, empty = same-origin) applied by
a small apiUrl() helper used for XHR/fetch and the few URL fallbacks.
Verified with a throwaway instance: preflight and GET responses carry the
allowed origin, foreign origins get nothing, media GETs include CORS for
cross-origin fetch(), and payload URLs use the request host (dev :8000
unchanged).
Adds a preferences JSON field on the user plus GET/POST
/api/auth/preferences/ (merge semantics, validated keys), surfaced in
/auth/me/ and typed on the frontend.
The Account page gains a Browsing preferences card: landing page,
default rating filter, default sort, items per page and thumbnail size.
Signed-in users also sync these while browsing (the Library sidebar's
rating/sort/per-page controls and the new thumbnail slider), debounced;
on load the account's values seed the local UI state, so settings follow
the user across browsers. Guests keep the existing localStorage
behaviour. The thumbnail size drives the media grids (Library, Online,
pool detail) between 140 and 320px columns.
Verified the API against the dev server: merge keeps untouched keys,
invalid values 400, values round-trip through /auth/me/.
Users can now set their own profile picture instead of asking staff:
POST /api/auth/avatar/ accepts a J-ID (or blank to clear) and reuses the
same item resolution as the staff endpoint. The Account page gains a
profile picture card with a searchable, paginated library grid — any
item works (the thumbnail is used), the current avatar is marked, and
the choice is confirmed before saving. Refreshing the signed-in user
updates the shell avatar immediately.
Verified against the dev server: set, clear, unknown J-ID -> 400,
anonymous -> 401.
DESIGN.md asks for metadata panels to slide up from the bottom under
768px. A BottomSheet component provides the trigger pill and the sheet
(backdrop blur, scroll lock, Escape to close) and ResponsivePanel swaps
between it and the existing desktop <aside>, so panel content is mounted
once either way.
Applied to the Library/Online/Similar detail asides and to long pool
descriptions. Upload needed nothing: its metadata editor is already a
full-screen modal that stacks cleanly on small screens.
Inline banners and per-row status text reported action results all over
the app; they are replaced by a small toast stack (bottom-right, Level 3
floating well styling) that only speaks for actions: successes fade,
errors stay until dismissed, and form-field validation stays inline.
Destructive actions no longer use bespoke inline confirm steps (the
library detail's Confirm delete button) or fire immediately (duplicate
copies/items, delete page selections, temp cleanup, upload discard, job
cancellation): they all go through one promise-based confirm dialog
(confirmAction) with a danger variant, Escape/backdrop to cancel.
Future sessions are told to fetch and grep https://e621.wiki/openapi.yaml
before touching e621 endpoints, with the response-shape gotchas we hit
(bare arrays vs wrapped objects, pool search parameters, the form-encoded
PATCH for user settings). The durable constraints — token auth, no
server-side media processing, no imgdd, no chat — are recorded too so a
compacted session cannot regress them.
- Animated WebP (J-82) was being flattened to its first frame: browsers
have no animated WebP encoder, so the modal now sniffs the file header
(4 KB range request), explains the limitation and disables Process
instead of overwriting the file with a single frame.
- APNGs saved as .png took the still-image path and lost their frames;
the header sniff looks for the acTL chunk, routes them to the animation
pipeline (all frames + delays) and switches the UI to the animation
options. The worker double-checks the header too, so no path can
flatten an APNG.
- New dependency-free imageformat module, verified against real files
(J-82 animated, static WebP/PNG, and a generated APNG named .png).
Something on the network polls /health (and /v1/models) every 30 seconds;
the former now answers 200 with a database check instead of a 404, and
django.request is limited to ERROR in the logging config so scanner 404s
stop filling backend/logs/j621.log. Real server errors still log, and the
dev server keeps printing every request to its console.
Jobs already run on the server — leaving the page or closing the tab does
not stop them — but the SPA lost its link to them because the task id
lived in component state. The online detail page now looks up the newest
task for the post: an active one resumes the progress bar and cancel
button, and a finished one shows "your last download for this post
finished — J-xx". The downloads list accepts a post_id filter for that
lookup.
The footer's Active Workers count is also a link to the staff stats
dashboard, which is the global view of running jobs.
- Active jobs on /stats get a cancel button wired to the existing
download/match cancel endpoints, showing "cancelling..." and an inline
error when the task already finished.
- Cancelling now sets the status immediately, so a task whose runner died
in a restart stops showing as "downloading".
- Download streams use a bounded read timeout (10 s connect / 60 s read):
a stalled socket fails within a minute (previously it could block
forever), and a task cancelled while stalled is marked cancelled rather
than error.
- The stats job list reaps stale download/match tasks, so phantom jobs
never appear on the dashboard.
Backend: GET /api/stats/ (staff only) gathers psutil CPU/memory counters,
nvidia-smi GPU stats, the cached disk numbers and the running/finished
download + match jobs. Root logging now also writes a rotating file
(backend/logs/j621.log) so the dashboard can tail it, and psutil joins the
requirements. The storage payload computation is shared with the existing
storage endpoint.
Frontend: a /stats route + Stats nav entry for staff, polling every 2 s —
per-core CPU bars, memory and swap, GPUs (utilization, VRAM, temperature),
disk with the media/temp breakdown, active jobs with progress bars,
recently finished jobs with summaries, and the log tail with level colours
and an auto-scroll toggle. Section 4 of the roadmap is complete.
The optimizer now shows which browser is running (e.g. "Firefox 141")
next to the encoder probe results, explains the common Firefox case
("Firefox does not implement H.264/HEVC encoding"), and offers a
one-click switch to the container that has working encoders.
Your machine's H.264/HEVC encoders report support and then emit no video
samples at all, which no configure-time check can see. The optimizer now
test-encodes three frames per codec (cached for 30 days), shows the result
as diagnostics in the modal (AVC x / VP9 (hardware) / ...), feeds the
working codecs to the worker so Auto tries them first, and tells you to
switch to WebM when a container has no usable encoder.
Video quality is now a 10-100 slider with a predicted bitrate and size
for this clip (mirroring Mediabunny's mapping), which makes the trade-off
visible instead of guessing from presets.
Your symptom (audio kept, video 0x0) means the hardware H.264 encode
silently produced no video samples while the audio was copied. The result
check now parses the moov box and requires a 'vide' handler, so an
audio-only output is treated as a failed attempt: the pipeline walks all
codec x hardware/software configurations (starting with hardware when
enabled) and only reports an error when every one fails, naming what each
attempt returned. Resizing is also skipped entirely unless a smaller
height was requested, keeping the scaler out of the pipeline.
StreamTarget closes the underlying writer when the output is finalized —
that is also what commits an OPFS file — so closing it ourselves threw
"Can not close locked stream". finalize() now simply reads the committed
file back, and every failure path (invalid attempt, encode error,
validation failure) aborts the write and deletes the temporary entry.
The hand-rolled chunk assembly was verified correct in Node but still
produced a broken MP4 in the browser, so stop relying on it: the muxer now
streams into an Origin Private File System file (random access is exactly
what MP4 needs), the worker returns the OPFS File directly, and the entry
is deleted after a successful apply (stale ones are purged hourly). The
chunk collector remains only as a fallback for browsers without OPFS.
The result is validated before it reaches the UI: MP4s must contain a
moov box and WebM files must start with the EBML magic, so a broken muxer
output surfaces as an error instead of a 0x0 preview.
When the muxer patched a byte range inside an already-written chunk (the
mdat header, for example), the merge trimmed the right side of the old
segment but left its full blob on the left, duplicating megabytes and
shifting every box offset — the MP4 still reported its duration but had
no usable video track (0x0 in the browser, "moov atom not found" in
ffprobe).
The collector/assembler now lives in its own module and was verified in
Node with a real transmux of J-81: the assembled file matches the source
(h264 1280x720, 500 frames, AAC, 20.84 s) byte for byte in structure.
Qualitative presets made Mediabunny use quantizer (CRF-like) encoding, but
hardware H.264 encoders commonly ignore the per-frame quantizer and fall
back to a very low default bitrate, producing files far smaller than the
preset implies. Passing preferBitrate makes the preset map to an explicit
bitrate so hardware and software paths agree.
- The streaming assembler applied chunks sorted by position, so header
patches written last could be overwritten by earlier data. Chunks now
apply in arrival order (newest write wins per byte range) and are only
laid out by position afterwards.
- The Optimize modal now reads the media metadata itself: both previews
show resolution and duration, and the processed video is flagged in red
when it comes out shorter than the original (a truncated encode is no
longer something you have to guess by file size).
- MatchCard and IqdbCard are siblings in the library detail aside and
both used key={item.j_id}, so React warned about duplicate children
(J-81 twice). Their keys are now unique per card.
- The video pipeline no longer decides "this browser can't encode" from
a single getFirstEncodableVideoCodec probe with source dimensions.
It now probes Conversion.init with the real (even) output size across
codec candidates and hardware preferences, uses the first valid
configuration, and reports exactly what was tried when nothing works.
It also fails fast with a clear message if VideoEncoder is missing in
the worker.
The Online sidebar's "Your blacklist" section is editable now: typing a
tag appends it and each entry gets an x to remove it. Changes are written
straight to the e621 account (PATCH /users/{id}.json with
user[blacklisted_tags]); the store keeps the fetched profile for the id,
updates the list optimistically and reloads it from e621. The Followed
page's blacklist cloud is rebuilt afterwards via the new
/api/follows/cloud/?refresh=1 force flag.
Verified against the live API with a reversible add/verify/restore test.
- PNG failed because wasm-bindgen's init only accepts a URL string (or a
module/buffer): passing { module_or_path } broke every PNG optimization.
- GIF/APNG showed flashing colors because patches were written with
putImageData, which ignores the transparency that means "keep the
previous frame". Patches now blend through drawImage with correct
disposal handling, and quantization keeps a transparent palette entry.
- Video no longer downloads the whole file into memory: Mediabunny reads
it with range requests (UrlSource) and the muxer streams into Blob
chunks (StreamTarget) that are assembled with last-write-wins range
merging. BufferTarget held the entire output in memory, which crashed
the tab on large files.
Backend:
- POST /api/files/J-x/optimize/ applies a browser-processed file: replaces
every copy (renaming when the extension changes), recomputes MD5, size
and perceptual hashes, seeds guest visibility; 400 when identical,
409 when the result matches another item, owner/staff only.
Frontend (no server-side processing by design):
- optimize.worker.ts + pipelines: Mediabunny/WebCodecs for video with a
prefer-hardware hint and per-browser codec detection; MozJPEG/OxiPNG/
libwebp (jSquash) for images; gifuct-js+gifenc and UPNG for GIF/APNG.
- OptimizeModal: per-file-type options, original vs processed previews
with sizes/savings, progress bar with ETA, then Apply (overwrite).
- Optimize button on the library detail for the uploader/staff.
Clicking the + follow toggle used to move focus onto that button, so the
next Enter re-triggered the follow instead of searching the highlighted
tag. The toggle now prevents the mouse-down focus steal, and the footer
hint spells out that Enter searches while + follows.
The command palette now searches e621 tags as you type (debounced
/tags.json suggestions with category chips and post counts), lets you
search a tag online, follow/unfollow it inline and open it on e621, and
remembers recent searches in localStorage. New navigation commands cover
Pools, Followed and Similar. On the online detail, F toggles the current
post's favorite.
run_download_task sets e621_match_status from MediaItem, but the module
only imported DownloadTask — every download that carried e621 metadata
failed right after indexing, leaving the file in the library unlinked.
Verified the runner end-to-end with a stubbed fetch.
- /similar (nav: Similar): drop a file to get the exact MD5 match, the
perceptual matches against the library, and e621 IQDB candidates
(auto-run for images when credentials are configured). Read-only —
nothing enters the library.
- SimilarityCheck model + /api/similarity/ (create/list/retrieve/delete)
with signed preview URLs and an expires_at timestamp.
- Temp files are wiped on startup (AppConfig.ready, file-only so no
database access during initialization), lazily past
SIMILARITY_TTL_MINUTES (default 30, env-overridable), on delete, and
by manage.py cleanup_similarity.
- uploadFile() takes a target path; .env.example documents the TTL.
Source-available, non-commercial licence (attribution + copyleft) at the
repo root as LICENSE, with the year/name filled in, ASCII punctuation,
a termination-on-breach clause with a 30-day cure, and a clearer
'no licence' wording in section 0. README gains a Licence section.
Logging out or switching accounts kept the previous user's React Query
cache (follows, feed, cloud, e621 pages), so the new account briefly
saw the old one's followed tags/pools until each query refetched. The
query client now lives in lib/queryClient.ts and login/register/logout
clear it alongside the e621 credential store.
- Remove the follow-a-tag / follow-a-pool forms; follows happen from tag
chips (+ on detail views) and the Follow pool button on pool pages, so
the empty states now point there instead.
- Move the blacklisted-tag cloud to a full-width horizontal panel at the
bottom of the page.
- Tag and pool card grids cap at two columns so the cover cards are
bigger.
PostCard's Link is inline by default; wrapped in a div on the pool detail
page it stopped being blockified like a grid child, so its rating-tinted
border collapsed into a vertical line on the left. The card is now
block-level, and the pool grid only wraps blacklisted posts (for the red
ring), so normal cards render exactly like the Online grid.
- /pools: search by name, category/active filters, sort options and
pagination per the OpenAPI spec, with covers taken from each pool's
first post in one batched post call; blacklisted covers fall back to a
placeholder and deleted pools get an archive marker.
- /pools/<id>: DText description, post grid kept in the pool's own order
with chunked loading, in-library badges, a blacklist reveal toggle and
a Follow pool button wired into the follows API.
- e621 client gains fetchPools/fetchPool; Pools nav entry added.
The chip toggle swaps its icon for a tiny current-color spinner from
the click until the follows list reflects the new state, covering the
refetch gap so '+' never flashes back before the checkmark.
TagChip gains a followable mode that renders a small toggle inside the
chip: '+' follows the tag, '✓' (click to unfollow) once followed, and a
red marker with the reason as its title when e621 rejects it. Enabled on
the library detail's e621 metadata card (matched posts) and on online
post detail tags; guests see plain chips and the label click keeps
working beside the toggle.
Backend (new apps.follows):
- FollowedTag/FollowedPool/FollowedPost models; per-user follows with
unseen tracking, plus FollowCloud for the cached blacklist cloud.
- Two periodic commands sharing one fetch path: sync_followed_tags and
sync_followed_pools fetch each followed tag/pool's newest posts (one
e621 search per unique follow), store unseen feed rows, refresh covers
and pool metadata; both fall back to anonymous e621 access.
- API: /api/follows/tags|pools (follow, unfollow, mark seen), a merged
feed with per-follow filtering, and /api/follows/cloud/ which rebuilds
the blacklisted-tag cloud in a daemon thread when its 10 min cache is
stale (polling returns building/ready).
- e621 client now supports anonymous reads; trimmed posts carry preview
URLs for covers and feed tiles.
Frontend:
- /followed page: follow forms, cover cards with unseen badges and
Mark seen, merged feed with filter/unseen toggle, and a blacklist
cloud panel that polls while building. Followed nav entry added.
- New IQDB card on local item pages: fetches the signed raw file, POSTs it
to e621's /iqdb_queries.json with the user's credentials, and lists
candidates as tiles (thumbnail, rating, score%) with exact-MD5 markers.
- Selecting a candidate opens a detail panel (rating, score, favs, size,
tag preview) and linking is an explicit confirm that reuses the e621
match endpoint; videos show a note since IQDB is image-only.
- Guests don't see the card; linking follows the uploader/staff rule.
A homescreen-style age gate shown before auth or any route: J621 brand
mark, the explicit 18+ check, an Enter action remembered per browser in
localStorage (j621.age-verified), and a blocked state if the visitor
chooses Leave.
- MediaItem gains e621_match_status (unknown/matched/not_found/deleted)
and e621_checked_at, backfilled for existing matched items.
- Server-side e621 client (apps/library/e621.py) using the user's stored
credentials, throttled to 2 req/s, with typed errors.
- Matching service: MD5 lookup, manual post linking (flags MD5
mismatches), unlink, metadata refresh, deleted-post detection.
- Detail actions POST /api/files/J-x/match/ and /unlink/ (uploader or
staff only).
- Background library scans: MatchTask + /api/matches/ with missing/all
scopes, progress polling, cancel and stale-task reaping; the scan
counts toward the footer's Active Workers. Same pass available as
manage.py match_e621 for cron.
- Library gains not_found/deleted status filters; the detail page adds
an e621 match card (check / link by post ID / unlink) and the metadata
card warns when a post was deleted on e621.
Items flagged hidden_from_guests (blacklisted tags) returned 404 for
<img> requests since tags cannot send the auth header. The API now
exposes signed raw_url/thumbnail_url fields (mirroring upload previews
and avatars), and the SPA uses them in the gallery, detail view,
duplicates and delete screens, and upload visual matches.
JakeBreath