The preview box is a flex item in the fixed-height modal column, so default
flex-shrink squeezed it and overflow-hidden cropped tall images to a slice.
shrink-0 keeps its natural size (capped at 60vh) and lets the column scroll.
The updater now resolves the newest non-draft desktop-v* release through the
Gitea API at check time (J621_UPDATE_REPO, lowercase because the API path is
case-sensitive), picks the platform's latest*.yml asset and uses that release
as a generic electron-updater feed; J621_UPDATE_URL still overrides
everything. Verified against the live API: release picked, yml fetched,
artifact HEAD 200.
electron-builder's publish.url is now metadata only (still needed so the
build emits latest*.yml). Docs updated: CD release assets are the feed, the
website /desktop/ feed only matters for installs before 0.1.2.
- the header row is no longer capped at 1600px, so the nav hugs the left
edge and the status/account controls the right; main content and footer
keep their width
- OnlinePage's blacklist chips and the Followed page's blacklisted-tag
cloud start collapsed behind a count toggle (N tags / N entries)
- near-fullscreen panel (up to 1400px / 92vh) with two independently
scrolling columns; the left preview uses self-start so its border hugs the
image instead of stretching to the modal height
- J-ID matches render as a larger tile grid (was 48px rows) and IQDB
candidates get bigger tiles too
- Link to e621 post and Custom metadata are shown inline instead of behind
tabs, each with its own heading
- new ThumbImage component: spinner while loading and a broken-image icon on
error, with alt text removed so a pending tile never reads as 'J-7786'
- write thumbnails to a .part file and os.replace() them, so concurrent
requests never read a half-written JPEG
- a stale thumbnail plus a vanished source no longer raises through the
request (getmtime on a missing file returned 500); it falls back cleanly
- ensure_thumbnail(item) warms the preview when a file is indexed, keeping
image decoding out of the request path
Match rows stored a signed URL minted when the scan ran, so it aged out (or
used the pre-stable signing scheme) and the modal showed broken tiles even
after legacy signatures were fixed. Rows now carry item_id/j_id and the
detail serializer mints a fresh thumbnail URL per request; matches whose
item no longer exists are dropped.
A TimestampSigner value is an HMAC over 'payload:timestamp', so a plain
Signer's HMAC check accepts it and the embedded timestamp then reached the
JSON decoder, raising JSONDecodeError (not BadSignature) and surfacing as a
500. That broke every stored visual-match thumbnail URL minted before the
stable scheme, so the J-ID match tiles never loaded on prod.
Detect the legacy shape by its extra separator and verify it with
TimestampSigner; malformed input returns None instead of raising.
The board rendered every persisted completed row with a dismiss button and a
dismiss-all that sent the whole column to the 1000-id bulk endpoint. Now that
the backend deletes completed rows and reports them through the status feed:
- the Auto-uploaded & Indexed column renders the live feed only; a reload or
leaving the page forgets them, and there is nothing to dismiss (just a
client-side clear)
- duplicates complete during staging and get their card immediately from the
upload response
- failures get their own persisted column with per-card retry and discard
- bulk discard chunks requests at 500 ids, so backlogs over the server's
1000-id cap are still removable in one action
Every auto-matched, duplicate or manually resolved upload left a completed
TempUpload row on the board until it was dismissed by hand, so the rows
accumulated without bound and the bulk dismiss (capped at 1000 ids) failed
once there were more. The original app never stored these: they are
notifications, not records.
- complete_temp_upload now appends {filename, J-ID, resolution, post} to a
bounded recent_completions feed on UploadRun and deletes the staged row
- staging duplicates never create a board record either; the create response
carries the J-ID and preview so the SPA can show the card immediately
- status_payload returns the feed (newest first, signed thumbnails) for the
live board; finalize_round counts deleted matches in processed
- resolve/link-bulk return synthetic completion payloads
- migration 0012 adds the field and purges the existing completed backlog
(and any stray staged files) on deploy
Signed media URLs embedded the current second (TimestampSigner), so every
API response re-minted every raw/thumbnail/staged URL and the browser
re-downloaded each file on every poll or navigation. Responses also carried
no cache headers at all.
- sign with a plain Signer plus a bucket-quantized exp (7d TTL, 24h bucket),
so a URL is byte-identical across responses and rotates once a day; legacy
TimestampSigner URLs stay accepted for one release
- add a v=<md5> version parameter to library media URLs so replacing a file
under the same J-ID (the optimize flow) busts caches exactly when needed
- serve_file now sends ETag/Last-Modified and a private Cache-Control and
answers conditional requests with 304; library media gets max-age 6d +
immutable, staged/similarity files 1h
- build cached 480px JPEG thumbnails for images (Pillow, keyed by MD5 under
MEDIA_ROOT/thumbs) instead of serving full-size originals through the
thumbnail endpoint; the library grid uses thumbnail_url for images too
curl exit 3 (malformed URL) on 'J621 Setup 0.1.1.exe': percent-encode the
asset name in the query string.
Also replace assets instead of skipping them on re-runs: NSIS builds are
not bit-reproducible, so latest.yml/latest-linux.yml must reference the
installers produced by the same run. Existing assets are deleted by id
before the fresh upload.
The Windows NSIS step failed under wine for two reasons: no X display
(nodrv_CreateWindow) and missing 32-bit libraries (failed to load
syswow64\ntdll.dll). The job now installs xvfb + wine32:i386 and runs the
desktop build under xvfb-run, with Gecko/Mono lookups disabled.
Also add `images` and `desktop` dispatch inputs so either half of the CD
can be skipped (e.g. desktop-only or images-only releases).
Gitea's container registry rejects the automatic job token
(go-gitea/gitea#23642 is still open), so the image push keeps a PAT with
only the write:package scope; a preflight step fails clearly when the
REGISTRY_USER/REGISTRY_TOKEN secrets are missing. Release creation needs
no PAT: the desktop job asks for contents: write on the job token.
The automatic job token creates the desktop release, so the CD desktop job
asks for contents: write; the images job keeps contents: read and asks for
packages: write so the job token can stand in for the scoped registry PAT.
CI stays read-only. The registry token itself remains a write:package-only
PAT (verified login + pull).
One dispatch builds and pushes both images and builds the desktop packages
into a Gitea release (desktop-v<version>, installers + latest*.yml attached,
idempotent on re-run). The live update feed stays a deploy-host operation:
CI has no SSH key for jakerasp, so push_desktop.sh --no-build remains the
way to publish it.
Repo secrets REGISTRY_USER/REGISTRY_TOKEN are set, so the image push uses
the Gitea registry credentials directly.
- ci.yml: connect to the test MariaDB as root so Django creates the test
database itself (no client install/grant step), and drop actions/cache
(cache: pip/npm): Gitea's cache service hangs the job on restore/save.
- publish.yml: prefer the REGISTRY_USER/REGISTRY_TOKEN secrets (as on other
repos) and fall back to the automatic Actions token.
- AGENTS.md: note the CI layout, the runner labels and the cache caveat.
- .gitea/workflows/ci.yml: on every push/PR, run Django checks + the full
backend suite against MariaDB/Redis services and the frontend
lint/type-check/build. Runs on the nitro-ci runner (ubuntu-latest).
- .gitea/workflows/publish.yml: manual dispatch; multi-arch build+push of
both images as :latest and :<short-sha> with GIT_HASH baked in.
- push_*.sh: non-interactive registry login for CI (REGISTRY_USER/
REGISTRY_TOKEN) and a PLATFORMS override.
Signed media URLs are fetched by <img>/<video> tags without an
Authorization header, so they were charged to the anonymous 120/min
bucket: past that, galleries and the fish-greeting download got 429 JSON
instead of image bytes. The raw/thumbnail/staged-file/similarity-file
actions are now exempt, and THROTTLE_ENABLED=false removes the general
anon+user limits for private/tailnet deployments (login/register/proxy
guards stay).
The SPA's e621 client also stops self-throttling so hard: 1s gap between
browsing calls (2.5s for the stricter IQDB endpoint) and a 15s cooldown
instead of 60s when e621 answers 429.
rsync without --delete left every previous version on the server (the
screenshot showed 0.1.0 and 0.1.1 side by side). The push now removes
non-current installers over ssh first, so the remote feed mirrors the local
one whether the transfer uses rsync or tar.
build_desktop.sh now reads the version first and removes anything in
desktop/release/ that is not that version (plus the regenerated unpacked
trees), so a version bump never leaves old installers lying around — the
same rule push_desktop.sh applies to the feed.
Both scripts now read the version from desktop/package.json: the build
report and checksums only cover the current version's artifacts, the feed
copy ignores older files, and publishing prunes previous installers from
the feed (latest*.yml only ever points at the current one).
The icon set, e621 referrer fix and setup-screen corrections shipped after
0.1.0, and the updater compares versions, so installed 0.1.0 builds would
never have seen them.
e621's CDN answers cross-site image loads that carry no Referer with a 403
(Chromium sends none from a custom-scheme page, then blocks the response as
ORB), so images never appeared in the desktop app. The main process now
attaches an e621 referrer to requests for its hosts.
The shell also answers /api, /admin, /static and /health with a 404 JSON
instead of the SPA fallback — that fallback made the setup screen's empty-URL
connection test report "Connected" against the shell itself. The setup screen
is now desktop-aware (no same-origin option, no "Use this server", clearer
copy), and the smoke test runs against a throwaway profile and covers both
regressions.
Redis backs the DRF throttles, and the stock RedisCache raises inside the
throttle check when Redis is unreachable or refusing writes (a failed RDB
snapshot disables writes by default) — turning a cache problem into a
blanket 500, which is exactly what took prod down. ResilientRedisCache
treats backend failures as cache misses, logs the first one per worker, and
lets rate limits degrade until Redis is back.
The remote feed copy now happens by default (overridable with
J621_DESKTOP_FEED_HOST or --host, skippable with --local), so a release is
one command on the build machine.
--host user@server:/path syncs deploy/data/desktop to the same path on the
server with rsync, falling back to tar over ssh when the server has no
rsync. A failed transfer now exits non-zero instead of claiming the feed is
live.
The backend only allows app://j621 through CORS_ALLOWED_ORIGINS, so
gen_env.sh now always writes that origin (plus the split-deploy frontend
when one is given) and --update keeps hand-added origins instead of
overwriting the list.
The Linux packages only installed a single 1024x1024 hicolor PNG, and
Plasma's icon lookup returns nothing for a lone oversized icon — confirmed
with kiconfinder6 under Papirus-Dark. Generate 16-1024 px PNGs from the
favicon and point electron-builder at the directory so every standard
hicolor size is installed.
deploy/build_desktop.sh builds the Arch, Debian and Windows packages into
desktop/release/ without publishing anything, lists what it produced with
sizes and SHA-256 sums for release notes, and prints the suggested Gitea
tag. Installing locally, handing the files out and attaching them to a
Gitea release all stay manual; push_desktop.sh remains the update-feed
publisher.
electron-builder now publishes latest-linux.yml / latest.yml and embeds
app-update.yml plus package-type, so electron-updater runs pacman -U or
dpkg -i through pkexec for packages and updates the per-user NSIS install
without elevation. The app checks only when asked (menu item), asks before
downloading and before installing, and J621_UPDATE_URL overrides the feed
for tests or forks.
deploy/push_desktop.sh builds and publishes the artifacts to
deploy/data/desktop, which the frontend nginx mounts read-only at
/desktop/. Verified detection and up-to-date handling against a local feed
with the packaged Arch build.
electron-builder produces j621-desktop_*_amd64.deb,
j621-desktop-*.pkg.tar.zst (zstd, lean Arch dependencies instead of the
Electron 2 era default set) and a per-user NSIS installer cross-built with
wine. The launcher entry and Electron's desktopName now agree on
io.j621.desktop so window association works, and package metadata points
at the repository homepage and the non-commercial licence.
desktop/ serves the normal frontend build over a privileged app://j621
scheme, so localStorage, OPFS, Web Workers, WebCodecs and history routing
behave exactly like Chrome. Development points at the Vite dev server;
`npm run smoke` runs headless Electron and checks the bundled app.
External links open in the system browser, and download navigations
(?download=1 or media URLs) are rerouted through webContents.downloadURL,
since preventing them cancels the download. The setup screen names the
shell's origin when the connection test fails, and the deploy docs list
app://j621 for CORS_ALLOWED_ORIGINS.
Batching must not start while files are still being uploaded, and the MD5
phase must move a whole chunk at once instead of one resolve per file:
- the upload queue drains completely first (failed uploads included) before
any matching starts;
- phase 1 asks e621 for every md5 (75 per posts.json request), builds the
md5 -> post map from the response, and sends the matches to the new
POST /api/uploads/link-bulk/ action, so a whole 75-file chunk moves into
Indexed in a single board update;
- link-bulk indexes the staged file directly when the post's MD5 matches
(identical bytes), so there is no per-file download round trip;
- phase 2 runs local visual similarity for whatever stayed pending, phase 3
the IQDB queue.
Verified end to end with real e621 files: one md5 query for the batch, one
link-bulk call, both matching files flipped to Indexed together, then the
visual and IQDB phases. 23 library tests green (link-bulk, visual phase,
deferred visual matching).
Uploads were doing md5 + local visual matching inside the upload request
(backend create) while the frontend later ran its own e621 MD5 pass, so the
pipeline looked interleaved per file. Now every step is a phase applied to
the whole batch in order:
1. upload (fast: md5 + exact-duplicate check only),
2. e621 MD5 lookup, 75 md5: metatags per posts.json request,
3. local visual similarity, one file at a time via the new
POST /api/uploads/<id>/visual-match/ action,
4. IQDB through the existing serial queue.
The board shows the active phase with its own progress bar (e621 MD5 in
peach, visual in lavender, IQDB in teal) and every step updates the staged
list as it lands. Verified from a headless run: one batched posts.json
request for 10 files, then 10 visual-match calls, then IQDB.
- MD5 auto-match now sends 75 md5: metatags per posts.json query, the same
batch size the original J621-Django app used (was 20); the limit cap no
longer truncates batches.
- every settled upload is upserted into the staged list right away, so the
Pending / Visual Similarity / Auto-uploaded columns move as files land
instead of waiting for the whole batch (auto-matched resolves and IQDB
results use the same path).
- finished upload tiles fade out and remove themselves ~2s after completing;
failures stay until cleared. Batch counters are tracked separately from the
visible tiles so the header and progress bar stay accurate as tiles vanish.
- AGENTS.md now points at the original Django app for reference behavior.
Verified live with a headless upload run: the columns showed the new files
immediately and the 8 tiles were gone ~3s after finishing.
e621 intermittently answers 429 to the IQDB endpoint; browsers hide that
status behind CORS ('Access-Control-Allow-Origin missing'), so the SPA
cannot read it. Treat every network-level failure as a possible rate limit
and pause all e621 traffic for a minute. The cooldown is shared through
localStorage so extra tabs respect it, requests are spaced 1.5s apart
instead of 1s, user-cancelled requests do not trigger a cooldown, and the
upload queue waits the cooldown out with a countdown instead of looking
stuck.
Server side: the per-process e621 gap goes from 0.5s to 1s so two gunicorn
workers cannot together exceed e621's 2/s hard limit.
The dev Vite proxy rewrites the request Host to 127.0.0.1:8000, so the
backend's absolute signed file URLs pointed at a different origin than the
SPA (localhost:5173). Images tolerated it, but the auth'd fetch that reads
the staging blob for IQDB was blocked ('Cross-Origin Request Blocked') and
every similarity check died before reaching e621.
apiUrl() now keeps API-built absolute URLs on the page's origin whenever the
SPA is in same-origin mode (dev proxy, deploy nginx) and leaves them
absolute when an explicit backend URL is configured. All consumers use it:
staging previews and the bulk modal, library cards, optimizer (range sniff +
worker), IQDB card, delete page, similar page.
e621 identification now follows the documented 'App/version (developer)'
form: server-side requests send 'J621/<hash> (JakeBreath)' and the browser
_client gets the same string, with the hash baked into the frontend image
(GIT_HASH build arg; guarded at runtime so the dev server still works).
IQDB stalls: requests now time out after 20s (a hung fetch used to block the
serialized e621 queue forever), and all checks run through one serial drain
so repeated 'Check similarity' clicks can no longer start overlapping runs
that re-download the same staging blobs. Auth/rate-limit/timeout/network
failures stop the queue with the reason and a retry button instead of
grinding through the rest.
Verified live: staged file URL is same-origin through the proxy and fetches
200 through it.
The upload grid was its own scroll container (max-height + overflow on the
same element). Firefox sizes auto grid rows to min-content in that setup,
so every tile collapsed to its footer height and the image was clipped to a
wide strip; Chromium sizes them to max-content, which is why this only
showed up in the user's browser. auto-rows-max pins rows to max-content in
both; verified with headless Firefox screenshots and Chromium measurements
(60 tiles render 152x194 each, 1845px of content in a 639px scroller).
The box is now 70vh so it behaves as a proper fixed gallery area with its
own scrollbar instead of shrinking with the item count.
IQDB progress: the page header now shows a live 'Checking IQDB — x/y'
counter with a bar while background checks run, so the queue is visible
without opening the per-file modal (which keeps its spinner, candidates
and per-file errors).
The modal held a snapshot of the staged upload, so IQDB results that landed
from the background check queue never appeared until it was closed and
reopened — the only hint a check was running was the e621 request history.
It now follows the live uploads query, so candidates, progress and errors
show up in place.
Related gaps fixed along the way:
- files flagged by the local visual-similarity check were skipped by the
IQDB pass entirely (only 'pending' files were checked), so their modal
could only ever show 'already in your library'; unresolved files of both
statuses are now checked, and the check button shows on visual-match
cards too;
- a check with no candidates posted nothing, leaving 'never checked' and
'checked, no match' indistinguishable; results are stored even when
empty and the modal now says which one it is;
- per-file failures surface in the modal instead of being swallowed, the
modal shows a spinner while the query runs and a check now/re-check
button, and auto-runs skip files already checked (and videos, since IQDB
is image-only).
Backend production code unchanged; tests pin the empty-result recording
(18 library tests, full suite 56 green).
A 200-file bulk move is one long server-side copy+index chain per file, so
the old single request sat on a spinner the whole time (and got uncomfortably
close to the 120s proxy timeout). The modal now resolves the selection in
chunks of 8:
- footer switches to a progress bar with 'n moved / done / total / percent'
while running, and the header explains that files are being indexed;
- the rating pills, selection actions, grid and close button are locked
while it runs so progress can't be lost by accident;
- failures are collected with their filenames, the modal stays open for a
summary, and 'Retry failed' re-selects only the files that are still
pending; a clean run still auto-closes with a toast.
Upload board:
- the tile grid no longer re-sorts itself as files finish (that reshuffled
the list under the cursor); it keeps insertion order, uses auto-fill tiles
of ~150px so they hold a readable size, scrolls inside a 60vh area and no
longer chains the page scroll (overscroll-contain);
- the files currently in flight are pinned in a small live strip above the
grid (name, percent, bar) so progress stays visible while the grid is
scrolled with hundreds of tiles.
Bulk rating: a 'bulk rate' button in the Pending & Unmatched header opens a
large modal with Safe/Questionable/Explicit pills, a tickable thumbnail grid
(Select all / Clear) and one confirm that moves every selected upload into
the library with that rating. Backed by POST /api/uploads/resolve-bulk/
(temp_ids + rating, own rows only): each staged file is resolved as a custom
entry (keeps its staged tags/notes), and already-completed or foreign ids are
reported per entry instead of failing the whole batch. Built for the
358-file backlog.
Tests: 4 bulk-resolve tests (resolution with the rating, input validation,
foreign ids untouched, mixed completed+pending) — full backend suite 53
green. Verified live end to end: staged a file, bulk-resolved it as 'q', saw
J-96 created with that rating, then removed the item, temp row and test
token.
The upload board partitions /api/uploads/ into Pending / Visual similarity /
Auto-uploaded, but the endpoint was paginated at 48 — a 69-file batch
silently lost 21 entries, and the similarity sweep (which reads the same
list back after uploading) only ever saw the first page. The staged-upload
list is now unpaginated: it is a transient per-user set, still limited to
the caller's rows and the uploader role. The page takes a plain array.
Watching progress with dozens of files was also poor:
- the queue uploads three files at a time instead of strictly one at a time;
- the Uploads section now shows a batch bar and 'n/m uploaded · x%' next to
the count, so the overall progress never scrolls out of sight;
- entries are ordered active-first (uploading, queued, failed, done) so the
file being uploaded is always at the top of the grid;
- tiles are larger (4 columns at lg instead of 5);
- the header reads 'Uploading n/m…' and 'Checking n file(s) against IQDB…'
instead of a bare spinner.
Tests: staged-upload list unpaginated past 48, per-user, uploader-only
(3 new; full suite 49 green). Live-checked the bare-array response.
Follow lists were paginated at the API default of 48, but the SPA treats
them as complete sets: the tag/pool toggles read their state from page one
(so the 49th follow looked unfollowed and its spinner waited for a page that
could never contain it) and the Followed page rendered only 48 cards while
showing that as the count. Both follow endpoints are now unpaginated — they
are per-user sets and still restricted to the caller's rows — and the three
consumers take plain arrays.
Post visibility: the old J621-Django online view fetched limit=320 (e621's
maximum) while ours hard-coded 48, and fetchPostsByIds capped id batches at
100. The Online browser now has a 'Posts per page' setting (48/100/200/320)
in its sidebar, mirrored in Account -> Browsing preferences, stored per user
as e621_per_page and also used for pool loading; the id-batch cap is raised
to 320.
Tests: follow list shape/isolation (4) and preference validation/merge (3)
added; the full backend suite is 46 green. Live-checked the array response
shape and the preference bounds (200 accepted, 500 rejected).
Two stale-state bugs came from react-router reusing the detail component
between posts (parent/child links hit the same /detail/<id> route):
- the previously viewed post's download panel kept rendering, so a freshly
opened post could claim 'Downloaded to the library J-xx'. The task view is
now gated on the task's post_id as well, and DetailPage keys the detail
views per item — component state (download panel, delete confirmation,
optimizer modal) cannot survive a post change any more.
- 'Your last download for this post finished' appeared on every revisit. It
now only shows when this visit actually saw the download running (derived
state, set during render), which still reports a re-attached download
finishing while staying quiet on later visits; the 'In library' button
remains the persistent indicator.
Library detail gets the same per-item key, so its delete confirmation and
optimizer modal reset between items too.
tsc, oxlint and the build are clean (the derived-state pattern was chosen
over a ref read in render / setState-in-effect, both flagged by the linter).
The build context is the repository root and there was no .dockerignore, so
'COPY backend/ ./' swept backend/venv (327 MB), backend/media (the actual
library, 224 MB), backend/logs, backend/staticfiles and backend/.env
(SECRET_KEY plus the database password) into the backend image: 1.43 GB per
architecture, including secrets headed for the registry. The frontend build
stage also copied the host's node_modules over the fresh install.
- Root .dockerignore excludes .git, virtualenvs, __pycache__, db.sqlite3,
logs/staticfiles, .env files, media/, node_modules, dist and the deploy
runtime state (data/, tailscale-state/).
- The backend Dockerfile now asserts the context is clean (.env, venv,
media/library, db.sqlite3 all absent) before collectstatic, so a missing
ignore file fails the build instead of leaking.
- Rebuilt: backend 1.43 GB -> 876 MB ('COPY backend/' is now 268 kB),
frontend stays at 65 MB. Verified by booting the compose stack with the
new image: migrations applied, /health ok, no .env or venv inside, and
/app/media is the mounted (empty) volume; the scheduler runs too.
- Removed the stale local images that still contained the library and the
dev .env.
JakeBreath