13 Commits
Author SHA1 Message Date
JakeBreath 7ca84f4fea Point desktop updates at the Gitea release feed
CI / Backend tests (push) Successful in 2m41s
CI / Frontend build & lint (push) Successful in 25s
The updater now resolves the newest non-draft desktop-v* release through the
Gitea API at check time (J621_UPDATE_REPO, lowercase because the API path is
case-sensitive), picks the platform's latest*.yml asset and uses that release
as a generic electron-updater feed; J621_UPDATE_URL still overrides
everything. Verified against the live API: release picked, yml fetched,
artifact HEAD 200.

electron-builder's publish.url is now metadata only (still needed so the
build emits latest*.yml). Docs updated: CD release assets are the feed, the
website /desktop/ feed only matters for installs before 0.1.2.
2026-09-23 22:00:49 -05:00
JakeBreath 5f237aa2e3 Widen the header and collapse blacklist listings by default
- the header row is no longer capped at 1600px, so the nav hugs the left
  edge and the status/account controls the right; main content and footer
  keep their width
- OnlinePage's blacklist chips and the Followed page's blacklisted-tag
  cloud start collapsed behind a count toggle (N tags / N entries)
2026-09-23 21:57:48 -05:00
JakeBreath 90ba2ecff7 Bump the desktop app to 0.1.2 2026-09-23 21:40:43 -05:00
JakeBreath 73bf4f9e38 Rework the metadata modal: fullscreen, inline sections, bigger previews
- near-fullscreen panel (up to 1400px / 92vh) with two independently
  scrolling columns; the left preview uses self-start so its border hugs the
  image instead of stretching to the modal height
- J-ID matches render as a larger tile grid (was 48px rows) and IQDB
  candidates get bigger tiles too
- Link to e621 post and Custom metadata are shown inline instead of behind
  tabs, each with its own heading
- new ThumbImage component: spinner while loading and a broken-image icon on
  error, with alt text removed so a pending tile never reads as 'J-7786'
2026-09-23 21:39:48 -05:00
JakeBreath 7c2569522f Make thumbnail generation atomic and warm it on import
- write thumbnails to a .part file and os.replace() them, so concurrent
  requests never read a half-written JPEG
- a stale thumbnail plus a vanished source no longer raises through the
  request (getmtime on a missing file returned 500); it falls back cleanly
- ensure_thumbnail(item) warms the preview when a file is indexed, keeping
  image decoding out of the request path
2026-09-23 21:35:09 -05:00
JakeBreath a62195ffce Re-sign visual-match thumbnails on every detail fetch
Match rows stored a signed URL minted when the scan ran, so it aged out (or
used the pre-stable signing scheme) and the modal showed broken tiles even
after legacy signatures were fixed. Rows now carry item_id/j_id and the
detail serializer mints a fresh thumbnail URL per request; matches whose
item no longer exists are dropped.
2026-09-23 21:33:20 -05:00
JakeBreath c73a81a5f4 Fix 500 on legacy signed URLs
A TimestampSigner value is an HMAC over 'payload:timestamp', so a plain
Signer's HMAC check accepts it and the embedded timestamp then reached the
JSON decoder, raising JSONDecodeError (not BadSignature) and surfacing as a
500. That broke every stored visual-match thumbnail URL minted before the
stable scheme, so the J-ID match tiles never loaded on prod.

Detect the legacy shape by its extra separator and verify it with
TimestampSigner; malformed input returns None instead of raising.
2026-09-23 21:31:34 -05:00
JakeBreath af378e7d71 Update the roadmap for cacheable media and session-only completions
CI / Backend tests (push) Successful in 2m29s
CI / Frontend build & lint (push) Successful in 25s
2026-09-23 18:29:49 -05:00
JakeBreath 59f397a96c Show indexed uploads as session notifications, add a Failed column
The board rendered every persisted completed row with a dismiss button and a
dismiss-all that sent the whole column to the 1000-id bulk endpoint. Now that
the backend deletes completed rows and reports them through the status feed:

- the Auto-uploaded & Indexed column renders the live feed only; a reload or
  leaving the page forgets them, and there is nothing to dismiss (just a
  client-side clear)
- duplicates complete during staging and get their card immediately from the
  upload response
- failures get their own persisted column with per-card retry and discard
- bulk discard chunks requests at 500 ids, so backlogs over the server's
  1000-id cap are still removable in one action
2026-09-23 18:28:35 -05:00
JakeBreath 9ababb8b48 Stop storing completed uploads; announce them through a live feed
Every auto-matched, duplicate or manually resolved upload left a completed
TempUpload row on the board until it was dismissed by hand, so the rows
accumulated without bound and the bulk dismiss (capped at 1000 ids) failed
once there were more. The original app never stored these: they are
notifications, not records.

- complete_temp_upload now appends {filename, J-ID, resolution, post} to a
  bounded recent_completions feed on UploadRun and deletes the staged row
- staging duplicates never create a board record either; the create response
  carries the J-ID and preview so the SPA can show the card immediately
- status_payload returns the feed (newest first, signed thumbnails) for the
  live board; finalize_round counts deleted matches in processed
- resolve/link-bulk return synthetic completion payloads
- migration 0012 adds the field and purges the existing completed backlog
  (and any stray staged files) on deploy
2026-09-23 18:24:38 -05:00
JakeBreath 37085c5dac Make media URLs stable and cacheable, add real image thumbnails
Signed media URLs embedded the current second (TimestampSigner), so every
API response re-minted every raw/thumbnail/staged URL and the browser
re-downloaded each file on every poll or navigation. Responses also carried
no cache headers at all.

- sign with a plain Signer plus a bucket-quantized exp (7d TTL, 24h bucket),
  so a URL is byte-identical across responses and rotates once a day; legacy
  TimestampSigner URLs stay accepted for one release
- add a v=<md5> version parameter to library media URLs so replacing a file
  under the same J-ID (the optimize flow) busts caches exactly when needed
- serve_file now sends ETag/Last-Modified and a private Cache-Control and
  answers conditional requests with 304; library media gets max-age 6d +
  immutable, staged/similarity files 1h
- build cached 480px JPEG thumbnails for images (Pillow, keyed by MD5 under
  MEDIA_ROOT/thumbs) instead of serving full-size originals through the
  thumbnail endpoint; the library grid uses thumbnail_url for images too
2026-09-23 18:13:52 -05:00
JakeBreath ecac4cb8b4 Fix the release asset upload for names with spaces
CI / Backend tests (push) Successful in 2m4s
CI / Frontend build & lint (push) Successful in 23s
curl exit 3 (malformed URL) on 'J621 Setup 0.1.1.exe': percent-encode the
asset name in the query string.

Also replace assets instead of skipping them on re-runs: NSIS builds are
not bit-reproducible, so latest.yml/latest-linux.yml must reference the
installers produced by the same run. Existing assets are deleted by id
before the fresh upload.
2026-09-23 07:14:19 -05:00
JakeBreath 2eb7af0d41 Fix the CD wine build and add per-part toggles
CI / Backend tests (push) Successful in 2m13s
CI / Frontend build & lint (push) Successful in 20s
The Windows NSIS step failed under wine for two reasons: no X display
(nodrv_CreateWindow) and missing 32-bit libraries (failed to load
syswow64\ntdll.dll). The job now installs xvfb + wine32:i386 and runs the
desktop build under xvfb-run, with Gecko/Mono lookups disabled.

Also add `images` and `desktop` dispatch inputs so either half of the CD
can be skipped (e.g. desktop-only or images-only releases).
2026-09-23 00:06:55 -05:00
28 changed files with 1662 additions and 609 deletions
+48 -14
View File
@@ -1,15 +1,18 @@
# J621 CD — manual release workflow (Actions tab -> "Run workflow").
#
# One dispatch does everything:
# One dispatch does everything; each half can be skipped with the `images`
# and `desktop` inputs:
# * builds and pushes the backend + frontend images (multi-arch, :latest
# and :<short-sha>, GIT_HASH baked in for the version pill),
# * builds the desktop packages and attaches them (plus the update
# metadata) to the Gitea release tagged `desktop-v<package.json version>`.
#
# The live update feed (deploy/data/desktop, served by the frontend nginx at
# /desktop/) is not touched here: it is runtime state on the deploy host and
# is still published with `deploy/push_desktop.sh --no-build` from a machine
# that can reach it.
# The desktop build also attaches the update metadata (latest*.yml) to the
# release; that is the desktop updater's feed, resolved through the Gitea API
# at check time (see desktop/README.md). The older website feed
# (deploy/data/desktop, served at /desktop/) is runtime state on the deploy
# host and only needed for installs before 0.1.2; it is refreshed with
# `deploy/push_desktop.sh` from a machine that can reach the deploy host.
#
# Registry login uses a repo PAT with the minimal write:package scope (the
# Gitea registry rejects the automatic job token, go-gitea/gitea#23642);
@@ -21,6 +24,14 @@ name: CD
on:
workflow_dispatch:
inputs:
images:
description: Build and push the Docker images
required: false
default: "true"
desktop:
description: Build the desktop release
required: false
default: "true"
platforms:
description: Image platforms (comma separated)
required: false
@@ -37,6 +48,7 @@ concurrency:
jobs:
images:
name: Build & push images
if: ${{ inputs.images != 'false' }}
runs-on: ubuntu-latest
# The Gitea container registry does not accept the automatic job token
# (go-gitea/gitea#23642 is still open), so the push uses a repo PAT with
@@ -73,6 +85,7 @@ jobs:
desktop:
name: Desktop release
if: ${{ inputs.desktop != 'false' }}
runs-on: ubuntu-latest
# Creating the release and uploading its assets uses the automatic job
# token, so it needs write access to the repository's releases.
@@ -90,7 +103,12 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends fakeroot libarchive-tools
if [ "${{ inputs.windows }}" = "true" ]; then
sudo apt-get install -y --no-install-recommends wine
# electron-builder runs the 32-bit NSIS installer under wine to
# build the uninstaller: that needs a virtual display (Xvfb) and
# 32-bit wine libraries.
sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install -y --no-install-recommends xvfb wine wine32:i386
fi
- name: Install frontend + desktop dependencies
@@ -99,9 +117,12 @@ jobs:
npm --prefix desktop ci --no-audit --no-fund
- name: Build desktop packages
env:
# Keep wine from trying to fetch Gecko/Mono on first run.
WINEDLLOVERRIDES: mscoree,mshtml=
run: |
if [ "${{ inputs.windows }}" = "true" ]; then
./deploy/build_desktop.sh --all
xvfb-run -a ./deploy/build_desktop.sh --all
else
./deploy/build_desktop.sh --linux
fi
@@ -143,8 +164,7 @@ jobs:
fi
EXISTING="$(curl -sf -H "$AUTH" "$API/releases/$RELEASE_ID/assets" \
| python3 -c 'import json,sys; print("\n".join(a["name"] for a in json.load(sys.stdin)))' \
|| true)"
|| echo '[]')"
for FILE in desktop/release/*"$VERSION"*.deb \
desktop/release/*"$VERSION"*.pkg.tar.zst \
desktop/release/latest-linux.yml \
@@ -153,11 +173,25 @@ jobs:
desktop/release/*"$VERSION"*.exe.blockmap; do
[ -e "$FILE" ] || continue
NAME="$(basename "$FILE")"
case "$EXISTING" in
*"$NAME"*) echo " already attached: $NAME"; continue ;;
esac
echo " attaching $NAME"
# Replace the asset when it is already there: latest*.yml must
# reference the installers built by *this* run (NSIS builds are
# not bit-reproducible), so old copies are deleted first.
ASSET_ID="$(printf '%s' "$EXISTING" | python3 -c '
import json, sys
name = sys.argv[1]
print(next((str(a["id"]) for a in json.load(sys.stdin) if a["name"] == name), ""))
' "$NAME")"
if [ -n "$ASSET_ID" ]; then
echo " replacing $NAME"
curl -sf -X DELETE -H "$AUTH" \
"$API/releases/$RELEASE_ID/assets/$ASSET_ID" >/dev/null
else
echo " attaching $NAME"
fi
# Names like "J621 Setup 0.1.1.exe" contain spaces: encode them
# or curl refuses the URL (exit 3).
ENCODED="$(python3 -c 'import sys, urllib.parse; print(urllib.parse.quote(sys.argv[1]))' "$NAME")"
curl -sf -X POST -H "$AUTH" -H "Content-Type: application/octet-stream" \
--data-binary @"$FILE" "$API/releases/$RELEASE_ID/assets?name=$NAME" >/dev/null
--data-binary @"$FILE" "$API/releases/$RELEASE_ID/assets?name=$ENCODED" >/dev/null
done
echo "Release: ${{ github.server_url }}/${{ github.repository }}/releases/tag/$TAG"
+4 -3
View File
@@ -56,9 +56,10 @@ Project constraints (do not regress):
`desktop-v<version>`). Jobs run on the user-scoped runners: `ubuntu-latest`
on nitro-ci, `desktop` on msi-mortar-ci. Do not add actions/cache
(`cache: pip`/`npm`) to these workflows: Gitea's cache service hangs the job
on restore/save. The live desktop update feed (deploy/data/desktop) is still
published with `deploy/push_desktop.sh --no-build` from a machine with SSH
to the deploy host — CI has no key for that.
on restore/save. Desktop updates read the release assets (latest*.yml) from
the Gitea API at check time; the older website feed (deploy/data/desktop)
only matters for installs before 0.1.2 and is refreshed with
`deploy/push_desktop.sh` from a machine with SSH to the deploy host.
- Security/permission tests live in backend/apps/core/tests and need a
one-time grant: GRANT ALL ON `test_j621`.* TO 'j621'@'%';
+14 -3
View File
@@ -20,6 +20,13 @@ they land.
- [x] Tag cloud in the sidebar (click to search, hidden from guests for
blacklisted items)
- [x] Status filter (matched / not_found / deleted / custom / unknown)
- [x] **Cacheable media serving**
- [x] Stable signed URLs (7 d TTL, 24 h rotation) plus a `v=<md5>` cache
buster, so replacing a file under the same J-ID invalidates it
- [x] ETag/Last-Modified and a private `Cache-Control` (immutable for
versioned media), conditional 304s
- [x] Real 480 px image thumbnails (cached by MD5) instead of serving
full-size originals through the thumbnail endpoint
- [x] **Random image** endpoint and page: `GET /api/random/` (aliases
`/random`, `/random/`) with `rating=s,q,e` filters; fastfetch mode
@@ -104,12 +111,16 @@ Files now stage first and are resolved before entering the library.
shell indicator; rate-limited runs retry with backoff
- [x] Perceptual-hash comparison against the library, loaded once per batch
- [x] IQDB candidates stored with one batched enrichment request
- [x] Indexed uploads are announced through a bounded per-run feed and the
staged row is deleted; nothing persists on the board to dismiss
- [x] **Upload UI**
- [x] Three-column board: Pending & Unmatched / Visual Similarity Detected /
Auto-uploaded & Indexed, private per user (staff included)
- [x] Four-column board: Pending & Unmatched / Visual Similarity Detected /
Auto-uploaded & Indexed (session feed) / Failed, private per user
(staff included)
- [x] Metadata modal (link to e621 post, IQDB candidates, custom metadata)
- [x] Per-file progress plus background pipeline status
- [x] Bulk actions: bulk rate, discard all (pending/visual), dismiss all
- [x] Bulk actions: bulk rate and discard all (chunked past the API's
1000-id cap)
## 4. Staff tools
+36 -22
View File
@@ -36,6 +36,7 @@ from apps.library.models import (
TempUpload,
)
from apps.library.services import MEDIA_FILE_SALT
from apps.library.signing_urls import sign_payload
User = get_user_model()
@@ -122,21 +123,15 @@ class SecurityTestCase(TestCase):
return item
def old_signature(self, item, action="raw", age=3 * 86400):
"""A valid signature minted `age` seconds ago."""
real_time = signing.time
class Backdated:
def time(self):
return real_time.time() - age
try:
signing.time = Backdated()
return signing.dumps(
{"item": item.id, "user": self.users["sec-uploader"].id, "action": action},
salt=MEDIA_FILE_SALT,
)
finally:
signing.time = real_time
"""A signed media URL whose expiry is `age` seconds in the past."""
return signing.Signer(salt=MEDIA_FILE_SALT).sign_object(
{
"item": item.id,
"user": self.users["sec-uploader"].id,
"action": action,
"exp": int(time.time()) - age,
}
)
class GuestVisibilityTests(SecurityTestCase):
@@ -183,9 +178,9 @@ class GuestVisibilityTests(SecurityTestCase):
def test_authenticated_users_and_signed_urls_see_protected_items(self):
uploader = self.client_for("sec-uploader")
self.assertEqual(uploader.get(f"/api/files/J-{self.hidden.id}/").status_code, 200)
signed = signing.dumps(
signed = sign_payload(
{"item": self.hidden.id, "user": self.users["sec-uploader"].id, "action": "raw"},
salt=MEDIA_FILE_SALT,
MEDIA_FILE_SALT,
)
self.assertEqual(
self.guest.get(f"/api/files/J-{self.hidden.id}/raw/?sig={signed}").status_code,
@@ -193,9 +188,9 @@ class GuestVisibilityTests(SecurityTestCase):
)
def test_signature_integrity(self):
signed = signing.dumps(
signed = sign_payload(
{"item": self.hidden.id, "user": self.users["sec-uploader"].id, "action": "raw"},
salt=MEDIA_FILE_SALT,
MEDIA_FILE_SALT,
)
raw = f"/api/files/J-{self.hidden.id}/raw/"
thumbnail = f"/api/files/J-{self.hidden.id}/thumbnail/"
@@ -203,10 +198,29 @@ class GuestVisibilityTests(SecurityTestCase):
self.assertEqual(self.guest.get(f"{raw}?sig={signed[:-4]}AAAA").status_code, 404)
# Valid signature, wrong action.
self.assertEqual(self.guest.get(f"{thumbnail}?sig={signed}").status_code, 404)
# Expired signature (minted three days ago).
# Expired signature (expiry three days ago).
expired = self.old_signature(self.hidden)
self.assertEqual(self.guest.get(f"{raw}?sig={expired}").status_code, 404)
def test_signed_media_urls_are_stable_and_versioned(self):
"""The same item must keep the same URL across responses.
A per-second signature made browsers re-download every image on every
poll; the MD5 version parameter busts caches only when the file itself
changes (the optimize flow rewrites files under the same J-ID).
"""
item = self.visible
first = services.signed_media_url(item, self.users["sec-uploader"])
time.sleep(1.1)
second = services.signed_media_url(item, self.users["sec-uploader"])
self.assertEqual(first, second)
self.assertIn(f"v={item.md5}", first)
MediaItem.objects.filter(pk=item.pk).update(md5="b" * 32)
item.refresh_from_db()
self.assertNotEqual(
services.signed_media_url(item, self.users["sec-uploader"]), first
)
class RoleBoundaryTests(SecurityTestCase):
def test_non_uploader_is_read_only(self):
@@ -451,9 +465,9 @@ class ThrottleTests(SecurityTestCase):
md5=hashlib.md5(b"throttle-temp").hexdigest(),
size=6,
)
signature = signing.dumps(
signature = sign_payload(
{"temp": str(temp.id), "user": self.users["sec-uploader"].id},
salt=services.UPLOAD_FILE_SALT,
services.UPLOAD_FILE_SALT,
)
url = f"/api/uploads/{temp.id}/file/?sig={signature}"
codes = {self.guest.get(url).status_code for _ in range(150)}
@@ -0,0 +1,34 @@
# Generated by Django 6.1.1 on 2026-09-23
from django.db import migrations, models
def purge_completed_uploads(apps, schema_editor):
"""Completed staged uploads are notifications, not records.
The board used to keep every auto-uploaded/indexed row until it was
dismissed by hand, so they accumulated without bound (and bulk dismissal
is capped at 1000 ids). Completions now live in a small per-run feed, so
the old rows are removed here.
"""
TempUpload = apps.get_model("library", "TempUpload")
for temp in TempUpload.objects.filter(status="completed").iterator():
if temp.file:
temp.file.delete(save=False)
temp.delete()
class Migration(migrations.Migration):
dependencies = [
("library", "0011_upload_pipeline"),
]
operations = [
migrations.AddField(
model_name="uploadrun",
name="recent_completions",
field=models.JSONField(blank=True, default=list),
),
migrations.RunPython(purge_completed_uploads, migrations.RunPython.noop),
]
+5
View File
@@ -229,6 +229,11 @@ class UploadRun(models.Model):
matched = models.IntegerField(default=0)
failed = models.IntegerField(default=0)
error = models.TextField(blank=True, default="")
# Rolling feed of recent completions for the upload board. Completed
# staged uploads are deleted as soon as they are indexed; this only tells
# the live page "filename -> J-x" while it watches. Bounded, never
# dismissed, and ignored by fresh page loads.
recent_completions = models.JSONField(default=list, blank=True)
started_at = models.DateTimeField(null=True, blank=True)
updated_at = models.DateTimeField(auto_now=True)
+57 -5
View File
@@ -3,7 +3,6 @@ from datetime import timedelta
from pathlib import Path
from django.conf import settings
from django.core import signing
from rest_framework import serializers
from .models import (
@@ -20,6 +19,7 @@ from .services import (
VIDEO_EXTENSIONS,
signed_media_url,
)
from .signing_urls import sign_payload
class MediaLocationSerializer(serializers.ModelSerializer):
@@ -150,6 +150,7 @@ class TempUploadSerializer(serializers.ModelSerializer):
iqdb_checked = serializers.SerializerMethodField()
processing = serializers.SerializerMethodField()
similar_count = serializers.SerializerMethodField()
visual_matches = serializers.SerializerMethodField()
class Meta:
model = TempUpload
@@ -199,9 +200,9 @@ class TempUploadSerializer(serializers.ModelSerializer):
user = self._request_user()
if user is None:
return None
signature = signing.dumps(
signature = sign_payload(
{"temp": str(obj.id), "user": user.id},
salt=UPLOAD_FILE_SALT,
UPLOAD_FILE_SALT,
)
url = f"/api/uploads/{obj.id}/file/?sig={signature}"
request = self.context.get("request")
@@ -242,6 +243,57 @@ class TempUploadSerializer(serializers.ModelSerializer):
def get_similar_count(self, obj):
return len(obj.iqdb_data or []) + len(obj.visual_matches or [])
@staticmethod
def _visual_item_id(entry):
if not isinstance(entry, dict):
return None
item_id = entry.get("item_id")
if item_id is None:
j_id = str(entry.get("j_id") or "")
if j_id.upper().startswith("J-"):
j_id = j_id[2:]
item_id = j_id if j_id.isdigit() else None
try:
return int(item_id)
except (TypeError, ValueError):
return None
def get_visual_matches(self, obj):
"""Rebuild match rows with fresh signed thumbnail URLs.
Storing the signed URL meant it aged out (or came from an older
signing scheme) and the "Already in your library" grid showed broken
tiles. The stored rows only carry the item reference now.
"""
entries = obj.visual_matches or []
if not entries:
return entries
wanted = {}
for entry in entries:
item_id = self._visual_item_id(entry)
if item_id is not None:
wanted[item_id] = None
items = MediaItem.objects.in_bulk(list(wanted))
user = self._request_user()
request = self.context.get("request")
matches = []
for entry in entries:
item_id = self._visual_item_id(entry)
item = items.get(item_id) if item_id is not None else None
if item is None:
continue
matches.append(
{
"j_id": f"J-{item.id}",
"filename": entry.get("filename") or item.md5,
"similarity": entry.get("similarity"),
"thumbnail_url": signed_media_url(
item, user, "thumbnail", request=request
),
}
)
return matches
class TempUploadListSerializer(TempUploadSerializer):
"""Compact staged-upload row for the board and the status polling.
@@ -339,9 +391,9 @@ class SimilarityCheckSerializer(serializers.ModelSerializer):
user = self._request_user()
if user is None or not obj.file:
return None
signature = signing.dumps(
signature = sign_payload(
{"check": str(obj.id), "user": user.id},
salt=UPLOAD_FILE_SALT,
UPLOAD_FILE_SALT,
)
url = f"/api/similarity/{obj.id}/file/?sig={signature}"
request = self.context.get("request")
+138 -16
View File
@@ -6,16 +6,21 @@ import os
import re
import shutil
import subprocess
import uuid
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import urlencode
import imagehash
from django.conf import settings
from django.core import signing
from django.http import FileResponse, Http404, HttpResponse
from django.utils.cache import get_conditional_response
from django.utils.http import http_date
from django.utils.text import get_valid_filename
from PIL import Image
from PIL import Image, ImageOps
from .models import MediaItem, MediaLocation
from .signing_urls import sign_payload
logger = logging.getLogger(__name__)
@@ -37,6 +42,11 @@ UPLOAD_FILE_SALT = "j621.upload-file"
MEDIA_FILE_SALT = "j621.media-file"
CHUNK_SIZE = 1024 * 1024
RANGE_RE = re.compile(r"bytes=(\d*)-(\d*)$")
# Versioned media URLs are immutable, so they may sit in the browser cache for
# as long as the signature is guaranteed to stay valid (7 days).
MEDIA_CACHE_SECONDS = 6 * 86400
# Staged uploads and similarity files can be deleted at any moment.
TEMP_CACHE_SECONDS = 3600
def compute_md5(path):
@@ -78,14 +88,24 @@ def signed_media_url(item, user, action="raw", request=None):
With a ``request`` the URL is absolute, so the SPA also works when it is
served from a different origin; without one it stays relative.
The ``v`` parameter is the item's MD5: it busts the browser cache exactly
when the file is replaced (the optimize flow rewrites files under the same
J-ID), which is what lets the URL be cached for days instead of re-minted
on every response.
"""
path = f"/api/files/J-{item.id}/{action}/"
params = {}
if user is not None and getattr(user, "is_authenticated", False):
signature = signing.dumps(
{"item": item.id, "user": user.id, "action": action},
salt=MEDIA_FILE_SALT,
)
path = f"{path}?sig={signature}"
params = {
"v": item.md5,
"sig": sign_payload(
{"item": item.id, "user": user.id, "action": action},
MEDIA_FILE_SALT,
),
}
if params:
path = f"{path}?{urlencode(params)}"
if request is None:
return path
return request.build_absolute_uri(path)
@@ -207,12 +227,36 @@ class RangeFileWrapper:
self.file.close()
def serve_file(request, path, download=False):
"""Serve a file with HTTP range support (needed for video seeking)."""
def _apply_cache_headers(response, cache_control, etag, mtime):
response["Cache-Control"] = cache_control
response["ETag"] = etag
response["Last-Modified"] = http_date(mtime)
return response
def serve_file(request, path, download=False, *, max_age=TEMP_CACHE_SECONDS, immutable=False):
"""Serve a file with HTTP range support (needed for video seeking).
Responses carry validators (ETag/Last-Modified) and a private
``Cache-Control`` so browsers reuse media instead of re-downloading it on
every SPA poll. ``max_age``/``immutable`` are chosen by the caller: versioned
library media can be cached hard, staged files only briefly.
"""
path = Path(path)
if not path.is_file():
raise Http404
size = path.stat().st_size
stat = path.stat()
size = stat.st_size
etag = f'W/"{size:x}-{stat.st_mtime_ns:x}"'
last_modified = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc)
conditional = get_conditional_response(
request, etag=etag, last_modified=last_modified
)
if conditional is not None:
return conditional
cache_control = f"private, max-age={int(max_age)}"
if immutable:
cache_control += ", immutable"
content_type = mimetypes.guess_type(str(path))[0] or "application/octet-stream"
range_header = request.headers.get("Range", "").strip()
if range_header:
@@ -240,7 +284,9 @@ def serve_file(request, path, download=False):
response["Content-Length"] = str(length)
response["Content-Range"] = f"bytes {start}-{end}/{size}"
response["Accept-Ranges"] = "bytes"
return response
return _apply_cache_headers(
response, cache_control, etag, stat.st_mtime
)
response = FileResponse(
open(path, "rb"),
content_type=content_type,
@@ -248,7 +294,7 @@ def serve_file(request, path, download=False):
filename=path.name,
)
response["Accept-Ranges"] = "bytes"
return response
return _apply_cache_headers(response, cache_control, etag, stat.st_mtime)
class DownloadCancelled(Exception):
@@ -436,15 +482,38 @@ def sanitize_iqdb_results(results):
return cleaned
def _thumbnail_is_fresh(target, path):
"""True when the cached thumbnail exists and is at least as new as source."""
try:
stat = target.stat()
if stat.st_size <= 0:
return False
return stat.st_mtime >= os.path.getmtime(path)
except OSError:
return False
def _thumbs_dir():
thumbs_dir = Path(settings.MEDIA_ROOT) / "thumbs"
thumbs_dir.mkdir(parents=True, exist_ok=True)
return thumbs_dir
def generate_video_thumbnail(md5, path):
"""Extract a JPEG thumbnail from a video, cached under MEDIA_ROOT/thumbs."""
if not shutil.which("ffmpeg"):
return None
thumbs_dir = Path(settings.MEDIA_ROOT) / "thumbs"
thumbs_dir.mkdir(parents=True, exist_ok=True)
try:
thumbs_dir = _thumbs_dir()
except OSError:
logger.exception("Could not create the thumbnail folder")
return None
target = thumbs_dir / f"{md5}.jpg"
if target.exists() and target.stat().st_mtime >= os.path.getmtime(path):
if _thumbnail_is_fresh(target, path):
return target
# Write beside the target and move it into place, so a concurrent request
# can never read a half-written JPEG.
temp = thumbs_dir / f".{md5}.{uuid.uuid4().hex}.part.jpg"
command = [
"ffmpeg",
"-y",
@@ -458,10 +527,63 @@ def generate_video_thumbnail(md5, path):
"scale=480:-2",
"-loglevel",
"error",
str(target),
str(temp),
]
try:
subprocess.run(command, check=True, capture_output=True, timeout=60)
os.replace(temp, target)
except (subprocess.SubprocessError, OSError):
logger.exception("Could not build a video thumbnail for %s", path)
temp.unlink(missing_ok=True)
return None
return target if target.exists() else None
def generate_image_thumbnail(md5, path):
"""Downscale an image, cached under MEDIA_ROOT/thumbs like video thumbs.
The thumbnail action used to serve full-size originals for images; a
cached 480px JPEG keeps the library grid light without touching the
original file. Returns ``None`` when the source is missing or Pillow
cannot decode it, so callers can fall back to the original.
"""
try:
thumbs_dir = _thumbs_dir()
except OSError:
logger.exception("Could not create the thumbnail folder")
return None
target = thumbs_dir / f"{md5}.jpg"
if _thumbnail_is_fresh(target, path):
return target
temp = thumbs_dir / f".{md5}.{uuid.uuid4().hex}.part.jpg"
try:
with Image.open(path) as image:
# Animated formats: the first frame is the preview.
image.seek(0)
frame = ImageOps.exif_transpose(image) or image
frame = frame.convert("RGB")
frame.thumbnail((480, 480))
frame.save(temp, "JPEG", quality=82, optimize=True)
os.replace(temp, target)
except Exception: # noqa: BLE001 - previews must never break serving
logger.exception("Could not build an image thumbnail for %s", path)
temp.unlink(missing_ok=True)
return None
return target if target.exists() else None
def ensure_thumbnail(item):
"""Generate an item's cached thumbnail if it is missing or stale.
Warming thumbnails when a file is indexed keeps image decoding out of the
request path, where the upload pipeline's hashing used to starve it.
"""
location = item.locations.first()
if location is None:
return None
path = Path(location.path)
if not path.is_file():
return None
if path.suffix.lower() in VIDEO_EXTENSIONS:
return generate_video_thumbnail(item.md5, path)
return generate_image_thumbnail(item.md5, path)
+64
View File
@@ -0,0 +1,64 @@
"""Stable, expiring signatures for media URLs.
The SPA loads media with ``<img>``/``<video>`` tags, which cannot send the
API's ``Authorization`` header, so those URLs carry a signature instead. The
signature has to be *stable*: a URL that changes on every response makes the
browser treat every refetch as a new resource and re-download the file.
URLs are signed with a plain ``Signer`` (no per-second timestamp) plus an
explicit ``exp`` claim quantized to a bucket, so every request inside a bucket
mints the exact same URL. The URL rotates once per bucket and is valid for at
least ``URL_TTL_SECONDS`` and at most ``URL_TTL_SECONDS + URL_BUCKET_SECONDS``.
"""
import time
from django.core import signing
URL_TTL_SECONDS = 7 * 86400
URL_BUCKET_SECONDS = 24 * 3600
_BUCKETS = URL_TTL_SECONDS // URL_BUCKET_SECONDS
def _expiry(now=None):
current = time.time() if now is None else now
bucket = int(current // URL_BUCKET_SECONDS)
return (bucket + _BUCKETS + 1) * URL_BUCKET_SECONDS
def sign_payload(payload, salt, now=None):
"""Sign a payload with a stable, bucket-quantized expiry."""
return signing.Signer(salt=salt).sign_object(
{**payload, "exp": _expiry(now)}
)
def load_payload(signature, salt, legacy_max_age=86400):
"""Verify a signed payload; ``None`` when missing, tampered with or expired.
Legacy ``TimestampSigner`` values are still accepted for one release.
Detect them by their extra separator (``payload:timestamp:signature``):
a plain ``Signer`` accepts the HMAC a ``TimestampSigner`` computed over
``payload:timestamp`` and then chokes on the embedded timestamp while
decoding the JSON payload, which used to surface as a 500.
"""
if not signature:
return None
if signature.count(":") >= 2:
try:
return signing.TimestampSigner(salt=salt).unsign_object(
signature, max_age=legacy_max_age
)
except (signing.BadSignature, ValueError):
return None
try:
data = signing.Signer(salt=salt).unsign_object(signature)
except (signing.BadSignature, ValueError):
return None
if not isinstance(data, dict):
return None
try:
expired = int(data.get("exp", 0)) < time.time()
except (TypeError, ValueError):
return None
return None if expired else data
+2 -9
View File
@@ -11,7 +11,6 @@ from datetime import timedelta
from pathlib import Path
from django.conf import settings
from django.core import signing
from django.utils import timezone
from rest_framework import mixins, status, viewsets
from rest_framework.decorators import action
@@ -22,6 +21,7 @@ from rest_framework.response import Response
from . import services
from .models import MediaItem, SimilarityCheck
from .serializers import SimilarityCheckSerializer
from .signing_urls import load_payload
from .tools import item_brief
from .uploads import find_library_matches
@@ -149,14 +149,7 @@ class SimilarityCheckViewSet(
check = self.get_queryset().filter(pk=pk).first()
else:
signature = request.query_params.get("sig")
payload = None
if signature:
try:
payload = signing.loads(
signature, salt=services.UPLOAD_FILE_SALT, max_age=86400
)
except signing.BadSignature:
payload = None
payload = load_payload(signature, services.UPLOAD_FILE_SALT) if signature else None
if payload and payload.get("check") == str(pk):
check = SimilarityCheck.objects.filter(pk=pk).first()
if check is None or not check.file:
@@ -0,0 +1,188 @@
"""Signed media URLs must be stable, versioned and cacheable.
Regression: signatures embedded the current second, so every API response
re-minted every URL and browsers re-downloaded each image on every poll; the
file responses also carried no cache headers at all.
"""
import base64
import hashlib
import io
import shutil
import tempfile
import time
from pathlib import Path
from unittest import mock
from django.contrib.auth import get_user_model
from django.core import signing
from django.core.files.uploadedfile import SimpleUploadedFile
from django.test import Client, TestCase, override_settings
from PIL import Image
from rest_framework.authtoken.models import Token
from apps.library import services
from apps.library.models import MediaItem, MediaLocation, TempUpload
from apps.library.signing_urls import load_payload, sign_payload
User = get_user_model()
TINY_PNG = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="
)
def png_bytes(width=1200, height=800, color=(20, 120, 200)):
buffer = io.BytesIO()
Image.new("RGB", (width, height), color).save(buffer, format="PNG")
return buffer.getvalue()
class MediaCacheTests(TestCase):
@classmethod
def setUpClass(cls):
super().setUpClass()
cls._tmp = tempfile.mkdtemp(prefix="j621-cache-")
cls._media = Path(cls._tmp) / "media"
cls._watched = cls._media / "library"
cls._watched.mkdir(parents=True, exist_ok=True)
cls._settings = override_settings(
MEDIA_ROOT=str(cls._media), WATCHED_FOLDER=str(cls._watched)
)
cls._settings.enable()
@classmethod
def tearDownClass(cls):
cls._settings.disable()
shutil.rmtree(cls._tmp, ignore_errors=True)
super().tearDownClass()
def setUp(self):
self.user = User.objects.create_user(
username="cache-uploader", password="cache-pass-123456"
)
self.user.role = "uploader"
self.user.save(update_fields=["role"])
self.token = Token.objects.create(user=self.user).key
payload = png_bytes()
path = self._watched / "cache-image.png"
path.write_bytes(payload)
self.item = MediaItem.objects.create(
md5=hashlib.md5(payload).hexdigest(), size=len(payload)
)
MediaLocation.objects.create(
item=self.item, path=str(path), rel_path=path.name, mtime=time.time()
)
self.client = Client()
def signed(self, action):
return {
"sig": sign_payload(
{"item": self.item.id, "user": self.user.id, "action": action},
services.MEDIA_FILE_SALT,
)
}
def test_media_response_carries_cache_headers(self):
response = self.client.get(
f"/api/files/J-{self.item.id}/raw/", self.signed("raw")
)
self.assertEqual(response.status_code, 200)
self.assertIn("private", response["Cache-Control"])
self.assertIn(
f"max-age={services.MEDIA_CACHE_SECONDS}", response["Cache-Control"]
)
self.assertIn("immutable", response["Cache-Control"])
self.assertTrue(response["ETag"])
self.assertTrue(response["Last-Modified"])
def test_media_revalidation_returns_304(self):
url = f"/api/files/J-{self.item.id}/raw/"
first = self.client.get(url, self.signed("raw"))
second = self.client.get(
url, self.signed("raw"), HTTP_IF_NONE_MATCH=first["ETag"]
)
self.assertEqual(second.status_code, 304)
self.assertEqual(second.content, b"")
def test_image_thumbnail_is_generated_and_reused(self):
url = f"/api/files/J-{self.item.id}/thumbnail/"
response = self.client.get(url, self.signed("thumbnail"))
self.assertEqual(response.status_code, 200)
self.assertEqual(response["Content-Type"], "image/jpeg")
thumb = self._media / "thumbs" / f"{self.item.md5}.jpg"
self.assertTrue(thumb.exists())
with Image.open(thumb) as image:
self.assertLessEqual(max(image.size), 480)
before = thumb.stat().st_mtime_ns
self.client.get(url, self.signed("thumbnail"))
self.assertEqual(thumb.stat().st_mtime_ns, before)
def test_ensure_thumbnail_reuses_the_cache(self):
first = services.ensure_thumbnail(self.item)
self.assertIsNotNone(first)
self.assertTrue(first.exists())
mtime = first.stat().st_mtime_ns
second = services.ensure_thumbnail(self.item)
self.assertEqual(second, first)
self.assertEqual(second.stat().st_mtime_ns, mtime)
def test_thumbnail_of_a_missing_source_does_not_error(self):
"""Regression: getmtime() on a vanished source used to raise a 500."""
thumbs = self._media / "thumbs"
thumbs.mkdir(parents=True, exist_ok=True)
(thumbs / f"{self.item.md5}.jpg").write_bytes(b"stale")
Path(self.item.locations.first().path).unlink()
self.assertIsNone(services.ensure_thumbnail(self.item))
response = self.client.get(
f"/api/files/J-{self.item.id}/thumbnail/", self.signed("thumbnail")
)
self.assertEqual(response.status_code, 404)
def test_staged_files_cache_briefly(self):
temp = TempUpload.objects.create(
user=self.user,
file=SimpleUploadedFile("staged.png", TINY_PNG, content_type="image/png"),
original_filename="staged.png",
md5=hashlib.md5(b"staged").hexdigest(),
size=len(TINY_PNG),
)
signature = sign_payload(
{"temp": str(temp.id), "user": self.user.id}, services.UPLOAD_FILE_SALT
)
response = self.client.get(
f"/api/uploads/{temp.id}/file/", {"sig": signature}
)
self.assertEqual(response.status_code, 200)
self.assertIn(
f"max-age={services.TEMP_CACHE_SECONDS}", response["Cache-Control"]
)
self.assertNotIn("immutable", response["Cache-Control"])
def test_legacy_timestamp_signatures_are_accepted(self):
"""URLs minted before the stable scheme must not 500.
A TimestampSigner HMAC also passes a plain Signer's check, so the
embedded timestamp used to reach the JSON decoder and blow up.
"""
payload = {"item": self.item.id, "user": self.user.id, "action": "raw"}
legacy = signing.dumps(payload, salt=services.MEDIA_FILE_SALT)
self.assertEqual(
load_payload(legacy, services.MEDIA_FILE_SALT)["item"], self.item.id
)
response = self.client.get(
f"/api/files/J-{self.item.id}/raw/", {"sig": legacy}
)
self.assertEqual(response.status_code, 200)
def test_expired_and_malformed_signatures_return_none(self):
payload = {"item": self.item.id, "user": self.user.id, "action": "raw"}
with mock.patch.object(signing, "time") as clock:
clock.time.return_value = time.time() - 3 * 86400
expired = signing.dumps(payload, salt=services.MEDIA_FILE_SALT)
self.assertIsNone(load_payload(expired, services.MEDIA_FILE_SALT))
self.assertIsNone(load_payload("bogus", services.MEDIA_FILE_SALT))
self.assertIsNone(load_payload("a:b", services.MEDIA_FILE_SALT))
self.assertIsNone(load_payload("", services.MEDIA_FILE_SALT))
+112 -21
View File
@@ -170,12 +170,18 @@ class StagedUploadWorkflowTests(TestCase):
self.assertEqual(len(body["resolved"]), 2)
self.assertEqual(body["errors"], [])
for temp in (first, second):
temp.refresh_from_db()
self.assertEqual(temp.status, TempUpload.STATUS_COMPLETED)
self.assertIsNotNone(temp.library_item_id)
self.assertEqual(temp.library_item.rating, "q")
self.assertEqual(temp.library_item.uploaded_by_id, self.uploader.id)
# Completed uploads are notifications now: the staged rows are gone
# and the live feed carries the filename -> J-ID mapping.
self.assertFalse(
TempUpload.objects.filter(pk__in=[first.id, second.id]).exists()
)
feed = UploadRun.objects.get(user=self.uploader).recent_completions
self.assertEqual(
{entry["filename"] for entry in feed}, {"one.png", "two.png"}
)
for item in MediaItem.objects.all():
self.assertEqual(item.rating, "q")
self.assertEqual(item.uploaded_by_id, self.uploader.id)
untouched.refresh_from_db()
self.assertEqual(untouched.status, TempUpload.STATUS_PENDING)
@@ -231,6 +237,25 @@ class StagedUploadWorkflowTests(TestCase):
self.assertEqual(response.status_code, 200)
return seed
def test_duplicate_upload_returns_a_completion_without_a_record(self):
client = self.api_client(self.uploader)
first = self.upload_via_api(client, "same.png")
self.assertEqual(first.status_code, 201)
# Index the first upload so the second one is byte-identical.
seed = TempUpload.objects.get(pk=first.json()["temp_id"])
self.resolve_bulk(client, [seed.id], "s")
response = self.upload_via_api(client, "same.png")
self.assertEqual(response.status_code, 201)
body = response.json()
self.assertEqual(body["status"], TempUpload.STATUS_COMPLETED)
self.assertEqual(body["resolution"], TempUpload.RESOLUTION_DUPLICATE)
self.assertTrue(body["library_j_id"].startswith("J-"))
# Duplicates never become board records; the feed announces them.
self.assertFalse(TempUpload.objects.filter(pk=body["temp_id"]).exists())
feed = UploadRun.objects.get(user=self.uploader).recent_completions
self.assertEqual(feed[-1]["filename"], "same.png")
def test_upload_defers_visual_similarity_to_its_phase(self):
client = self.api_client(self.uploader)
self.seed_library_item(client, "seed-defer")
@@ -262,6 +287,37 @@ class StagedUploadWorkflowTests(TestCase):
self.assertEqual(body["visual_matches"], [])
self.assertEqual(body["status"], TempUpload.STATUS_PENDING)
def test_detail_resigns_stored_visual_match_urls(self):
"""Stored matches carry only the item reference; URLs are re-minted.
Embedding the signed URL meant it expired (or used an older signing
scheme) and the modal showed alt text instead of thumbnails.
"""
client = self.api_client(self.uploader)
self.seed_library_item(client, "seed-resign")
item = MediaItem.objects.get()
temp = self.make_temp(self.uploader, "resign")
TempUpload.objects.filter(pk=temp.pk).update(
visual_matches=[
{
"item_id": item.id,
"j_id": f"J-{item.id}",
"filename": "seed-resign.png",
"similarity": 96.5,
"thumbnail_url": "/api/files/J-x/thumbnail/?sig=stale",
},
{"item_id": 999999, "j_id": "J-999999", "filename": "gone.png"},
]
)
body = client.get(f"/api/uploads/{temp.id}/").json()
self.assertEqual(len(body["visual_matches"]), 1)
match = body["visual_matches"][0]
self.assertEqual(match["j_id"], f"J-{item.id}")
self.assertEqual(match["similarity"], 96.5)
self.assertNotIn("stale", match["thumbnail_url"])
self.assertIn("/thumbnail/", match["thumbnail_url"])
self.assertIn(f"v={item.md5}", match["thumbnail_url"])
def test_visual_match_phase_rejects_completed_uploads(self):
client = self.api_client(self.uploader)
temp = self.make_temp(
@@ -312,14 +368,23 @@ class StagedUploadWorkflowTests(TestCase):
body = response.json()
self.assertEqual(body["errors"], [])
self.assertEqual(len(body["updated"]), 2)
for temp, post_id in ((first, 900001), (second, 900002)):
temp.refresh_from_db()
self.assertEqual(temp.status, TempUpload.STATUS_COMPLETED)
self.assertEqual(temp.library_item_id is not None, True)
self.assertEqual(temp.e621_post_id, post_id)
self.assertEqual(temp.resolution, TempUpload.RESOLUTION_AUTO_MD5)
self.assertEqual(temp.library_item.e621_post_id, post_id)
self.assertEqual(MediaItem.objects.count(), 2)
for entry, post_id in zip(body["updated"], (900001, 900002)):
self.assertEqual(entry["resolution"], TempUpload.RESOLUTION_AUTO_MD5)
self.assertEqual(entry["e621_post_id"], post_id)
self.assertTrue(entry["library_j_id"].startswith("J-"))
# Indexed uploads no longer leave a board record; the completion feed
# carries them for the live page instead.
self.assertFalse(TempUpload.objects.exists())
self.assertEqual(
{item.e621_post_id for item in MediaItem.objects.all()},
{900001, 900002},
)
feed = UploadRun.objects.get(user=self.uploader).recent_completions
self.assertEqual(len(feed), 2)
self.assertEqual(
{entry["filename"] for entry in feed},
{"bulk-link-1.png", "bulk-link-2.png"},
)
class IqdbRecordingTests(TestCase):
@@ -461,6 +526,7 @@ class UploadPipelineTests(TestCase):
def test_md5_match_auto_imports_the_file(self):
temp = self.stage(label="match")
temp_id = temp.id
post = {
"id": 123456,
"rating": "s",
@@ -477,17 +543,42 @@ class UploadPipelineTests(TestCase):
):
upload_pipeline.run_pipeline(self.uploader.id)
temp.refresh_from_db()
self.assertEqual(temp.status, TempUpload.STATUS_COMPLETED)
self.assertEqual(temp.resolution, TempUpload.RESOLUTION_AUTO_MD5)
self.assertEqual(temp.e621_post_id, 123456)
self.assertIsNotNone(temp.library_item_id)
self.assertIsNotNone(temp.e621_checked_at)
self.assertIsNone(temp.claimed_at)
# The indexed upload leaves no board record; the feed reports it.
self.assertFalse(TempUpload.objects.filter(pk=temp_id).exists())
item = MediaItem.objects.get(e621_post_id=123456)
self.assertEqual(item.uploaded_by_id, self.uploader.id)
run = UploadRun.objects.get(user=self.uploader)
self.assertEqual(run.status, UploadRun.STATUS_IDLE)
self.assertEqual(run.matched, 1)
self.assertEqual(run.processed, 1)
self.assertEqual(len(run.recent_completions), 1)
entry = run.recent_completions[0]
self.assertEqual(entry["id"], str(temp_id))
self.assertEqual(entry["item_id"], item.id)
self.assertEqual(entry["filename"], "match.png")
self.assertEqual(entry["resolution"], TempUpload.RESOLUTION_AUTO_MD5)
def test_status_reports_the_completion_feed(self):
temp = self.stage(label="feed")
client = self.api_client(self.uploader)
post = {
"id": 654321,
"rating": "s",
"file": {"md5": temp.md5, "url": "https://static1.e621.net/data/f.png"},
}
with mock.patch.object(
upload_pipeline.e621,
"check_md5_batch",
return_value={temp.md5: post},
):
upload_pipeline.run_pipeline(self.uploader.id)
body = client.get("/api/uploads/status/").json()
completions = body["recent_completions"]
self.assertEqual(len(completions), 1)
self.assertEqual(completions[0]["filename"], "feed.png")
self.assertEqual(completions[0]["j_id"], f"J-{MediaItem.objects.get().id}")
self.assertIn("/thumbnail/", completions[0]["thumbnail_url"])
def test_unmatched_file_runs_every_phase(self):
temp = self.stage(label="nomatch")
+73 -14
View File
@@ -29,7 +29,7 @@ from django.db.models import F, Q
from django.utils import timezone
from . import e621, services
from .models import TempUpload, UploadRun
from .models import MediaItem, TempUpload, UploadRun
logger = logging.getLogger(__name__)
@@ -46,6 +46,9 @@ MAX_ATTEMPTS = 3
# Round-level e621 retries before the run is paused.
ROUND_ATTEMPTS = 3
ROUND_RETRY_SECONDS = 20
# Completions kept in the live feed. The board only shows what happened while
# the page was open, so a bounded rolling window is plenty.
COMPLETION_FEED_LIMIT = 200
WORK_STATUSES = (TempUpload.STATUS_PENDING, TempUpload.STATUS_VISUAL_MATCH)
VIDEO_RE = r"\.(mp4|webm)$"
@@ -116,7 +119,7 @@ def waiting_counts(user):
}
def status_payload(user):
def status_payload(user, request=None):
"""Cheap state for the shell/upload page to poll."""
run = UploadRun.objects.filter(user=user).first()
outstanding = count_outstanding(user)
@@ -141,10 +144,67 @@ def status_payload(user):
"error": run.error if run is not None else "",
"outstanding": outstanding,
"waiting": waiting_counts(user),
"recent_completions": completion_payload(run, user, request=request),
"updated_at": run.updated_at.isoformat() if run is not None else None,
}
def completion_payload(run, user, request=None):
"""The live completion feed: filename -> J-ID for freshly indexed uploads.
Completed ``TempUpload`` rows are deleted, so this is the only place the
board learns about them. It is a notification feed, not durable state:
bounded, never dismissed, and ignored by fresh page loads.
"""
entries = list(run.recent_completions or []) if run is not None else []
if not entries:
return []
ids = [entry.get("item_id") for entry in entries if entry.get("item_id")]
items = MediaItem.objects.in_bulk(ids)
out = []
for entry in reversed(entries): # newest first
item = items.get(entry.get("item_id"))
if item is None:
continue
out.append(
{
"id": entry.get("id"),
"filename": entry.get("filename"),
"j_id": f"J-{item.id}",
"resolution": entry.get("resolution", ""),
"post_id": entry.get("post_id"),
"thumbnail_url": services.signed_media_url(
item, user, "thumbnail", request=request
),
"at": entry.get("at"),
}
)
return out
def record_completion(temp, item, resolution=""):
"""Append one completion to the owner's feed; never fails an import."""
entry = {
"id": str(temp.pk),
"filename": temp.original_filename,
"item_id": item.pk,
"resolution": resolution or temp.resolution or "",
"post_id": temp.e621_post_id,
"at": timezone.now().isoformat(),
}
try:
with transaction.atomic():
run, _ = UploadRun.objects.select_for_update().get_or_create(
user_id=temp.user_id
)
feed = list(run.recent_completions or [])
feed.append(entry)
run.recent_completions = feed[-COMPLETION_FEED_LIMIT:]
run.save(update_fields=["recent_completions", "updated_at"])
except Exception: # noqa: BLE001 - a notification must not break an import
logger.exception("Could not record the completion of %s", temp.pk)
def reap_stale_claims():
"""Queue rows left claimed by a recycled worker and pause dead runs."""
cutoff = timezone.now() - STALE_CLAIM_AFTER
@@ -237,20 +297,17 @@ def run_pipeline(user_id):
break
process_round(run, user, rows)
except PipelinePaused as exc:
run.status = UploadRun.STATUS_PAUSED
run.phase = ""
run.error = str(exc)
run.save()
_save_run(run, status=UploadRun.STATUS_PAUSED, phase="", error=str(exc))
except Exception as exc: # noqa: BLE001 - surface crashes as a run error
logger.exception("Upload pipeline for user %s failed", user_id)
run.status = UploadRun.STATUS_ERROR
run.phase = ""
run.error = f"The upload pipeline stopped: {exc}"
run.save()
_save_run(
run,
status=UploadRun.STATUS_ERROR,
phase="",
error=f"The upload pipeline stopped: {exc}",
)
else:
run.status = UploadRun.STATUS_IDLE
run.phase = ""
run.save()
_save_run(run, status=UploadRun.STATUS_IDLE, phase="")
def claim_round(user, size=CLAIM_SIZE):
@@ -460,7 +517,9 @@ def finalize_round(run, ids, matched):
TempUpload.objects.filter(pk__in=release).update(claimed_at=None)
_save_run(
run,
processed=run.processed + len(finished),
# Completed rows are deleted as they are imported, so they cannot be
# seen in the refreshed rows; count the matches explicitly.
processed=run.processed + len(finished) + matched,
failed=run.failed + len(failed - finished),
matched=run.matched + matched,
)
+53 -19
View File
@@ -16,7 +16,6 @@ from urllib.parse import urlparse
from django.conf import settings
from django.contrib.auth import get_user_model
from django.core import signing
from django.core.exceptions import ValidationError
from django.http import Http404
from django.utils import timezone
@@ -30,6 +29,7 @@ from . import services
from .models import MediaItem, TempUpload
from .permissions import CanUpload
from .serializers import TempUploadListSerializer, TempUploadSerializer
from .signing_urls import load_payload
from .tools import HASH_FIELDS, hashed_items, hashes_similarity
logger = logging.getLogger(__name__)
@@ -60,6 +60,7 @@ def match_hashes(hashes, index, limit=10, user=None, request=None):
location = item.locations.first()
matches.append(
{
"item_id": item.id,
"j_id": f"J-{item.id}",
"filename": Path(location.rel_path).name if location else item.md5,
"similarity": round(similarity * 100, 1),
@@ -127,6 +128,12 @@ def complete_temp_upload(temp, download_url=None):
services.ensure_visual_hashes(item)
temp.file.delete(save=False)
# Warm the preview while the import is still off the request path.
try:
services.ensure_thumbnail(item)
except Exception: # noqa: BLE001 - a preview must not fail the import
logger.exception("Could not warm the thumbnail for J-%s", item.id)
temp.library_item = item
temp.status = TempUpload.STATUS_COMPLETED
@@ -158,6 +165,13 @@ def complete_temp_upload(temp, download_url=None):
item.save(update_fields=update_fields + ["updated_at"])
temp.save()
from .upload_pipeline import record_completion
record_completion(temp, item)
# The board learns about completions from the live feed, so the record is
# deleted as soon as the file is indexed: nothing left to dismiss. Delete
# through the queryset so callers keep ``temp.pk`` for their response.
TempUpload.objects.filter(pk=temp.pk).delete()
return item
@@ -190,6 +204,28 @@ class TempUploadViewSet(
user=self.request.user
)
def _completed_payload(self, temp, item):
"""Synthetic row for an upload that is indexed immediately.
Duplicates and resolved uploads never leave a board record; the SPA
turns this response (or the live completion feed) into a "J-x
uploaded" card that lives only in the page session.
"""
return {
"temp_id": str(temp.pk),
"original_filename": temp.original_filename,
"md5": temp.md5,
"size": temp.size,
"status": TempUpload.STATUS_COMPLETED,
"resolution": temp.resolution,
"e621_post_id": temp.e621_post_id,
"library_j_id": f"J-{item.id}",
"file_url": None,
"preview_url": services.signed_media_url(
item, self.request.user, "thumbnail", request=self.request
),
}
def create(self, request):
upload = request.FILES.get("file")
if upload is None:
@@ -217,6 +253,13 @@ class TempUploadViewSet(
temp.resolution = TempUpload.RESOLUTION_DUPLICATE
temp.library_item = existing
temp.file.delete(save=False)
temp.save()
from .upload_pipeline import record_completion
record_completion(temp, existing)
payload = self._completed_payload(temp, existing)
TempUpload.objects.filter(pk=temp.pk).delete()
return Response(payload, status=status.HTTP_201_CREATED)
# Visual similarity and IQDB run in the background pipeline so a large
# batch uploads at full speed and the work survives the browser.
temp.save()
@@ -257,7 +300,7 @@ class TempUploadViewSet(
"""Cheap pipeline state for the shell indicator and the upload page."""
from .upload_pipeline import status_payload
return Response(status_payload(request.user))
return Response(status_payload(request.user, request=request))
@action(detail=False, methods=["post"])
def process(self, request):
@@ -269,7 +312,7 @@ class TempUploadViewSet(
from .upload_pipeline import start_pipeline, status_payload
start_pipeline(request.user)
return Response(status_payload(request.user))
return Response(status_payload(request.user, request=request))
@action(detail=True, methods=["post"])
def retry(self, request, pk=None):
@@ -314,7 +357,7 @@ class TempUploadViewSet(
update["status"] = TempUpload.STATUS_PENDING
TempUpload.objects.filter(pk=temp.pk).update(**update)
start_pipeline(request.user)
return Response(status_payload(request.user))
return Response(status_payload(request.user, request=request))
@action(detail=False, methods=["post"], url_path="retry-all")
def retry_all(self, request):
@@ -340,7 +383,7 @@ class TempUploadViewSet(
e621_checked_at=None,
)
start_pipeline(request.user)
return Response(status_payload(request.user))
return Response(status_payload(request.user, request=request))
@action(detail=False, methods=["post"], url_path="discard-bulk")
def discard_bulk(self, request):
@@ -393,14 +436,7 @@ class TempUploadViewSet(
if user is None:
signature = request.query_params.get("sig")
if signature:
try:
payload = signing.loads(
signature,
salt=services.UPLOAD_FILE_SALT,
max_age=86400,
)
except signing.BadSignature:
payload = None
payload = load_payload(signature, services.UPLOAD_FILE_SALT)
if payload and str(payload.get("temp")) == str(pk):
user = (
get_user_model()
@@ -501,7 +537,7 @@ class TempUploadViewSet(
temp.save()
try:
complete_temp_upload(temp, download_url=download_url)
item = complete_temp_upload(temp, download_url=download_url)
except Exception as exc: # noqa: BLE001 - report completion failures
logger.exception("Could not complete staged upload %s", temp.id)
temp.status = TempUpload.STATUS_ERROR
@@ -510,8 +546,7 @@ class TempUploadViewSet(
{"detail": f"Could not finish the upload: {exc}"},
status=status.HTTP_400_BAD_REQUEST,
)
temp.refresh_from_db()
return Response(self.get_serializer(temp).data)
return Response(self._completed_payload(temp, item))
@action(detail=False, methods=["post"], url_path="link-bulk")
def link_bulk(self, request):
@@ -583,15 +618,14 @@ class TempUploadViewSet(
candidate_url = ""
temp.save()
try:
complete_temp_upload(temp, download_url=candidate_url or None)
item = complete_temp_upload(temp, download_url=candidate_url or None)
except Exception as exc: # noqa: BLE001 - report per-file failures
logger.exception("Could not complete staged upload %s", temp.id)
temp.status = TempUpload.STATUS_ERROR
temp.save(update_fields=["status", "updated_at"])
errors.append({"temp_id": temp_id, "error": str(exc)})
continue
temp.refresh_from_db()
updated.append(self.get_serializer(temp).data)
updated.append(self._completed_payload(temp, item))
return Response({"updated": updated, "errors": errors})
+28 -14
View File
@@ -5,7 +5,6 @@ from pathlib import Path
from urllib.parse import urlparse
from django.conf import settings
from django.core import signing
from django.db.models import Min, Q
from django.http import Http404, StreamingHttpResponse
from django.shortcuts import get_object_or_404
@@ -31,6 +30,7 @@ from .serializers import (
MatchTaskSerializer,
MediaItemSerializer,
)
from .signing_urls import load_payload
LIST_ORDERINGS = {"name", "-name", "size", "-size", "created_at", "-created_at"}
MD5_RE = re.compile(r"[0-9a-fA-F]{32}")
@@ -132,11 +132,8 @@ class MediaItemViewSet(
signature = request.query_params.get("sig")
if not signature:
return None
try:
payload = signing.loads(
signature, salt=services.MEDIA_FILE_SALT, max_age=86400
)
except signing.BadSignature:
payload = load_payload(signature, services.MEDIA_FILE_SALT)
if payload is None:
return None
if payload.get("action") != action_name:
return None
@@ -158,7 +155,11 @@ class MediaItemViewSet(
status=status.HTTP_404_NOT_FOUND,
)
return services.serve_file(
request, location.path, download=request.query_params.get("download") == "1"
request,
location.path,
download=request.query_params.get("download") == "1",
max_age=services.MEDIA_CACHE_SECONDS,
immutable=True,
)
@action(detail=True, methods=["get"], throttle_classes=[])
@@ -173,13 +174,26 @@ class MediaItemViewSet(
path = Path(location.path)
if path.suffix.lower() in services.VIDEO_EXTENSIONS:
thumbnail = services.generate_video_thumbnail(item.md5, path)
if thumbnail is None:
return Response(
{"detail": "Thumbnail unavailable."},
status=status.HTTP_404_NOT_FOUND,
)
return services.serve_file(request, thumbnail)
return services.serve_file(request, path)
else:
thumbnail = services.generate_image_thumbnail(item.md5, path)
if thumbnail is not None:
return services.serve_file(
request,
thumbnail,
max_age=services.MEDIA_CACHE_SECONDS,
immutable=True,
)
if path.suffix.lower() in services.VIDEO_EXTENSIONS:
return Response(
{"detail": "Thumbnail unavailable."},
status=status.HTTP_404_NOT_FOUND,
)
return services.serve_file(
request,
path,
max_age=services.MEDIA_CACHE_SECONDS,
immutable=True,
)
@action(detail=False, methods=["post"], permission_classes=[AllowAny])
def lookup(self, request):
+9 -12
View File
@@ -145,9 +145,12 @@ Build the desktop installers without publishing anything:
```
Hand the files out or attach them to a Gitea release manually — the script
prints sizes and SHA-256 sums for the release notes.
prints sizes and SHA-256 sums for the release notes. The release assets are
also the desktop update feed; the app resolves the newest `desktop-v*`
release on Gitea at check time (see `desktop/README.md`).
Push the desktop builds and their update metadata to the frontend's feed:
The frontend's `/desktop/` feed is optional now — kept for manual downloads
and for installs older than 0.1.2. To publish it:
```bash
./push_desktop.sh # build Linux packages + copy the feed to jakerasp
@@ -160,19 +163,13 @@ The remote copy defaults to `jakerasp:/home/jake/servers/J621`, or
`$J621_DESKTOP_FEED_HOST` when set. Artifacts land in `deploy/data/desktop/`,
which the frontend nginx mounts read-only and serves at `/desktop/`. The
remote copy uses rsync when both ends have it, tar over ssh when the server
does not. The desktop app's "Check for updates…" menu item reads
`latest-linux.yml` / `latest.yml` from there (see `desktop/README.md`).
Backend-only composes have no frontend, so no feed.
does not. Backend-only composes have no frontend, so no website feed.
The manual **CD** workflow (Actions tab) builds the desktop packages on the
runner and attaches them plus the update metadata to the Gitea release
`desktop-v<version>`; it does not touch the live feed, because that is
runtime state on the deploy host and CI has no SSH key for it. After a CD run,
publish the feed from a machine that can reach the deploy checkout:
```bash
./push_desktop.sh --no-build --local # or without --local to also copy it
```
`desktop-v<version>`; that is what the desktop updater reads. The website feed
is not touched by CI (runtime state on the deploy host, no SSH key there); use
`push_desktop.sh` when it needs refreshing for old installs.
## Scheduled jobs
+7 -6
View File
@@ -1,11 +1,12 @@
#!/bin/bash
# Build the J621 desktop packages and publish them to the update feed.
# Build the J621 desktop packages, optionally publish them to the website
# feed.
#
# Locally the feed is deploy/data/desktop, which the frontend nginx mounts
# read-only and serves at /desktop/. With --host the same directory is also
# copied to a remote deploy checkout (rsync, or tar over ssh when the server
# has no rsync). electron-updater reads latest-linux.yml / latest.yml from
# there; the feed URL comes from desktop/electron-builder.yml.
# Desktop updates no longer depend on this: the app resolves the newest
# `desktop-v*` release on Gitea at check time (see desktop/README.md). This
# script builds the packages and can copy them to deploy/data/desktop, which
# the frontend nginx mounts read-only and serves at /desktop/ for manual
# downloads and for pre-0.1.2 installs.
#
# Usage: ./push_desktop.sh [--win] [--no-build] [--local] [--host user@server:/path]
# --win also cross-build the Windows NSIS installer (needs wine)
+19 -8
View File
@@ -49,7 +49,8 @@ npm run dist:all
`deploy/build_desktop.sh` wraps the same commands, installs dependencies on
first run and prints sizes plus SHA-256 sums for release notes. Nothing is
published by it; `deploy/push_desktop.sh` is the one that feeds auto-updates.
published by it; the CD workflow attaches the artifacts to the Gitea release,
which is also the update feed.
The Arch package can be installed and removed with pacman:
@@ -68,19 +69,29 @@ will warn, and it has not been smoke-tested on real Windows.
The app checks only when asked (**J621 → Check for updates…** in the menu):
Linux packages install through pacman/dpkg, which needs administrator rights,
and the Windows build is unsigned, so nothing installs silently. The check
reads `latest-linux.yml` / `latest.yml` from the feed configured in
`electron-builder.yml` (`publish.url`, baked into `app-update.yml`); set
`J621_UPDATE_URL` to point a build at another feed (the smoke test uses this).
and the Windows build is unsigned, so nothing installs silently.
The check resolves the feed itself: it asks the Gitea API for the newest
non-draft `desktop-v*` release (`J621_UPDATE_REPO`, default
`https://gitea.rainbow-herring.ts.net/jakebreath/j621` — lowercase on purpose,
the API path is case-sensitive), picks the `latest-linux.yml` / `latest.yml`
asset for the platform and uses that release as an electron-updater generic
feed. The baked `publish.url` in `electron-builder.yml` is metadata only.
`J621_UPDATE_URL` overrides the whole lookup (the smoke test uses this).
Publishing a release:
1. Bump `version` in `desktop/package.json` — that is what the updater compares.
2. `./deploy/push_desktop.sh --win` builds deb/pacman/NSIS and copies the
artifacts plus both channel files into `deploy/data/desktop/`, which the
frontend nginx serves read-only at `/desktop/`.
2. Run the manual CD workflow, which builds the packages and attaches them
plus both channel files to the Gitea release `desktop-v<version>`.
`./deploy/push_desktop.sh --win` does the same build locally (and can also
copy the files to the website feed, which is optional now).
3. Existing installs find the new version on their next manual check.
Note for the 0.1.1 → 0.1.2 step: 0.1.1 only knows the old `/desktop/` feed, so
publish 0.1.2 there once (`./deploy/push_desktop.sh --no-build`, or install it
manually). From 0.1.2 on, updates come from Gitea.
`package-type` in the app resources tells electron-updater whether to run
`pacman -U` or `dpkg -i` (both via pkexec/sudo); the per-user NSIS install
updates without elevation.
+5 -4
View File
@@ -2,12 +2,13 @@ appId: io.j621.desktop
productName: J621
copyright: Copyright (c) 2026 JakeBreath — Jake Labs Non-Commercial Software Licence
# Update feed served by the frontend nginx (deploy/data/desktop, published
# with deploy/push_desktop.sh). Baked into resources/app-update.yml; override
# at runtime with J621_UPDATE_URL for a fork or a test feed.
# Update feed: desktop/src/main.ts resolves the newest desktop-v* release on
# Gitea at check time (J621_UPDATE_REPO). This block only tells electron-builder
# to emit latest.yml/latest-linux.yml next to the installers; J621_UPDATE_URL
# overrides the feed for a fork or a test.
publish:
provider: generic
url: https://j621.rainbow-herring.ts.net/desktop
url: https://gitea.rainbow-herring.ts.net/JakeBreath/J621/releases
directories:
output: release
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "j621-desktop",
"version": "0.1.1",
"version": "0.1.2",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "j621-desktop",
"version": "0.1.1",
"version": "0.1.2",
"license": "LicenseRef-Jake-Labs-Non-Commercial",
"dependencies": {
"electron-updater": "^6.8.9"
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "j621-desktop",
"productName": "J621",
"version": "0.1.1",
"version": "0.1.2",
"private": true,
"description": "Desktop shell for the J621 self-hosted media archive",
"author": {
+82 -4
View File
@@ -238,8 +238,9 @@ function isDownloadNavigation(url: URL): boolean {
/**
* Updates are manual by design: Linux packages install through pacman/dpkg
* (pkexec/sudo) and the Windows build is unsigned, so the app asks before
* downloading and again before installing. The feed comes from `publish` in
* electron-builder.yml and can be overridden with J621_UPDATE_URL.
* downloading and again before installing. The feed is resolved at check time
* from the newest `desktop-v*` release on Gitea (`J621_UPDATE_REPO`);
* `J621_UPDATE_URL` overrides it for forks and the smoke test.
*/
type UpdateEvent =
| { state: "available"; version: string }
@@ -250,9 +251,84 @@ type UpdateEvent =
let updateReporter: ((event: UpdateEvent) => void) | null = null;
let updateCheckRunning = false;
function setUpUpdates(win: BrowserWindow): void {
const UPDATE_REPO =
process.env.J621_UPDATE_REPO?.trim() ||
"https://gitea.rainbow-herring.ts.net/jakebreath/j621";
interface ReleaseAsset {
name: string;
browser_download_url: string;
}
interface Release {
tag_name: string;
draft: boolean;
prerelease: boolean;
assets?: ReleaseAsset[];
}
function releaseVersion(tag: string): [number, number, number] | null {
const match = /^desktop-v(\d+)\.(\d+)\.(\d+)$/.exec(tag);
if (!match) return null;
return [Number(match[1]), Number(match[2]), Number(match[3])];
}
function compareVersions(
a: [number, number, number],
b: [number, number, number],
): number {
for (let index = 0; index < 3; index += 1) {
if (a[index] !== b[index]) return a[index] - b[index];
}
return 0;
}
/**
* Resolve the generic feed base electron-updater should use.
*
* Gitea's API path is case-sensitive (owner/repo must match the login), while
* the asset URLs it returns are canonical, so the base is derived from the
* platform's metadata asset (`latest-linux.yml` / `latest.yml`).
*/
async function resolveReleaseFeed(): Promise<string> {
const override = process.env.J621_UPDATE_URL?.trim();
if (override) autoUpdater.setFeedURL({ provider: "generic", url: override });
if (override) return override;
const match = /^(https?:\/\/[^/]+)\/([^/]+)\/([^/]+?)\/?$/.exec(UPDATE_REPO);
if (!match) {
throw new Error(
`J621_UPDATE_REPO must be <origin>/<owner>/<repo> (got ${UPDATE_REPO}).`,
);
}
const [, origin, owner, repo] = match;
const response = await fetch(
`${origin}/api/v1/repos/${owner}/${repo}/releases?limit=50`,
{ headers: { Accept: "application/json" } },
);
if (!response.ok) {
throw new Error(`Release lookup on Gitea failed (HTTP ${response.status}).`);
}
const releases = (await response.json()) as Release[];
const assetName =
process.platform === "win32" ? "latest.yml" : "latest-linux.yml";
let best: { version: [number, number, number]; asset: ReleaseAsset } | null =
null;
for (const release of releases) {
if (release.draft || release.prerelease) continue;
const version = releaseVersion(release.tag_name);
if (!version) continue;
const asset = release.assets?.find((entry) => entry.name === assetName);
if (!asset) continue;
if (!best || compareVersions(version, best.version) > 0) {
best = { version, asset };
}
}
if (!best) {
throw new Error(`No ${assetName} asset found in ${UPDATE_REPO} releases.`);
}
return best.asset.browser_download_url.replace(/\/[^/]*$/, "");
}
function setUpUpdates(win: BrowserWindow): void {
if (!app.isPackaged) autoUpdater.forceDevUpdateConfig = true;
autoUpdater.autoDownload = false;
autoUpdater.autoInstallOnAppQuit = false;
@@ -336,6 +412,8 @@ async function checkForUpdates(win: BrowserWindow): Promise<void> {
if (updateCheckRunning) return;
updateCheckRunning = true;
try {
const feed = await resolveReleaseFeed();
autoUpdater.setFeedURL({ provider: "generic", url: feed });
await autoUpdater.checkForUpdates();
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
+1 -1
View File
@@ -113,7 +113,7 @@ export function AppShell() {
return (
<div className="flex min-h-screen flex-col">
<header className="sticky top-0 z-40 border-b border-ctp-surface0 bg-ctp-crust/95 backdrop-blur">
<div className="mx-auto flex h-14 w-full max-w-[1600px] items-center gap-4 px-4">
<div className="flex h-14 w-full items-center gap-4 px-3 sm:px-4">
<Link to="/" className="flex shrink-0 items-center">
<span className="rounded bg-ctp-mauve px-2 py-1 font-mono text-xs font-bold tracking-wide text-ctp-crust">
J621
+3 -3
View File
@@ -18,9 +18,9 @@ const ratingLabels: Record<string, string> = {
};
export function MediaCard({ item }: { item: MediaItem }) {
const preview = apiUrl(
item.kind === "video" ? item.thumbnail_url : item.raw_url,
);
// The thumbnail endpoint now builds real 480px previews for images too, so
// the grid no longer pulls full-size originals.
const preview = apiUrl(item.thumbnail_url);
const rating = item.display_rating;
return (
+51 -34
View File
@@ -1,5 +1,5 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Eye, StarOff } from "lucide-react";
import { ChevronDown, ChevronRight, Eye, StarOff } from "lucide-react";
import { useState } from "react";
import { Link } from "react-router-dom";
@@ -148,48 +148,65 @@ function BlacklistCloudPanel() {
q.state.data?.status === "building" ? 2000 : false,
});
const cloud = query.data;
const [expanded, setExpanded] = useState(false);
return (
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
<div className="flex flex-wrap items-center justify-between gap-2">
<button
type="button"
onClick={() => setExpanded((value) => !value)}
title={expanded ? "Hide the tags" : "Show the tags"}
className="flex w-full items-center justify-between gap-2 text-left"
>
<h2 className="text-sm font-semibold text-ctp-subtext1">
Blacklisted tags
</h2>
<p className="text-xs text-ctp-overlay0">
{cloud?.source === "user"
? "From your e621 blacklist"
: "From e621's anonymous default blacklist"}
{cloud ? ` · ${cloud.blacklist_count} entries` : ""}
{cloud ? ` · ${cloud.posts} feed post(s) scanned` : ""}
{cloud?.computed_at ? ` · computed ${formatDate(cloud.computed_at)}` : ""}
<span className="flex items-center gap-1.5 font-mono text-[11px] text-ctp-overlay0">
{cloud ? `${cloud.blacklist_count} entries` : "…"}
{cloud?.status === "building" ? (
<span className="ml-2 inline-flex items-center gap-1.5">
<Spinner className="h-3 w-3" /> building…
</span>
<Spinner className="h-3 w-3" />
) : null}
</p>
</div>
{expanded ? (
<ChevronDown className="h-3.5 w-3.5" />
) : (
<ChevronRight className="h-3.5 w-3.5" />
)}
</span>
</button>
{query.isPending ? (
<div className="mt-3 flex justify-center py-4">
<Spinner className="h-4 w-4" />
</div>
) : cloud && cloud.tags.length > 0 ? (
<div className="mt-3 flex flex-wrap gap-1.5">
{cloud.tags.map(([tag, count]) => (
<span
key={tag}
className="rounded border border-ctp-red/30 bg-ctp-red/10 px-1.5 py-0.5 font-mono text-[11px] text-ctp-red"
>
{tag} ({count})
</span>
))}
</div>
) : (
<p className="mt-3 text-xs text-ctp-overlay0">
No blacklisted tags in your feeds.
</p>
)}
{expanded ? (
<>
<p className="mt-2 text-xs text-ctp-overlay0">
{cloud?.source === "user"
? "From your e621 blacklist"
: "From e621's anonymous default blacklist"}
{cloud ? ` · ${cloud.posts} feed post(s) scanned` : ""}
{cloud?.computed_at
? ` · computed ${formatDate(cloud.computed_at)}`
: ""}
</p>
{query.isPending ? (
<div className="mt-3 flex justify-center py-4">
<Spinner className="h-4 w-4" />
</div>
) : cloud && cloud.tags.length > 0 ? (
<div className="mt-3 flex flex-wrap gap-1.5">
{cloud.tags.map(([tag, count]) => (
<span
key={tag}
className="rounded border border-ctp-red/30 bg-ctp-red/10 px-1.5 py-0.5 font-mono text-[11px] text-ctp-red"
>
{tag} ({count})
</span>
))}
</div>
) : (
<p className="mt-3 text-xs text-ctp-overlay0">
No blacklisted tags in your feeds.
</p>
)}
</>
) : null}
</section>
);
}
+83 -64
View File
@@ -1,5 +1,5 @@
import { keepPreviousData, useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { X } from "lucide-react";
import { ChevronDown, ChevronRight, X } from "lucide-react";
import { useEffect, useMemo, useRef, useState } from "react";
import { Link, useLocation, useSearchParams } from "react-router-dom";
@@ -43,6 +43,7 @@ export default function OnlinePage() {
const queryClient = useQueryClient();
const [draft, setDraft] = useState<string | null>(null);
const [blacklistDraft, setBlacklistDraft] = useState("");
const [showBlacklist, setShowBlacklist] = useState(false);
const tagInput = draft ?? tags;
const zoom = useUi((state) => state.zoom);
const perPage = useUi((state) => state.e621PerPage);
@@ -271,73 +272,91 @@ export default function OnlinePage() {
</div>
<div className="flex flex-col gap-2">
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Your blacklist
</span>
{credentials?.configured ? (
<>
{blacklistEntries.length > 0 ? (
<div className="flex flex-wrap gap-1.5">
{blacklistEntries.map(({ line, index }) => (
<span
key={`${line}-${index}`}
className="inline-flex items-center gap-1 rounded border border-ctp-red/30 bg-ctp-red/10 px-1.5 py-0.5 font-mono text-[11px] text-ctp-red"
>
{line}
<button
type="button"
title={`Remove ${line}`}
disabled={blacklistMutation.isPending}
onClick={() => removeBlacklistEntry(index)}
className="text-ctp-red/70 transition hover:text-ctp-red disabled:opacity-40"
>
<X className="h-3 w-3" />
</button>
</span>
))}
</div>
<button
type="button"
onClick={() => setShowBlacklist((value) => !value)}
title={showBlacklist ? "Hide the blacklist" : "Show the blacklist"}
className="flex items-center justify-between gap-2 text-left"
>
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Your blacklist
</span>
<span className="flex items-center gap-1 font-mono text-[11px] text-ctp-overlay0">
{blacklistEntries.length} tag
{blacklistEntries.length === 1 ? "" : "s"}
{showBlacklist ? (
<ChevronDown className="h-3.5 w-3.5" />
) : (
<p className="text-xs text-ctp-overlay0">
No blacklisted tags.
</p>
<ChevronRight className="h-3.5 w-3.5" />
)}
<form
onSubmit={(event) => {
event.preventDefault();
addBlacklistEntry();
}}
className="flex gap-2"
>
<input
className={cn(inputClass, "font-mono")}
placeholder="Add a tag…"
value={blacklistDraft}
onChange={(event) => setBlacklistDraft(event.target.value)}
/>
<Button
type="submit"
variant="secondary"
disabled={!blacklistDraft.trim() || blacklistMutation.isPending}
</span>
</button>
{showBlacklist ? (
credentials?.configured ? (
<>
{blacklistEntries.length > 0 ? (
<div className="flex flex-wrap gap-1.5">
{blacklistEntries.map(({ line, index }) => (
<span
key={`${line}-${index}`}
className="inline-flex items-center gap-1 rounded border border-ctp-red/30 bg-ctp-red/10 px-1.5 py-0.5 font-mono text-[11px] text-ctp-red"
>
{line}
<button
type="button"
title={`Remove ${line}`}
disabled={blacklistMutation.isPending}
onClick={() => removeBlacklistEntry(index)}
className="text-ctp-red/70 transition hover:text-ctp-red disabled:opacity-40"
>
<X className="h-3 w-3" />
</button>
</span>
))}
</div>
) : (
<p className="text-xs text-ctp-overlay0">
No blacklisted tags.
</p>
)}
<form
onSubmit={(event) => {
event.preventDefault();
addBlacklistEntry();
}}
className="flex gap-2"
>
{blacklistMutation.isPending ? (
<Spinner className="h-3.5 w-3.5" />
) : (
"Add"
)}
</Button>
</form>
<p className="text-[11px] text-ctp-overlay0">
Saved to your e621 account.
<input
className={cn(inputClass, "font-mono")}
placeholder="Add a tag…"
value={blacklistDraft}
onChange={(event) => setBlacklistDraft(event.target.value)}
/>
<Button
type="submit"
variant="secondary"
disabled={!blacklistDraft.trim() || blacklistMutation.isPending}
>
{blacklistMutation.isPending ? (
<Spinner className="h-3.5 w-3.5" />
) : (
"Add"
)}
</Button>
</form>
<p className="text-[11px] text-ctp-overlay0">
Saved to your e621 account.
</p>
</>
) : (
<p className="text-xs text-ctp-overlay0">
<Link to="/account" className="text-ctp-blue hover:underline">
Add e621 credentials
</Link>{" "}
to manage your blacklist.
</p>
</>
) : (
<p className="text-xs text-ctp-overlay0">
<Link to="/account" className="text-ctp-blue hover:underline">
Add e621 credentials
</Link>{" "}
to manage your blacklist.
</p>
)}
)
) : null}
</div>
<div className="flex flex-col gap-2">
File diff suppressed because it is too large Load Diff
+24
View File
@@ -388,6 +388,28 @@ export interface TempUpload {
updated_at: string;
}
/** One upload the pipeline indexed while the page was watching. */
export interface UploadCompletion {
id: string;
filename: string;
j_id: string;
resolution: string;
post_id: number | null;
thumbnail_url: string | null;
at: string;
}
/** Response of POST /api/uploads/ — a pending row or an instant completion. */
export interface UploadStagingResult {
temp_id: string;
original_filename: string;
status: "pending" | "visual_match" | "completed" | "error";
resolution: string;
library_j_id: string | null;
preview_url: string | null;
e621_post_id?: number | null;
}
/** Progress of the server-side staged-upload pipeline. */
export interface UploadStatus {
status: "idle" | "running" | "paused" | "error";
@@ -400,6 +422,8 @@ export interface UploadStatus {
error: string;
outstanding: number;
waiting: { md5: number; visual: number; iqdb: number };
/** Session notification feed: indexed uploads, newest first. */
recent_completions: UploadCompletion[];
updated_at: string | null;
}