Compare commits

...
26 Commits
Author SHA1 Message Date
JakeBreath cea34422ec Add the Jake Labs Non-Commercial Software Licence
Source-available, non-commercial licence (attribution + copyleft) at the
repo root as LICENSE, with the year/name filled in, ASCII punctuation,
a termination-on-breach clause with a 30-day cure, and a clearer
'no licence' wording in section 0. README gains a Licence section.
2026-09-17 15:14:05 -05:00
JakeBreath e95dc5f265 Clear the query cache when the signed-in account changes
Logging out or switching accounts kept the previous user's React Query
cache (follows, feed, cloud, e621 pages), so the new account briefly
saw the old one's followed tags/pools until each query refetched. The
query client now lives in lib/queryClient.ts and login/register/logout
clear it alongside the e621 credential store.
2026-09-17 14:41:56 -05:00
JakeBreath 62561a13f3 Show the Followed tag and pool cards in four columns on wide screens 2026-09-17 14:39:04 -05:00
JakeBreath 4c0b8cb1d8 Simplify the Followed page layout
- Remove the follow-a-tag / follow-a-pool forms; follows happen from tag
  chips (+ on detail views) and the Follow pool button on pool pages, so
  the empty states now point there instead.
- Move the blacklisted-tag cloud to a full-width horizontal panel at the
  bottom of the page.
- Tag and pool card grids cap at two columns so the cover cards are
  bigger.
2026-09-17 14:37:59 -05:00
JakeBreath 26f5069aa2 Fix pool post cards rendering as a colored sliver
PostCard's Link is inline by default; wrapped in a div on the pool detail
page it stopped being blockified like a grid child, so its rating-tinted
border collapsed into a vertical line on the left. The card is now
block-level, and the pool grid only wraps blacklisted posts (for the red
ring), so normal cards render exactly like the Online grid.
2026-09-17 14:32:24 -05:00
JakeBreath 0a7dc5a991 Add a Pools browser backed by e621's /pools endpoint
- /pools: search by name, category/active filters, sort options and
  pagination per the OpenAPI spec, with covers taken from each pool's
  first post in one batched post call; blacklisted covers fall back to a
  placeholder and deleted pools get an archive marker.
- /pools/<id>: DText description, post grid kept in the pool's own order
  with chunked loading, in-library badges, a blacklist reveal toggle and
  a Follow pool button wired into the follows API.
- e621 client gains fetchPools/fetchPool; Pools nav entry added.
2026-09-17 14:26:20 -05:00
JakeBreath c396c368dc Animate the tag follow toggle while the request settles
The chip toggle swaps its icon for a tiny current-color spinner from
the click until the follows list reflects the new state, covering the
refetch gap so '+' never flashes back before the checkmark.
2026-09-17 14:14:58 -05:00
JakeBreath 4585a8ac43 Add follow toggles to tag chips on detail views
TagChip gains a followable mode that renders a small toggle inside the
chip: '+' follows the tag, '✓' (click to unfollow) once followed, and a
red marker with the reason as its title when e621 rejects it. Enabled on
the library detail's e621 metadata card (matched posts) and on online
post detail tags; guests see plain chips and the label click keeps
working beside the toggle.
2026-09-17 14:12:46 -05:00
JakeBreath 03dd235f3a Follows: followed tags/pools, feeds, unseen badges and blacklist cloud
Backend (new apps.follows):
- FollowedTag/FollowedPool/FollowedPost models; per-user follows with
  unseen tracking, plus FollowCloud for the cached blacklist cloud.
- Two periodic commands sharing one fetch path: sync_followed_tags and
  sync_followed_pools fetch each followed tag/pool's newest posts (one
  e621 search per unique follow), store unseen feed rows, refresh covers
  and pool metadata; both fall back to anonymous e621 access.
- API: /api/follows/tags|pools (follow, unfollow, mark seen), a merged
  feed with per-follow filtering, and /api/follows/cloud/ which rebuilds
  the blacklisted-tag cloud in a daemon thread when its 10 min cache is
  stale (polling returns building/ready).
- e621 client now supports anonymous reads; trimmed posts carry preview
  URLs for covers and feed tiles.

Frontend:
- /followed page: follow forms, cover cards with unseen badges and
  Mark seen, merged feed with filter/unseen toggle, and a blacklist
  cloud panel that polls while building. Followed nav entry added.
2026-09-17 14:09:30 -05:00
JakeBreath c55fa1fdca IQDB reverse search from the library detail
- New IQDB card on local item pages: fetches the signed raw file, POSTs it
  to e621's /iqdb_queries.json with the user's credentials, and lists
  candidates as tiles (thumbnail, rating, score%) with exact-MD5 markers.
- Selecting a candidate opens a detail panel (rating, score, favs, size,
  tag preview) and linking is an explicit confirm that reuses the e621
  match endpoint; videos show a note since IQDB is image-only.
- Guests don't see the card; linking follows the uploader/staff rule.
2026-09-17 13:53:23 -05:00
JakeBreath a0d36911e7 Add an 18+ entry screen before the app renders
A homescreen-style age gate shown before auth or any route: J621 brand
mark, the explicit 18+ check, an Enter action remembered per browser in
localStorage (j621.age-verified), and a blocked state if the visitor
chooses Leave.
2026-09-17 13:47:51 -05:00
JakeBreath 09405d1a0f Match local files to e621: MD5 lookups, manual links, batch scans
- MediaItem gains e621_match_status (unknown/matched/not_found/deleted)
  and e621_checked_at, backfilled for existing matched items.
- Server-side e621 client (apps/library/e621.py) using the user's stored
  credentials, throttled to 2 req/s, with typed errors.
- Matching service: MD5 lookup, manual post linking (flags MD5
  mismatches), unlink, metadata refresh, deleted-post detection.
- Detail actions POST /api/files/J-x/match/ and /unlink/ (uploader or
  staff only).
- Background library scans: MatchTask + /api/matches/ with missing/all
  scopes, progress polling, cancel and stale-task reaping; the scan
  counts toward the footer's Active Workers. Same pass available as
  manage.py match_e621 for cron.
- Library gains not_found/deleted status filters; the detail page adds
  an e621 match card (check / link by post ID / unlink) and the metadata
  card warns when a post was deleted on e621.
2026-09-17 13:41:08 -05:00
JakeBreath db74f7ab18 Fix hidden library items not rendering in the browser
Items flagged hidden_from_guests (blacklisted tags) returned 404 for
<img> requests since tags cannot send the auth header. The API now
exposes signed raw_url/thumbnail_url fields (mirroring upload previews
and avatars), and the SPA uses them in the gallery, detail view,
duplicates and delete screens, and upload visual matches.
2026-09-17 13:25:20 -05:00
JakeBreath 4df573da43 Library search upgrades: tag search, tag cloud, status filter
Backend:
- MediaItem gains search_tags (custom + e621 tags, lowercase) and
  has_custom_data, maintained on save with a data migration backfill
- File list search accepts search_type=filename|tags|both (tag search is
  word-AND across the flattened tag text) and status=matched|custom|
  unknown filters
- New /api/tags/cloud/ endpoint (cached 2 min per guest/auth, invalidated
  on item changes and deletions) returning the most-used tags, honouring
  guest visibility

Frontend:
- Library sidebar: Filename/Tags/Both selector, status pill toggles
  (persisted), and a clickable tag cloud that runs a tag search
- Roadmap updated
2026-09-17 13:19:17 -05:00
JakeBreath 7ec8ee974e Add a delete button to the library detail page
Staff and uploaders (for their own items) get a Delete action on
/detail/J-<id> with an inline confirmation; on success it returns to the
library and refreshes files, duplicates and storage.
2026-09-17 12:57:26 -05:00
JakeBreath cd490b0a23 Duplicates, delete & storage, users page with J-ID avatars
Backend:
- Perceptual hashes (aHash/dHash/pHash/wHash via imagehash, no imgdd)
  stored on items, computed on upload/download and by the new
  compute_visual_hashes command
- Duplicates API: exact duplicates (multi-location items), visual matches
  for one item, union-find similarity groups with pagination
- Delete API with ownership/staff checks, per-item and per-copy deletion,
  watched-folder path validation; storage overview and temp cleanup;
  file list accepts j_ids batches
- Staged uploads are flagged visual_match with their library matches
  (threshold via VISUAL_MATCH_THRESHOLD)
- Staff users API: list with upload counts, set role and avatar by J-ID;
  User.avatar FK with signed avatar URLs
- Download threads close their DB connection and stale tasks are reaped,
  keeping behaviour Gunicorn-friendly

Frontend:
- /duplicates: exact duplicate groups with per-copy delete, visual
  similarity controls, search similar to a J-ID, paginated groups with
  selection, bulk delete and dismiss
- /delete: storage cards, delete by J-ID with preview grid, temp cleanup
- /users: staff directory with role selects and avatar J-ID inputs
- Nav + command palette entries; top-bar avatar; upload cards and the
  metadata modal show library visual matches
2026-09-17 12:49:10 -05:00
JakeBreath 75b7ed35eb Live worker counts and an e621 request-history dropdown
- Status polling drops from 30s to 5s, and download start/finish/cancel
  invalidates it immediately, so the footer's Active Workers reflects
  running download tasks in near real time
- The e621 time in the status pill is now a button: it opens a dropdown
  with the session's request history (clock time, endpoint, duration,
  colour-coded) plus totals; closes on outside click or Escape
- Metrics store keeps the last 50 requests
2026-09-17 12:20:27 -05:00
JakeBreath 6962e483fc Async Download to Library with progress; Download to client
Backend:
- DownloadTask model + background thread runner: streams the file with
  progress (%, bytes, speed) and a cancel flag, then indexes it, names it
  J-<id>.<ext> and applies the e621 metadata
- DownloadTaskViewSet (create/retrieve/cancel) replaces the synchronous
  endpoint; the status footer's worker counts now reflect download jobs
- Client download proxy (/api/online/file/) streams an e621 original to
  the browser with Content-Disposition: attachment, restricted to the
  configured e621 CDN hosts so it cannot be used as an open proxy

Frontend:
- Online detail: progress bar with percentage, transferred size, speed
  and cancel while downloading; success links to the new J-ID
- New 'Download to client' button available to everyone (guests too)
2026-09-17 12:06:15 -05:00
JakeBreath bf00cf36a2 Name uploaded and downloaded files J-<id>.<ext>
Files added through the upload pipeline and Download to Library are
renamed to their J-ID right after indexing, so every new library file is
traceable by its identifier (scanned files keep their existing names).
index_file now returns the created location so callers can rename it;
the location record is updated to the new path.
2026-09-17 11:56:01 -05:00
JakeBreath d4f5df2e4e Metadata modal: size the preview to the image's own ratio
The modal preview used a fixed 200px column with object-contain, so wide
images rendered as a slim letterboxed rectangle with dead space. The
preview now sizes naturally (max 320px wide / 55vh tall, aspect ratio
preserved) and the modal column follows it.
2026-09-17 11:51:04 -05:00
JakeBreath a3d1063946 Upload page: auto-upload and a 5x3 progress grid
- Files upload as soon as they are dropped or selected; a sequential
  queue processes them and picks up files added while uploading
- The progress list is now a grid: 5 columns with about three rows
  visible (scrolls beyond) showing thumbnails, per-file progress bars
  and status; 'clear finished' remains, object URLs are revoked
- Videos show an icon instead of a thumbnail
2026-09-17 11:35:10 -05:00
JakeBreath 1086beb974 Upload board: dismiss all, real previews for indexed records
- 'dismiss all' clears every indexed record at once
- Indexed cards show the actual file preview: completed records now get a
  signed library media URL (raw for images, thumbnail for videos) so
  <img>/<video> tags can load it, including items hidden from guests
- Media raw/thumbnail endpoints accept the signature for anonymous
  requests and fall back to the normal guest-filtered path otherwise
- Guest blacklist keeps a persistent Redis mirror: an expired TTL or an
  unreachable e621 keeps the last successful list instead of falling
  back to the small local list
2026-09-17 11:29:56 -05:00
JakeBreath b71ec729e0 Enrich IQDB candidates; link downloads the e621 original
- IQDB responses carry no preview/file data, so candidates only showed an
  ID; the SPA now enriches them with one batched posts lookup (preview,
  rating, score, favourites, dimensions, tag preview)
- Candidate tiles are selectable instead of instantly resolving: picking
  one shows its info and an explicit 'Link selected post' button
- Linking a post now fetches the e621 original into the library and
  drops the staged upload; when the staged file's MD5 already equals the
  post's file, the staged copy is moved instead (identical bytes)
- Keep the file URL in stored e621 metadata; sanitize the new candidate
  fields server-side
2026-09-17 11:24:01 -05:00
JakeBreath 7deb6084b6 Fix staged upload previews and allow WebP
- Staged files are now served through a signed URL (Django signing, 24h)
  so <img>/<video> tags can load previews without an Authorization
  header; the file endpoint accepts header auth or a valid signature,
  rejects tampered signatures, and still scopes access to the owner
- Serializer responses now carry the request context so URLs are signed
  per user
- Add .webp to the allowed extensions (backend + upload hint)
2026-09-17 11:17:04 -05:00
JakeBreath d0e2901c92 Upload pipeline: staging, MD5 auto-match, IQDB, three-column board
Backend:
- TempUpload model: staged files (pending / visual_match / completed /
  error) with resolution, e621 payload, custom metadata and IQDB data
- Files land in a temp folder and only move into the watched library
  folder once resolved; duplicates resolve immediately without a copy
- Endpoints: stage (multipart), list, retrieve, temp file, IQDB save,
  resolve (link to post or custom metadata), discard/dismiss
- cleanup_temp_uploads command for old staged files
- Replaces the old direct-to-library upload endpoint

Frontend:
- Upload page is now a three-column board (Pending & Unmatched /
  Visual Similarity Detected / Auto-uploaded & Indexed)
- After upload: MD5s are batch-checked against e621 and matches
  auto-complete with full post metadata; remaining files run through
  IQDB and move to the similarity column when candidates exist
- Metadata modal with IQDB candidates, post-ID linking and custom
  tags/rating/notes; discard and dismiss actions
- e621 client gains fetchPostsByMd5 and iqdbSearch helpers

Roadmap updated with the completed upload items.
2026-09-17 11:12:03 -05:00
JakeBreath 60e1113231 Add roadmap with the remaining work
Includes the newly identified gaps: a download progress bar for
Download to Library, and the upload pipeline rework (staging storage,
MD5 auto-match, IQDB/visual similarity, three-column board).
2026-09-17 10:59:18 -05:00
78 changed files with 8881 additions and 267 deletions
+81
View File
@@ -0,0 +1,81 @@
Jake Labs Non-Commercial Software Licence
Copyright (c) 2026 JakeBreath
0. Authorized Distribution and Scope
This Licence applies exclusively to copies of the Software that have been
lawfully obtained from a public distribution channel explicitly designated
by the Licensor (e.g., GitHub, GitLab, Codeberg, Itch.io, Steam, or any
similar public repository or platform) where the Software has been published
by the copyright holder. Any copy of the Software acquired through any other
means - including but not limited to breach of contract, violation of a
non-disclosure agreement, unauthorized access to private repositories,
exploitation of security vulnerabilities, hacking, or accessing a developer's
personal computer, local network, or development device - is not authorized,
and this Licence does not grant any rights whatsoever with respect to such
copies. Such copies receive no licence and remain fully protected by
copyright. No rights under this Licence are conveyed to any recipient of such
an unauthorized copy.
1. Grant of Rights and Conditions
Subject to the terms of this Licence, you are granted a worldwide,
royalty-free, non-exclusive, irrevocable licence to use, copy, modify, merge,
publish, distribute, and sublicense copies of the Software, provided that:
a) Attribution: You must give appropriate credit, provide a link to this
Licence, and indicate if changes were made. You may do so in any
reasonable manner, but not in any way that suggests the Licensor
endorses you or your use.
b) Copyleft: Any modified version of the Software that you distribute must
be licensed as a whole under this same Licence, and you must make the
corresponding source code readily available to recipients under these
same terms. This requirement applies to the modified work as a whole,
including any combined or derivative works that include any portion of
the Software.
c) Strictly Non-Commercial: Commercial use of the Software, in original or
modified form, is prohibited. "Commercial use" includes, but is not
limited to:
(i) selling the Software, including it in a product or service for
which a fee is charged, or using it to provide a paid service;
(ii) using the Software (directly or indirectly) on any website,
platform, application, or online service that displays
advertisements, sponsored content, or other promotional
material intended to generate revenue - including incidental
or automatically injected advertisements;
(iii) using the Software as part of the backend infrastructure,
content delivery network (CDN), or server-side processing of
any website, application, or service that restricts access to
content behind a paywall, subscription, or other payment
mechanism, or that otherwise monetizes content through
advertisements, commercial data collection, or similar means.
Any output, data, or results generated by the Software may not be
used in connection with any such monetized or advertisement-supported
platform.
2. Term and Termination
Your rights under this Licence terminate automatically if you violate any of
its terms. If you cure the violation within 30 days of it occurring, your
rights are reinstated automatically as of the date the violation is cured. On
termination you must stop using and distributing the Software. Termination
does not affect the rights of recipients who obtained copies from you in
compliance with this Licence. The Disclaimer of Warranty survives
termination.
3. Disclaimer of Warranty
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.
+8
View File
@@ -36,3 +36,11 @@ npm run dev # http://localhost:5173, proxies /api to
Not wired up yet — planned: Nginx serving the SPA build, `/media` and `/library` media
directly, and proxying `/api` to Waitress/Django. See `frontend/` design docs for the UI
specification.
## Licence
Source-available, **non-commercial**: personal and other non-commercial use is
welcome under the [Jake Labs Non-Commercial Software Licence](LICENSE), which
requires attribution and keeps derivative works under the same licence.
Commercial use is not permitted. Third-party dependencies keep their own
licences (all permissive: MIT, BSD, Apache-2.0, ISC).
+133
View File
@@ -0,0 +1,133 @@
# J621 Roadmap / To-Dos
Working list of what's still missing, roughly in priority order. Check items off as
they land.
## 1. Library management
- [x] **Duplicates engine**
- [x] Exact MD5 duplicate detection with grouped results
- [x] Perceptual similarity (aHash, dHash, pHash, wHash) with threshold slider
and algorithm toggles (`imagehash` server-side)
- [x] Visual similarity groups: pagination, selection, delete and dismiss
- [x] **Delete & storage page**
- [x] Storage overview (watched folder, media folder, temp)
- [x] Delete by J-ID with preview grid and bulk selection (plus per-copy
deletion of duplicate locations)
- [x] Temp folder cleanup
- [x] **Library search upgrades**
- [x] Search by tags (custom + e621 tags) with a Filename / Tags / Both selector
- [x] Tag cloud in the sidebar (click to search, hidden from guests for
blacklisted items)
- [x] Status filter (matched / custom / unknown — not_found/deleted arrive with
the e621 match cache)
## 2. e621 integration
- [x] **Download progress bar on the detail view**
- [x] Backend download task (status, progress %, downloaded/total bytes, speed)
- [x] Progress endpoint the SPA polls; cancel support
- [x] Frontend progress bar with speed and cancel on `/detail/<post>`
- [x] The status footer's "Active Workers" now counts running download tasks
- [x] **Download to client** — streams the e621 original straight to the
browser (works for guests; host-restricted proxy, no library write)
- [x] **Match local files to e621**
- [x] Match by MD5 from the library detail, plus manual post ID linking
(with an MD5-mismatch warning) and unlink
- [x] Batch cache status for the whole library (`matched` / `not_found` /
`deleted`) via background scan tasks (`/api/matches/`) and
`manage.py match_e621`
- [x] Show match status + e621 metadata in the library: status badge and
match card on the detail, and `not_found` / `deleted` status filters
- [x] **IQDB reverse search**
- [x] Search from a local file (`POST /iqdb_queries.json` from the library detail,
using the signed raw file)
- [x] Show candidate posts to link (thumbnail, rating, score, favs, tags;
select then confirm — same flow as uploads, exact MD5 marked)
- [x] **Follows**
- [x] Follow tags/pools from the `/followed` page (validated against e621,
first feed fetched immediately)
- [x] Followed screens: cover cards (latest image), unseen badges and
Mark seen per follow
- [x] Merged newest-first feed with per-follow filter and unseen-only toggle
- [x] Blacklisted-tag cloud built in a daemon thread, then polled
(10 min TTL; the user's e621 blacklist, guest default as fallback)
- [x] "Followed" nav entry
- [x] Periodic sync commands: `sync_followed_tags` and
`sync_followed_pools` (one e621 search per unique tag/pool)
- [x] **Pools browser** (`/pools` + `/pools/<id>`)
- [x] Index search by name; category, active/deleted and sort filters;
pagination according to the e621 OpenAPI spec
- [x] Cover thumbnails from each pool's first post (one batched post call,
blacklist-aware) and deleted-pool markers
- [x] Pool detail: DText description, posts kept in pool order with chunked
loading and in-library badges, blacklist reveal toggle, Follow button
- [ ] `F` shortcut to favorite a post, tag finder in the Ctrl+K palette
## 3. Uploads & ingestion pipeline
Files now stage first and are resolved before entering the library.
- [x] **Backend staging storage**
- [x] `TempUpload` model: user, md5, filename, temp path, status
(`pending` / `visual_match` / `completed` / `error`), resolution
- [x] Files land in a temp folder first; only completed uploads move into
the watched library folder
- [x] Pending/upload list endpoint, file serving, discard endpoint
- [x] `cleanup_temp_uploads` command for old staged files
- [x] **Auto-upload / auto-match**
- [x] MD5 computed on staging; exact duplicates resolve immediately
- [x] MD5 batch-checked against e621; matches auto-complete with post
metadata stored and rating seeded
- [x] **IQDB similarity on upload** (SPA-driven)
- [x] Automatic + manual IQDB checks with candidate posts
- [x] "Visual Similarity Detected" state with candidate picker
- [x] Perceptual-hash comparison against the library (staged uploads are
flagged with their library matches as soon as they land)
- [x] **Upload UI**
- [x] Three-column board: Pending & Unmatched / Visual Similarity Detected /
Auto-uploaded & Indexed
- [x] Metadata modal (link to e621 post, IQDB candidates, custom metadata)
- [x] Per-file progress plus batch processing indicator
## 4. Staff tools
- [ ] **Optimization modal** (image quality, GIF/APNG resolution+FPS+compression,
video bitrate/presets/two-pass/hw-accel) with original vs processed preview
and override
- [ ] **Stats dashboard**
- [ ] CPU / RAM / GPU / disk usage
- [ ] Active jobs list (this is what makes the footer "Active Workers" real)
- [ ] Live log tail
## 5. Shell & polish
- [x] 18+ entry screen: an age check gates the app before anything renders
(remembered per browser in `localStorage`)
- [ ] Toasts instead of inline messages / confirm dialogs
- [ ] Mobile drawer polish for metadata panels (design spec §layout)
- [x] Profile pictures: staff Users page sets avatars from library J-IDs
(self-service picker in Account still pending)
- [ ] Profile extras (per-user browse preferences)
- [ ] Command palette: tag finder, recent searches
## 6. Infrastructure
- [ ] Guest blacklist refresh on a timer (`refresh_guest_blacklist` via cron/systemd)
- [ ] Follow sync on a timer (`sync_followed_tags` + `sync_followed_pools` via
cron/systemd, e.g. every 30 minutes)
- [ ] Production setup: build the SPA, serve via Nginx (static + `/media` +
`/library`), systemd unit for Waitress
- [ ] Automated tests (backend API + frontend components)
- [ ] Backfill e621 metadata for items downloaded before metadata was stored
(re-download or a "match" action)
## Dependencies / notes
- Duplicates and upload visual-similarity share the perceptual hashing layer
(`imagehash`/`imgdd` server-side + a hash cache table).
- Download progress, optimization jobs and the stats "Active Workers" count all
want the same background-task/progress primitive — design it once.
- IQDB and e621 matching depend on e621 credentials being configured; the SPA
talks to e621 directly for browsing, while the backend e621 client
(`apps/library/e621.py`) handles metadata matching and batch scans.
+8
View File
@@ -0,0 +1,8 @@
from rest_framework.routers import DefaultRouter
from .views import UserViewSet
router = DefaultRouter()
router.register("users", UserViewSet, basename="user")
urlpatterns = router.urls
@@ -0,0 +1,20 @@
# Generated by Django 6.1.1 on 2026-09-17 17:34
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('accounts', '0003_user_role'),
('library', '0006_mediaitem_ahash_mediaitem_dhash_mediaitem_phash_and_more'),
]
operations = [
migrations.AddField(
model_name='user',
name='avatar',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to='library.mediaitem'),
),
]
+7
View File
@@ -15,6 +15,13 @@ class User(AbstractUser):
]
role = models.CharField(max_length=20, choices=ROLE_CHOICES, default=ROLE_USER)
avatar = models.ForeignKey(
"library.MediaItem",
null=True,
blank=True,
on_delete=models.SET_NULL,
related_name="+",
)
e621_username = models.CharField(max_length=100, blank=True, default="")
e621_api_key = models.CharField(max_length=100, blank=True, default="")
e621_base_url = models.CharField(max_length=200, default="https://e621.net")
+70
View File
@@ -1,12 +1,39 @@
from pathlib import Path
from django.contrib.auth.password_validation import validate_password
from django.core import signing
from rest_framework import serializers
from apps.library.services import MEDIA_FILE_SALT, VIDEO_EXTENSIONS
from .models import User
def signed_media_url(request, item):
"""A signed media URL an <img> tag can load for an authenticated user."""
if item is None:
return None
user = getattr(request, "user", None)
if user is None or not getattr(user, "is_authenticated", False):
return None
location = item.locations.first()
is_video = (
location is not None
and Path(location.rel_path).suffix.lower() in VIDEO_EXTENSIONS
)
action = "thumbnail" if is_video else "raw"
signature = signing.dumps(
{"item": item.id, "user": user.id, "action": action},
salt=MEDIA_FILE_SALT,
)
return f"/api/files/J-{item.id}/{action}/?sig={signature}"
class UserSerializer(serializers.ModelSerializer):
e621_configured = serializers.BooleanField(read_only=True)
can_upload = serializers.BooleanField(read_only=True)
avatar_j_id = serializers.SerializerMethodField()
avatar_url = serializers.SerializerMethodField()
class Meta:
model = User
@@ -19,11 +46,54 @@ class UserSerializer(serializers.ModelSerializer):
"date_joined",
"role",
"can_upload",
"avatar_j_id",
"avatar_url",
"e621_username",
"e621_configured",
]
read_only_fields = fields
def get_avatar_j_id(self, obj):
return f"J-{obj.avatar_id}" if obj.avatar_id else None
def get_avatar_url(self, obj):
return signed_media_url(self.context.get("request"), obj.avatar)
class UserListSerializer(serializers.ModelSerializer):
avatar_j_id = serializers.SerializerMethodField()
avatar_url = serializers.SerializerMethodField()
uploads_count = serializers.IntegerField(read_only=True)
class Meta:
model = User
fields = [
"id",
"username",
"email",
"role",
"is_staff",
"is_superuser",
"date_joined",
"avatar_j_id",
"avatar_url",
"uploads_count",
]
read_only_fields = fields
def get_avatar_j_id(self, obj):
return f"J-{obj.avatar_id}" if obj.avatar_id else None
def get_avatar_url(self, obj):
return signed_media_url(self.context.get("request"), obj.avatar)
class UserUpdateSerializer(serializers.Serializer):
avatar_j_id = serializers.CharField(
required=False, allow_blank=True, allow_null=True
)
role = serializers.ChoiceField(choices=User.ROLE_CHOICES, required=False)
class RegisterSerializer(serializers.ModelSerializer):
password = serializers.CharField(write_only=True, validators=[validate_password])
+90 -3
View File
@@ -1,13 +1,19 @@
from rest_framework import status
from rest_framework import mixins, permissions, status, viewsets
from rest_framework.authtoken.models import Token
from rest_framework.permissions import AllowAny, IsAuthenticated
from rest_framework.response import Response
from rest_framework.views import APIView
from django.db.models import Count, Q
from apps.library.models import MediaItem
from .models import User
from .serializers import (
E621CredentialsSerializer,
RegisterSerializer,
UserListSerializer,
UserSerializer,
UserUpdateSerializer,
)
@@ -20,7 +26,10 @@ class RegisterView(APIView):
user = serializer.save()
token, _ = Token.objects.get_or_create(user=user)
return Response(
{"token": token.key, "user": UserSerializer(user).data},
{
"token": token.key,
"user": UserSerializer(user, context={"request": request}).data,
},
status=status.HTTP_201_CREATED,
)
@@ -29,7 +38,7 @@ class MeView(APIView):
permission_classes = [IsAuthenticated]
def get(self, request):
return Response(UserSerializer(request.user).data)
return Response(UserSerializer(request.user, context={"request": request}).data)
class LogoutView(APIView):
@@ -71,3 +80,81 @@ class E621CredentialsView(APIView):
update_fields=["e621_username", "e621_api_key", "e621_base_url"]
)
return Response(self._payload(user))
class IsStaffUser(permissions.BasePermission):
message = "Staff only."
def has_permission(self, request, view):
user = request.user
return bool(
user
and user.is_authenticated
and (
user.is_superuser
or user.is_staff
or user.role == user.ROLE_STAFF
)
)
class UserViewSet(
mixins.ListModelMixin,
mixins.RetrieveModelMixin,
viewsets.GenericViewSet,
):
"""Staff user directory: roles and J-ID avatars."""
permission_classes = [IsStaffUser]
http_method_names = ["get", "patch", "head", "options"]
def get_queryset(self):
queryset = User.objects.annotate(uploads_count=Count("uploads")).order_by("id")
search = self.request.query_params.get("search", "").strip()
if search:
queryset = queryset.filter(
Q(username__icontains=search) | Q(email__icontains=search)
)
return queryset
def get_serializer_class(self):
if self.request.method in {"PATCH", "PUT"}:
return UserUpdateSerializer
return UserListSerializer
def update(self, request, *args, **kwargs):
user = self.get_object()
serializer = UserUpdateSerializer(data=request.data)
serializer.is_valid(raise_exception=True)
data = serializer.validated_data
update_fields = []
if "role" in data:
user.role = data["role"]
update_fields.append("role")
if "avatar_j_id" in data:
value = str(data.get("avatar_j_id") or "").strip()
if not value:
user.avatar = None
else:
numeric = value[2:] if value.upper().startswith("J-") else value
item = (
MediaItem.objects.filter(pk=int(numeric)).first()
if numeric.isdigit()
else None
)
if item is None:
return Response(
{"detail": f"No library item {value}."},
status=status.HTTP_400_BAD_REQUEST,
)
user.avatar = item
update_fields.append("avatar")
if update_fields:
user.save(update_fields=update_fields)
return Response(
UserListSerializer(user, context=self.get_serializer_context()).data
)
partial_update = update
+10 -5
View File
@@ -9,11 +9,16 @@ from .system_info import get_os_info
def _worker_counts():
"""Active/queued background jobs.
"""Active/queued background jobs (downloads and e621 match scans)."""
from apps.library.models import DownloadTask, MatchTask
No task pipeline exists yet, so this reports honest zeros for now.
"""
return {"active": 0, "queued": 0}
active = DownloadTask.objects.filter(
status=DownloadTask.STATUS_DOWNLOADING
).count() + MatchTask.objects.filter(status=MatchTask.STATUS_RUNNING).count()
queued = DownloadTask.objects.filter(
status=DownloadTask.STATUS_PENDING
).count() + MatchTask.objects.filter(status=MatchTask.STATUS_PENDING).count()
return {"active": active, "queued": queued}
class StatusView(APIView):
@@ -30,7 +35,7 @@ class StatusView(APIView):
"version": settings.APP_VERSION,
"os": get_os_info(),
"watched_folder": settings.WATCHED_FOLDER,
"e621_time_ms": None, # no e621 calls in the backend yet
"e621_time_ms": None, # e621 latency is measured client-side
"workers": _worker_counts(),
}
payload["server_time_ms"] = round((time.perf_counter() - started) * 1000, 1)
View File
+5
View File
@@ -0,0 +1,5 @@
from django.apps import AppConfig
class FollowsConfig(AppConfig):
name = "apps.follows"
+97
View File
@@ -0,0 +1,97 @@
"""Blacklisted-tag cloud for the Followed page.
The cloud counts tags from the user's e621 blacklist that appear in their
followed feeds. Building it means one e621 request (the blacklist itself)
plus a local scan of stored feed posts, so it runs in a daemon thread and
the page polls until the fresh result lands.
"""
import logging
import threading
from datetime import timedelta
from fnmatch import fnmatch
from django.contrib.auth import get_user_model
from django.db import connection
from django.utils import timezone
from apps.library import e621, guest_filter
from .models import FollowCloud, FollowedPost
logger = logging.getLogger(__name__)
CLOUD_TTL = timedelta(minutes=10)
BUILD_STALE = timedelta(minutes=5)
MAX_CLOUD_TAGS = 60
def blacklist_patterns_for(user):
"""(patterns, source): the user's e621 blacklist, else the guest default."""
if e621.credentials_configured(user):
try:
me = e621.get(user, "/users/me.json")
lines = str(me.get("blacklisted_tags") or "").splitlines()
patterns = guest_filter.parse_blacklist_patterns(lines)
if patterns:
return patterns, "user"
except e621.E621Error as exc:
logger.warning("Could not fetch the e621 blacklist for %s: %s", user, exc)
return guest_filter.guest_blacklist_patterns(), "default"
def compute_cloud(user):
"""Count blacklist matches across every stored feed post of one user."""
patterns, source = blacklist_patterns_for(user)
counts = {}
seen_posts = set()
rows = FollowedPost.objects.filter(user=user).values_list("post_id", "data")
for post_id, data in rows:
if post_id in seen_posts:
continue
seen_posts.add(post_id)
if not isinstance(data, dict):
continue
categories = data.get("tags")
if not isinstance(categories, dict):
continue
for values in categories.values():
if not isinstance(values, list):
continue
for tag in values:
name = str(tag).strip().lower()
if name and any(fnmatch(name, pattern) for pattern in patterns):
counts[name] = counts.get(name, 0) + 1
tags = sorted(counts.items(), key=lambda entry: (-entry[1], entry[0]))
return {
"tags": [[tag, count] for tag, count in tags[:MAX_CLOUD_TAGS]],
"posts": len(seen_posts),
"source": source,
"blacklist_count": len(patterns),
}
def start_cloud_build(user_id):
thread = threading.Thread(target=run_cloud_build, args=(user_id,), daemon=True)
thread.start()
def run_cloud_build(user_id):
User = get_user_model()
try:
user = User.objects.filter(id=user_id).first()
if user is None:
return
data = compute_cloud(user)
FollowCloud.objects.filter(user_id=user_id).update(
data=data, computed_at=timezone.now()
)
except Exception: # noqa: BLE001 - background builds must not crash workers
logger.exception("Blacklist cloud build failed for user %s", user_id)
finally:
FollowCloud.objects.filter(user_id=user_id).update(
building=False, building_since=None
)
# Background threads hold their own DB connection; release it so
# Gunicorn workers do not leak connections when threads finish.
connection.close()
@@ -0,0 +1,62 @@
from django.core.management.base import BaseCommand, CommandError
from apps.follows import sync
from apps.follows.models import FollowedPool
from apps.library import e621
class Command(BaseCommand):
help = (
"Fetch new posts for every followed pool: fills each feed, refreshes "
"the card cover (latest image), pool name/count and unseen badges."
)
def add_arguments(self, parser):
parser.add_argument(
"--user",
help="Only sync follows belonging to this username "
"(default: every user's follows).",
)
parser.add_argument(
"--limit",
type=int,
default=sync.FEED_LIMIT,
help=f"Posts fetched per pool (default {sync.FEED_LIMIT}).",
)
def handle(self, *args, **options):
username = options.get("user")
if username and sync.preferred_fetch_user(username) is None:
raise CommandError(f"No such user: {username}")
fetch_user = sync.preferred_fetch_user(username)
follows = FollowedPool.objects.select_related("user").order_by(
"pool_id", "user_id"
)
if username:
follows = follows.filter(user__username=username)
follows = list(follows)
grouped = {}
for follow in follows:
grouped.setdefault(follow.pool_id, []).append(follow)
created = 0
for pool_id, group in grouped.items():
try:
posts = sync.search_posts(
fetch_user, f"pool:{pool_id}", options["limit"]
)
except e621.E621Error as exc:
raise CommandError(f"Aborted at pool #{pool_id}: {exc}")
for follow in group:
created += sync.sync_pool_follow(
follow, fetch_user, posts=posts
)
identity = getattr(fetch_user, "username", None) or "anonymous"
self.stdout.write(
self.style.SUCCESS(
f"Synced {len(grouped)} pool(s) across {len(follows)} follow(s) "
f"as {identity}: {created} new post(s)."
)
)
@@ -0,0 +1,60 @@
from django.core.management.base import BaseCommand, CommandError
from apps.follows import sync
from apps.follows.models import FollowedTag
from apps.library import e621
class Command(BaseCommand):
help = (
"Fetch new posts for every followed tag: fills each feed, refreshes "
"the card cover (latest image) and unseen badges."
)
def add_arguments(self, parser):
parser.add_argument(
"--user",
help="Only sync follows belonging to this username "
"(default: every user's follows).",
)
parser.add_argument(
"--limit",
type=int,
default=sync.FEED_LIMIT,
help=f"Posts fetched per tag (default {sync.FEED_LIMIT}).",
)
def handle(self, *args, **options):
username = options.get("user")
if username and sync.preferred_fetch_user(username) is None:
raise CommandError(f"No such user: {username}")
fetch_user = sync.preferred_fetch_user(username)
follows = FollowedTag.objects.select_related("user").order_by(
"tag", "user_id"
)
if username:
follows = follows.filter(user__username=username)
follows = list(follows)
grouped = {}
for follow in follows:
grouped.setdefault(follow.tag, []).append(follow)
created = 0
for tag, group in grouped.items():
try:
posts = sync.search_posts(fetch_user, tag, options["limit"])
except e621.E621Error as exc:
raise CommandError(f"Aborted at tag '{tag}': {exc}")
for follow in group:
created += sync.sync_tag_follow(
follow, fetch_user, posts=posts
)
identity = getattr(fetch_user, "username", None) or "anonymous"
self.stdout.write(
self.style.SUCCESS(
f"Synced {len(grouped)} tag(s) across {len(follows)} follow(s) "
f"as {identity}: {created} new post(s)."
)
)
@@ -0,0 +1,94 @@
# Generated by Django 6.1.1 on 2026-09-17 19:04
import django.db.models.deletion
from django.conf import settings
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name='FollowCloud',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('data', models.JSONField(blank=True, null=True)),
('computed_at', models.DateTimeField(blank=True, null=True)),
('building', models.BooleanField(default=False)),
('building_since', models.DateTimeField(blank=True, null=True)),
('user', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='follow_cloud', to=settings.AUTH_USER_MODEL)),
],
),
migrations.CreateModel(
name='FollowedPool',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('pool_id', models.IntegerField()),
('name', models.CharField(blank=True, default='', max_length=200)),
('post_count', models.IntegerField(default=0)),
('cover_data', models.JSONField(blank=True, null=True)),
('last_synced_at', models.DateTimeField(blank=True, null=True)),
('created_at', models.DateTimeField(auto_now_add=True)),
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='followed_pools', to=settings.AUTH_USER_MODEL)),
],
options={
'ordering': ['pool_id'],
},
),
migrations.CreateModel(
name='FollowedTag',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('tag', models.CharField(max_length=200)),
('cover_data', models.JSONField(blank=True, null=True)),
('last_synced_at', models.DateTimeField(blank=True, null=True)),
('created_at', models.DateTimeField(auto_now_add=True)),
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='followed_tags', to=settings.AUTH_USER_MODEL)),
],
options={
'ordering': ['tag'],
},
),
migrations.CreateModel(
name='FollowedPost',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('post_id', models.IntegerField(db_index=True)),
('data', models.JSONField(default=dict)),
('seen', models.BooleanField(db_index=True, default=False)),
('created_at', models.DateTimeField(auto_now_add=True)),
('pool', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='posts', to='follows.followedpool')),
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='followed_posts', to=settings.AUTH_USER_MODEL)),
('tag', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='posts', to='follows.followedtag')),
],
options={
'ordering': ['-post_id'],
},
),
migrations.AddConstraint(
model_name='followedpool',
constraint=models.UniqueConstraint(fields=('user', 'pool_id'), name='uniq_followed_pool_per_user'),
),
migrations.AddConstraint(
model_name='followedtag',
constraint=models.UniqueConstraint(fields=('user', 'tag'), name='uniq_followed_tag_per_user'),
),
migrations.AddIndex(
model_name='followedpost',
index=models.Index(fields=['user', 'post_id'], name='follows_fol_user_id_47fdab_idx'),
),
migrations.AddConstraint(
model_name='followedpost',
constraint=models.UniqueConstraint(fields=('tag', 'post_id'), name='uniq_followed_tag_post'),
),
migrations.AddConstraint(
model_name='followedpost',
constraint=models.UniqueConstraint(fields=('pool', 'post_id'), name='uniq_followed_pool_post'),
),
]
+114
View File
@@ -0,0 +1,114 @@
from django.conf import settings
from django.db import models
class FollowedTag(models.Model):
"""A tag one user subscribes to; feeds are filled by the sync command."""
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.CASCADE,
related_name="followed_tags",
)
tag = models.CharField(max_length=200)
cover_data = models.JSONField(null=True, blank=True)
last_synced_at = models.DateTimeField(null=True, blank=True)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["tag"]
constraints = [
models.UniqueConstraint(
fields=["user", "tag"], name="uniq_followed_tag_per_user"
),
]
def __str__(self):
return f"{self.user} follows tag:{self.tag}"
class FollowedPool(models.Model):
"""A pool one user subscribes to; feeds are filled by the sync command."""
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.CASCADE,
related_name="followed_pools",
)
pool_id = models.IntegerField()
name = models.CharField(max_length=200, blank=True, default="")
post_count = models.IntegerField(default=0)
cover_data = models.JSONField(null=True, blank=True)
last_synced_at = models.DateTimeField(null=True, blank=True)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["pool_id"]
constraints = [
models.UniqueConstraint(
fields=["user", "pool_id"], name="uniq_followed_pool_per_user"
),
]
def __str__(self):
return f"{self.user} follows pool:{self.pool_id}"
class FollowedPost(models.Model):
"""One post seen in a followed feed (stored once per follow source)."""
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.CASCADE,
related_name="followed_posts",
)
tag = models.ForeignKey(
FollowedTag,
null=True,
blank=True,
on_delete=models.CASCADE,
related_name="posts",
)
pool = models.ForeignKey(
FollowedPool,
null=True,
blank=True,
on_delete=models.CASCADE,
related_name="posts",
)
post_id = models.IntegerField(db_index=True)
data = models.JSONField(default=dict)
seen = models.BooleanField(default=False, db_index=True)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["-post_id"]
indexes = [models.Index(fields=["user", "post_id"])]
constraints = [
models.UniqueConstraint(
fields=["tag", "post_id"], name="uniq_followed_tag_post"
),
models.UniqueConstraint(
fields=["pool", "post_id"], name="uniq_followed_pool_post"
),
]
def __str__(self):
return f"#{self.post_id} for {self.user}"
class FollowCloud(models.Model):
"""Cached blacklisted-tag cloud for the Followed page, built in a thread."""
user = models.OneToOneField(
settings.AUTH_USER_MODEL,
on_delete=models.CASCADE,
related_name="follow_cloud",
)
data = models.JSONField(null=True, blank=True)
computed_at = models.DateTimeField(null=True, blank=True)
building = models.BooleanField(default=False)
building_since = models.DateTimeField(null=True, blank=True)
def __str__(self):
return f"Follow cloud for {self.user}"
+53
View File
@@ -0,0 +1,53 @@
from rest_framework import serializers
from .models import FollowedPool, FollowedTag
class FollowedTagSerializer(serializers.ModelSerializer):
unseen_count = serializers.IntegerField(read_only=True)
class Meta:
model = FollowedTag
fields = [
"id",
"tag",
"unseen_count",
"cover_data",
"last_synced_at",
"created_at",
]
read_only_fields = [
"id",
"tag",
"unseen_count",
"cover_data",
"last_synced_at",
"created_at",
]
class FollowedPoolSerializer(serializers.ModelSerializer):
unseen_count = serializers.IntegerField(read_only=True)
class Meta:
model = FollowedPool
fields = [
"id",
"pool_id",
"name",
"post_count",
"unseen_count",
"cover_data",
"last_synced_at",
"created_at",
]
read_only_fields = [
"id",
"pool_id",
"name",
"post_count",
"unseen_count",
"cover_data",
"last_synced_at",
"created_at",
]
+171
View File
@@ -0,0 +1,171 @@
"""Fetch and store followed tag/pool feeds from e621.
Used by the two periodic management commands (`sync_followed_tags`,
`sync_followed_pools`) and by the follow API for the first fill of a new
subscription. One e621 search per unique tag/pool is shared across every
user following it.
"""
import logging
from django.contrib.auth import get_user_model
from django.utils import timezone
from apps.library import e621
from apps.library.services import trim_e621_post
from .models import FollowedPool, FollowedPost, FollowedTag
logger = logging.getLogger(__name__)
FEED_LIMIT = 24
PRUNE_KEEP = 200
def preferred_fetch_user(username=None):
"""A configured account for sync calls; None means run anonymously.
e621 serves post searches and pool reads anonymously, so a missing
account is not fatal — credentials mainly raise rate limits.
"""
User = get_user_model()
if username:
return User.objects.filter(username=username).first()
return (
User.objects.filter(is_staff=True)
.exclude(e621_username="")
.exclude(e621_api_key="")
.order_by("id")
.first()
)
def search_posts(user, term, limit=FEED_LIMIT):
"""Newest posts for a search term (``tag`` or ``pool:<id>``)."""
payload = e621.get(
user,
"/posts.json",
params={"tags": term, "limit": limit},
require_auth=False,
)
posts = payload.get("posts") if isinstance(payload, dict) else None
return posts or []
def cover_payload(post):
"""Lightweight card cover: enough to render a thumbnail and rating."""
if not isinstance(post, dict):
return None
preview = post.get("preview") or {}
score = post.get("score") or {}
return {
"post_id": post.get("id"),
"rating": post.get("rating"),
"preview_url": preview.get("url"),
"score": score.get("total"),
"fav_count": post.get("fav_count"),
}
def feed_payload(post):
"""Compact stored post for the feed (tags included for the tag cloud)."""
trimmed = trim_e621_post(post)
if trimmed is None:
return None
return {
"id": trimmed.get("id"),
"created_at": trimmed.get("created_at"),
"rating": trimmed.get("rating"),
"tags": trimmed.get("tags") or {},
"score": trimmed.get("score") or {},
"fav_count": trimmed.get("fav_count"),
"file": trimmed.get("file") or {},
"preview": trimmed.get("preview") or {},
"uploader_name": trimmed.get("uploader_name"),
}
def _store_posts(follow, posts, tag=None, pool=None):
"""Insert unseen rows for new posts. Returns how many were added."""
existing = set(follow.posts.values_list("post_id", flat=True))
rows = []
for post in posts:
post_id = post.get("id") if isinstance(post, dict) else None
if not isinstance(post_id, int) or post_id in existing:
continue
data = feed_payload(post)
if data is None:
continue
rows.append(
FollowedPost(
user=follow.user,
post_id=post_id,
data=data,
tag=tag,
pool=pool,
)
)
if rows:
FollowedPost.objects.bulk_create(rows, ignore_conflicts=True)
_prune(follow)
return len(rows)
def _prune(follow):
"""Keep only the newest PRUNE_KEEP posts per follow."""
keep = list(
follow.posts.order_by("-post_id").values_list("id", flat=True)[
:PRUNE_KEEP
]
)
follow.posts.exclude(id__in=keep).delete()
def sync_tag_follow(follow: FollowedTag, fetch_user=None, limit=FEED_LIMIT, posts=None):
"""Fetch new posts for one followed tag; updates cover and sync time."""
if posts is None:
posts = search_posts(fetch_user, follow.tag, limit)
added = _store_posts(follow, posts, tag=follow)
updates = ["last_synced_at"]
cover = cover_payload(posts[0]) if posts else None
if cover is not None:
follow.cover_data = cover
updates.append("cover_data")
follow.last_synced_at = timezone.now()
follow.save(update_fields=updates)
return added
def sync_pool_follow(
follow: FollowedPool,
fetch_user=None,
limit=FEED_LIMIT,
posts=None,
refresh_meta=True,
):
"""Fetch new posts for one followed pool; updates cover and metadata."""
if posts is None:
posts = search_posts(fetch_user, f"pool:{follow.pool_id}", limit)
added = _store_posts(follow, posts, pool=follow)
updates = ["last_synced_at"]
cover = cover_payload(posts[0]) if posts else None
if cover is not None:
follow.cover_data = cover
updates.append("cover_data")
if refresh_meta or not follow.name:
try:
pool = e621.get(
fetch_user,
f"/pools/{follow.pool_id}.json",
require_auth=False,
)
except e621.E621Error as exc:
logger.warning("Could not refresh pool %s: %s", follow.pool_id, exc)
else:
if isinstance(pool, dict) and pool.get("id"):
follow.name = str(pool.get("name") or "")[:200]
follow.post_count = int(pool.get("post_count") or 0)
updates += ["name", "post_count"]
follow.last_synced_at = timezone.now()
follow.save(update_fields=updates)
return added
+25
View File
@@ -0,0 +1,25 @@
from django.urls import include, path
from rest_framework.routers import DefaultRouter
from .views import (
FollowCloudView,
FollowedFeedView,
FollowedPoolViewSet,
FollowedPostSeenView,
FollowedTagViewSet,
)
router = DefaultRouter()
router.register("follows/tags", FollowedTagViewSet, basename="followed-tag")
router.register("follows/pools", FollowedPoolViewSet, basename="followed-pool")
urlpatterns = [
path("", include(router.urls)),
path("follows/feed/", FollowedFeedView.as_view(), name="followed_feed"),
path("follows/cloud/", FollowCloudView.as_view(), name="followed_cloud"),
path(
"follows/posts/<int:post_id>/seen/",
FollowedPostSeenView.as_view(),
name="followed_post_seen",
),
]
+317
View File
@@ -0,0 +1,317 @@
import re
from django.db.models import Count, Q
from django.utils import timezone
from rest_framework import mixins, status, viewsets
from rest_framework.decorators import action
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from rest_framework.views import APIView
from apps.library import e621
from . import cloud as cloud_service
from . import sync
from .models import FollowCloud, FollowedPool, FollowedPost, FollowedTag
from .serializers import FollowedPoolSerializer, FollowedTagSerializer
POOL_URL_RE = re.compile(r"/pools/(\d+)")
def _parse_page(request, default_per_page=48):
try:
per_page = int(request.query_params.get("per_page", default_per_page))
except ValueError:
per_page = default_per_page
try:
page = int(request.query_params.get("page", 1))
except ValueError:
page = 1
return max(1, page), min(200, max(1, per_page))
class FollowedTagViewSet(
mixins.ListModelMixin,
mixins.CreateModelMixin,
mixins.DestroyModelMixin,
viewsets.GenericViewSet,
):
"""Tags the current user follows."""
serializer_class = FollowedTagSerializer
permission_classes = [IsAuthenticated]
http_method_names = ["get", "post", "delete", "head", "options"]
def get_queryset(self):
return (
FollowedTag.objects.filter(user=self.request.user)
.annotate(
unseen_count=Count("posts", filter=Q(posts__seen=False), distinct=True)
)
.order_by("tag")
)
def create(self, request):
tag = str(request.data.get("tag") or "").strip().lower()[:200]
if not tag:
return Response(
{"detail": "A tag is required."}, status=status.HTTP_400_BAD_REQUEST
)
if FollowedTag.objects.filter(user=request.user, tag=tag).exists():
return Response(
{"detail": f"Already following '{tag}'."},
status=status.HTTP_400_BAD_REQUEST,
)
try:
results = e621.get(
request.user,
"/tags.json",
params={"search[name_matches]": tag, "limit": 5},
require_auth=False,
)
except e621.E621Error as exc:
return Response(
{"detail": str(exc)}, status=status.HTTP_502_BAD_GATEWAY
)
names = {
str(entry.get("name", "")).lower()
for entry in results
if isinstance(entry, dict)
}
if tag not in names:
return Response(
{"detail": f"e621 has no tag named '{tag}'."},
status=status.HTTP_400_BAD_REQUEST,
)
follow = FollowedTag.objects.create(user=request.user, tag=tag)
FollowCloud.objects.filter(user=request.user).update(computed_at=None)
try:
sync.sync_tag_follow(follow, request.user)
except e621.E621Error:
# The periodic command will catch up; the follow still stands.
pass
follow = self.get_queryset().get(pk=follow.pk)
return Response(
self.get_serializer(follow).data, status=status.HTTP_201_CREATED
)
def perform_destroy(self, instance):
instance.delete()
FollowCloud.objects.filter(user=self.request.user).update(computed_at=None)
@action(detail=True, methods=["post"])
def seen(self, request, pk=None):
follow = self.get_object()
follow.posts.update(seen=True)
return Response({"success": True})
class FollowedPoolViewSet(
mixins.ListModelMixin,
mixins.CreateModelMixin,
mixins.DestroyModelMixin,
viewsets.GenericViewSet,
):
"""Pools the current user follows."""
serializer_class = FollowedPoolSerializer
permission_classes = [IsAuthenticated]
http_method_names = ["get", "post", "delete", "head", "options"]
def get_queryset(self):
return (
FollowedPool.objects.filter(user=self.request.user)
.annotate(
unseen_count=Count("posts", filter=Q(posts__seen=False), distinct=True)
)
.order_by("pool_id")
)
def create(self, request):
raw = str(
request.data.get("pool") or request.data.get("pool_id") or ""
).strip()
match = POOL_URL_RE.search(raw)
if match:
pool_id = int(match.group(1))
elif raw.isdigit():
pool_id = int(raw)
else:
return Response(
{"detail": "Enter an e621 pool ID or pool URL."},
status=status.HTTP_400_BAD_REQUEST,
)
if FollowedPool.objects.filter(user=request.user, pool_id=pool_id).exists():
return Response(
{"detail": f"Already following pool #{pool_id}."},
status=status.HTTP_400_BAD_REQUEST,
)
try:
pool = e621.get(
request.user, f"/pools/{pool_id}.json", require_auth=False
)
except e621.E621NotFound:
return Response(
{"detail": f"e621 has no pool #{pool_id}."},
status=status.HTTP_400_BAD_REQUEST,
)
except e621.E621Error as exc:
return Response(
{"detail": str(exc)}, status=status.HTTP_502_BAD_GATEWAY
)
follow = FollowedPool.objects.create(
user=request.user,
pool_id=pool_id,
name=str(pool.get("name") or "")[:200],
post_count=int(pool.get("post_count") or 0),
)
FollowCloud.objects.filter(user=request.user).update(computed_at=None)
try:
sync.sync_pool_follow(follow, request.user, refresh_meta=False)
except e621.E621Error:
pass
follow = self.get_queryset().get(pk=follow.pk)
return Response(
self.get_serializer(follow).data, status=status.HTTP_201_CREATED
)
def perform_destroy(self, instance):
instance.delete()
FollowCloud.objects.filter(user=self.request.user).update(computed_at=None)
@action(detail=True, methods=["post"])
def seen(self, request, pk=None):
follow = self.get_object()
follow.posts.update(seen=True)
return Response({"success": True})
class FollowedFeedView(APIView):
"""Merged newest-first feed across followed tags and pools."""
permission_classes = [IsAuthenticated]
def get(self, request):
page, per_page = _parse_page(request)
queryset = FollowedPost.objects.filter(user=request.user).order_by(
"-post_id"
)
tag_id = request.query_params.get("tag", "")
pool_id = request.query_params.get("pool", "")
if tag_id.isdigit():
queryset = queryset.filter(tag_id=int(tag_id))
if pool_id.isdigit():
queryset = queryset.filter(pool_id=int(pool_id))
if request.query_params.get("unseen") == "1":
queryset = queryset.filter(seen=False)
tag_names = dict(
FollowedTag.objects.filter(user=request.user).values_list("id", "tag")
)
pool_names = dict(
FollowedPool.objects.filter(user=request.user).values_list(
"id", "name"
)
)
entries = {}
for post_id, seen, data, tag, pool in queryset.values_list(
"post_id", "seen", "data", "tag_id", "pool_id"
)[:2000]:
entry = entries.get(post_id)
if entry is None:
entry = entries[post_id] = {
"post_id": post_id,
"seen": seen,
"data": data,
"sources": [],
}
else:
entry["seen"] = entry["seen"] and seen
if tag and not any(
source["kind"] == "tag" and source["id"] == tag
for source in entry["sources"]
):
entry["sources"].append(
{
"kind": "tag",
"id": tag,
"label": tag_names.get(tag, f"tag:{tag}"),
}
)
if pool and not any(
source["kind"] == "pool" and source["id"] == pool
for source in entry["sources"]
):
entry["sources"].append(
{
"kind": "pool",
"id": pool,
"label": pool_names.get(pool) or f"Pool #{pool}",
}
)
items = list(entries.values())
count = len(items)
start = (page - 1) * per_page
return Response(
{
"count": count,
"page": page,
"per_page": per_page,
"has_next": start + per_page < count,
"results": items[start : start + per_page],
}
)
class FollowedPostSeenView(APIView):
"""Mark one post seen across every follow it appears in."""
permission_classes = [IsAuthenticated]
def post(self, request, post_id):
FollowedPost.objects.filter(user=request.user, post_id=post_id).update(
seen=True
)
return Response({"success": True})
class FollowCloudView(APIView):
"""The blacklisted-tag cloud, rebuilt in a thread when stale."""
permission_classes = [IsAuthenticated]
def get(self, request):
cloud, _ = FollowCloud.objects.get_or_create(user=request.user)
now = timezone.now()
stale = cloud.computed_at is None or now - cloud.computed_at > (
cloud_service.CLOUD_TTL
)
if stale:
reclaim = Q(building=False) | Q(
building_since__lt=now - cloud_service.BUILD_STALE
)
started = (
FollowCloud.objects.filter(pk=cloud.pk)
.filter(reclaim)
.update(building=True, building_since=now)
)
if started:
cloud_service.start_cloud_build(request.user.id)
cloud.refresh_from_db()
building = cloud.building and not (
cloud.building_since
and now - cloud.building_since > cloud_service.BUILD_STALE
)
data = cloud.data or {}
return Response(
{
"status": "building" if building else "ready",
"stale": stale,
"tags": data.get("tags", []),
"posts": data.get("posts", 0),
"source": data.get("source"),
"blacklist_count": data.get("blacklist_count", 0),
"computed_at": cloud.computed_at,
}
)
+157
View File
@@ -0,0 +1,157 @@
"""Background 'Download to Library' jobs with progress tracking."""
import logging
import threading
import time
from datetime import timedelta
from pathlib import Path
from urllib.parse import urlparse
from django.conf import settings
from django.db import connection
from django.utils import timezone
from . import services
from .models import DownloadTask
logger = logging.getLogger(__name__)
STALE_AFTER = timedelta(minutes=30)
def reap_stale_downloads():
"""Mark tasks left hanging by a recycled worker as failed.
Gunicorn recycles workers (--max-requests, timeouts); a download thread
dies with its worker, so long-stuck tasks are surfaced as errors instead
of pretending to run forever.
"""
cutoff = timezone.now() - STALE_AFTER
return DownloadTask.objects.filter(
status__in=[DownloadTask.STATUS_PENDING, DownloadTask.STATUS_DOWNLOADING],
updated_at__lt=cutoff,
).update(
status=DownloadTask.STATUS_ERROR,
error="The worker restarted before this download finished.",
speed=None,
updated_at=timezone.now(),
)
def start_download_task(task_id):
thread = threading.Thread(target=run_download_task, args=(task_id,), daemon=True)
thread.start()
def run_download_task(task_id):
task = DownloadTask.objects.filter(id=task_id).first()
if task is None:
return
folder = Path(settings.WATCHED_FOLDER)
name = (
task.filename
or Path(urlparse(task.url).path).name
or f"post-{task.post_id or 'download'}"
)
destination = services.unique_destination(folder, name)
state = {
"progress_at": 0.0,
"bytes_at": 0,
"cancel_at": 0.0,
"cancelled": False,
}
def should_cancel():
now = time.monotonic()
if now - state["cancel_at"] >= 1.0:
state["cancelled"] = DownloadTask.objects.filter(
id=task_id, cancelled=True
).exists()
state["cancel_at"] = now
return state["cancelled"]
def on_progress(downloaded, total):
now = time.monotonic()
if now - state["progress_at"] < 0.5 and (total == 0 or downloaded < total):
return
elapsed = max(now - state["progress_at"], 0.001)
speed = None
if state["bytes_at"] > 0 and downloaded >= state["bytes_at"]:
speed = (downloaded - state["bytes_at"]) / elapsed
state["progress_at"] = now
state["bytes_at"] = downloaded
DownloadTask.objects.filter(id=task_id).update(
downloaded=downloaded,
total=total,
progress=int(downloaded * 100 / total) if total else 0,
speed=speed,
updated_at=timezone.now(),
)
DownloadTask.objects.filter(id=task_id).update(
status=DownloadTask.STATUS_DOWNLOADING, updated_at=timezone.now()
)
try:
services.download_file(
task.url,
destination,
progress_callback=on_progress,
should_cancel=should_cancel,
)
item, _, location, _ = services.index_file(destination, folder)
services.rename_location_to_j_id(item, location)
services.ensure_visual_hashes(item)
update_fields = []
if item.uploaded_by_id is None and task.user_id is not None:
item.uploaded_by = task.user
update_fields.append("uploaded_by")
if task.e621_data:
item.e621_post_id = task.post_id
item.e621_data = task.e621_data
item.e621_match_status = MediaItem.E621_MATCHED
item.e621_checked_at = timezone.now()
update_fields += [
"e621_post_id",
"e621_data",
"e621_match_status",
"e621_checked_at",
]
rating = (task.e621_data or {}).get("rating")
if not item.rating and rating in {"s", "q", "e"}:
item.rating = rating
update_fields.append("rating")
if update_fields:
item.save(update_fields=update_fields + ["updated_at"])
DownloadTask.objects.filter(id=task_id).update(
status=DownloadTask.STATUS_COMPLETE,
progress=100,
speed=None,
library_item=item,
updated_at=timezone.now(),
)
except services.DownloadCancelled:
destination.unlink(missing_ok=True)
DownloadTask.objects.filter(id=task_id).update(
status=DownloadTask.STATUS_CANCELLED,
progress=0,
downloaded=0,
speed=None,
updated_at=timezone.now(),
)
except Exception as exc: # noqa: BLE001 - report background failures
destination.unlink(missing_ok=True)
logger.exception("Download task %s failed", task_id)
DownloadTask.objects.filter(id=task_id).update(
status=DownloadTask.STATUS_ERROR,
error=str(exc),
speed=None,
updated_at=timezone.now(),
)
finally:
# Background threads hold their own DB connection; release it so
# Gunicorn workers do not leak connections when threads finish.
connection.close()
+93
View File
@@ -0,0 +1,93 @@
"""Minimal e621 API client for server-side matching and metadata refresh.
The SPA talks to e621 directly for browsing; this client exists for work the
browser cannot do reliably: long batch scans, and requests tied to a library
item rather than an open page. It uses the requesting user's stored
credentials and a global throttle (e621 asks for at most two requests per
second).
"""
import threading
import time
import requests
from django.conf import settings
REQUEST_INTERVAL = 0.5 # seconds between requests, per process
class E621Error(Exception):
"""A non-404 failure while talking to e621."""
class E621NotFound(E621Error):
"""The requested post does not exist (HTTP 404)."""
_throttle_lock = threading.Lock()
_last_request_at = 0.0
def credentials_configured(user):
return bool(user is not None and getattr(user, "e621_configured", False))
def _wait_for_slot():
global _last_request_at
with _throttle_lock:
delay = _last_request_at + REQUEST_INTERVAL - time.monotonic()
if delay > 0:
time.sleep(delay)
_last_request_at = time.monotonic()
def get(user, path, params=None, timeout=30, require_auth=True):
"""GET an e621 API path using the user's credentials.
Reads that e621 serves anonymously (searches, pools, tags) can pass
require_auth=False; matching endpoints keep requiring credentials.
Raises E621NotFound for 404s and E621Error for everything else that isn't
a 2xx, so callers never see requests exceptions.
"""
configured = credentials_configured(user)
if require_auth and not configured:
raise E621Error("Configure your e621 credentials in Account first.")
base = (getattr(user, "e621_base_url", "") or "https://e621.net").rstrip("/")
_wait_for_slot()
try:
response = requests.get(
f"{base}{path}",
params=params,
auth=(user.e621_username, user.e621_api_key) if configured else None,
headers={"User-Agent": settings.USER_AGENT},
timeout=timeout,
)
except requests.RequestException as exc:
raise E621Error(f"Could not reach e621: {exc}") from exc
if response.status_code == 404:
raise E621NotFound(f"e621 returned 404 for {path}")
if response.status_code >= 400:
raise E621Error(f"e621 replied {response.status_code} for {path}")
try:
return response.json()
except ValueError as exc:
raise E621Error("e621 returned an unexpected response.") from exc
def find_post_by_md5(user, md5):
"""The e621 post with this exact MD5, or None."""
payload = get(user, "/posts.json", params={"tags": f"md5:{md5}", "limit": 1})
posts = payload.get("posts") if isinstance(payload, dict) else None
if not posts:
return None
return posts[0]
def fetch_post(user, post_id):
"""One post by id. Raises E621NotFound when the post is gone."""
payload = get(user, f"/posts/{int(post_id)}.json")
post = payload.get("post") if isinstance(payload, dict) else None
if not isinstance(post, dict):
raise E621Error("e621 returned an unexpected post payload.")
return post
+15 -2
View File
@@ -20,6 +20,7 @@ from django.core.cache import cache
logger = logging.getLogger(__name__)
CACHE_KEY = "j621.guest_blacklist"
MIRROR_KEY = "j621.guest_blacklist.mirror"
def parse_blacklist_patterns(lines):
@@ -37,10 +38,18 @@ def parse_blacklist_patterns(lines):
def cached_guest_blacklist():
"""Raw mirrored blacklist lines. Never hits the network."""
"""Raw mirrored blacklist lines. Never hits the network.
Prefers the TTL'd entry, falls back to the persistent mirror (last
successful e621 fetch) and only then to the local fallback list, so an
expired TTL never degrades filtering.
"""
stored = cache.get(CACHE_KEY)
if stored is not None:
return list(stored)
mirror = cache.get(MIRROR_KEY)
if mirror is not None:
return list(mirror)
return list(settings.GUEST_BLACKLIST_FALLBACK)
@@ -101,7 +110,11 @@ def refresh_guest_blacklist():
lines = fetch_e621_default_blacklist()
used_fallback = lines is None
if used_fallback:
lines = list(settings.GUEST_BLACKLIST_FALLBACK)
# Keep the last successful mirror (or the local list) rather than
# degrading guest filtering when e621 is unreachable.
lines = cache.get(MIRROR_KEY) or list(settings.GUEST_BLACKLIST_FALLBACK)
else:
cache.set(MIRROR_KEY, lines, timeout=None)
cache.set(CACHE_KEY, lines, settings.GUEST_BLACKLIST_TTL)
patterns = parse_blacklist_patterns(lines)
@@ -0,0 +1,35 @@
from datetime import timedelta
from django.core.management.base import BaseCommand
from django.utils import timezone
from apps.library.models import TempUpload
class Command(BaseCommand):
help = "Delete staged uploads older than N hours (default 24)."
def add_arguments(self, parser):
parser.add_argument("--hours", type=int, default=24)
parser.add_argument(
"--include-completed",
action="store_true",
help="Also remove completed staging records.",
)
def handle(self, *args, **options):
cutoff = timezone.now() - timedelta(hours=options["hours"])
queryset = TempUpload.objects.filter(created_at__lt=cutoff)
if not options["include_completed"]:
queryset = queryset.exclude(status=TempUpload.STATUS_COMPLETED)
removed = 0
for temp in queryset:
if temp.file:
temp.file.delete(save=False)
temp.delete()
removed += 1
self.stdout.write(
self.style.SUCCESS(f"Removed {removed} staged upload(s).")
)
@@ -0,0 +1,43 @@
from django.core.management.base import BaseCommand
from django.db.models import Q
from apps.library.models import MediaItem
from apps.library.services import HASH_FIELDS, ensure_visual_hashes
class Command(BaseCommand):
help = "Compute perceptual hashes for library items missing them."
def add_arguments(self, parser):
parser.add_argument(
"--force",
action="store_true",
help="Recompute hashes even when they already exist.",
)
def handle(self, *args, **options):
queryset = MediaItem.objects.prefetch_related("locations").order_by("id")
if not options["force"]:
missing = Q()
for field in HASH_FIELDS:
missing |= Q(**{field: ""})
queryset = queryset.filter(missing)
total = queryset.count()
if total == 0:
self.stdout.write(self.style.SUCCESS("All items already have hashes."))
return
self.stdout.write(f"Computing hashes for {total} item(s) ...")
done = 0
for item in queryset.iterator(chunk_size=100):
if options["force"]:
item.ahash = item.dhash = item.phash = item.whash = ""
ensure_visual_hashes(item)
done += 1
if done % 100 == 0:
self.stdout.write(f"Processed {done} ...")
self.stdout.write(
self.style.SUCCESS(f"Done. Processed {done} item(s).")
)
@@ -0,0 +1,70 @@
from django.contrib.auth import get_user_model
from django.core.management.base import BaseCommand, CommandError
from apps.library import e621, matching
from apps.library.models import MatchTask, MediaItem
class Command(BaseCommand):
help = (
"Match library items against e621 by MD5 and refresh matched metadata "
"(also detects deleted posts). Uses a user's stored e621 credentials."
)
def add_arguments(self, parser):
parser.add_argument(
"--user",
help="Username whose e621 credentials to use "
"(defaults to the first staff account with credentials).",
)
parser.add_argument(
"--scope",
choices=[MatchTask.SCOPE_MISSING, MatchTask.SCOPE_ALL],
default=MatchTask.SCOPE_MISSING,
help="'missing' checks unchecked items only; 'all' also refreshes "
"already matched items (default: missing).",
)
def handle(self, *args, **options):
User = get_user_model()
username = options.get("user")
if username:
user = User.objects.filter(username=username).first()
if user is None:
raise CommandError(f"No such user: {username}")
else:
user = (
User.objects.filter(
is_staff=True, e621_api_key__gt="", e621_username__gt=""
)
.order_by("id")
.first()
)
if not e621.credentials_configured(user):
raise CommandError(
"No e621 credentials found. Pass --user or configure them in Account."
)
queryset = matching.scope_queryset(options["scope"])
total = queryset.count()
counters = {"matched": 0, "not_found": 0, "deleted": 0}
for index, item_id in enumerate(queryset.values_list("id", flat=True), start=1):
item = MediaItem.objects.filter(id=item_id).first()
if item is None:
continue
try:
status = matching.check_item(item, user)
except e621.E621Error as exc:
raise CommandError(f"Aborted after {index - 1}/{total}: {exc}")
counters[status] = counters.get(status, 0) + 1
self.stdout.write(
f"[{index}/{total}] J-{item_id}: {status}", ending="\r"
)
self.stdout.write(
self.style.SUCCESS(
"\nDone: "
f"{counters['matched']} matched, {counters['not_found']} not found, "
f"{counters['deleted']} deleted (scope: {options['scope']})."
)
)
@@ -67,7 +67,9 @@ class Command(BaseCommand):
continue
path = Path(root) / name
try:
item, created_item, created_location = index_file(path, folder)
item, created_item, _location, created_location = index_file(
path, folder
)
except OSError as exc:
skipped += 1
self.stderr.write(f"Skipped {path}: {exc}")
+209
View File
@@ -0,0 +1,209 @@
"""Server-side matching of library items against e621.
MD5 lookups, manual post links, metadata refresh (which is also how deleted
posts are detected) and the background library-wide scan.
"""
import logging
import threading
from datetime import timedelta
from django.db import connection
from django.utils import timezone
from . import e621
from .models import MatchTask, MediaItem
from .services import trim_e621_post
logger = logging.getLogger(__name__)
STALE_AFTER = timedelta(minutes=30)
RATING_VALUES = {"s", "q", "e"}
# --- single-item operations -------------------------------------------------
def apply_post_to_item(item, post):
"""Store trimmed e621 metadata on an item and mark it matched."""
trimmed = trim_e621_post(post)
if trimmed is None or not trimmed.get("id"):
raise e621.E621Error("e621 returned an unexpected post payload.")
item.e621_post_id = int(trimmed["id"])
item.e621_data = trimmed
item.e621_match_status = MediaItem.E621_MATCHED
item.e621_checked_at = timezone.now()
update_fields = [
"e621_post_id",
"e621_data",
"e621_match_status",
"e621_checked_at",
]
rating = trimmed.get("rating")
if not item.rating and rating in RATING_VALUES:
item.rating = rating
update_fields.append("rating")
item.save(update_fields=update_fields + ["updated_at"])
return trimmed
def mark_not_found(item):
item.e621_match_status = MediaItem.E621_NOT_FOUND
item.e621_checked_at = timezone.now()
item.save(update_fields=["e621_match_status", "e621_checked_at", "updated_at"])
def mark_deleted(item):
item.e621_match_status = MediaItem.E621_DELETED
item.e621_checked_at = timezone.now()
item.save(update_fields=["e621_match_status", "e621_checked_at", "updated_at"])
def match_item_by_md5(item, user):
"""Look the item's MD5 up on e621. Returns the resulting match status."""
post = e621.find_post_by_md5(user, item.md5)
if post is None:
mark_not_found(item)
return item.e621_match_status
apply_post_to_item(item, post)
return item.e621_match_status
def refresh_item(item, user):
"""Re-fetch a matched post: refresh metadata or flag it as deleted."""
if not item.e621_post_id:
return match_item_by_md5(item, user)
try:
post = e621.fetch_post(user, item.e621_post_id)
except e621.E621NotFound:
mark_deleted(item)
return item.e621_match_status
apply_post_to_item(item, post)
return item.e621_match_status
def link_item_to_post(item, post_id, user):
"""Manually link an item to a post id. Returns True if the MD5s match."""
post = e621.fetch_post(user, post_id)
trimmed = apply_post_to_item(item, post)
post_md5 = str((trimmed.get("file") or {}).get("md5") or "").strip().lower()
return bool(post_md5) and post_md5 == item.md5.lower()
def check_item(item, user):
"""Best check for this item: refresh a known link, else MD5 lookup."""
if item.e621_post_id and item.e621_match_status == MediaItem.E621_MATCHED:
return refresh_item(item, user)
return match_item_by_md5(item, user)
def unlink_item(item):
"""Forget the e621 link so the item can be checked again."""
item.e621_post_id = None
item.e621_data = None
item.e621_match_status = MediaItem.E621_UNKNOWN
item.e621_checked_at = None
item.save(
update_fields=[
"e621_post_id",
"e621_data",
"e621_match_status",
"e621_checked_at",
"updated_at",
]
)
return item
# --- batch scan -------------------------------------------------------------
def reap_stale_match_tasks():
"""Surface scans left hanging by a recycled worker as errors."""
cutoff = timezone.now() - STALE_AFTER
return MatchTask.objects.filter(
status__in=[MatchTask.STATUS_PENDING, MatchTask.STATUS_RUNNING],
updated_at__lt=cutoff,
).update(
status=MatchTask.STATUS_ERROR,
error="The worker restarted before this scan finished.",
updated_at=timezone.now(),
)
def scope_queryset(scope):
queryset = MediaItem.objects.order_by("id")
if scope == MatchTask.SCOPE_ALL:
return queryset
return queryset.filter(e621_match_status=MediaItem.E621_UNKNOWN)
def start_match_task(task_id):
thread = threading.Thread(target=run_match_task, args=(task_id,), daemon=True)
thread.start()
def run_match_task(task_id):
task = MatchTask.objects.filter(id=task_id).first()
if task is None:
return
item_ids = list(scope_queryset(task.scope).values_list("id", flat=True))
counters = {"matched": 0, "not_found": 0, "deleted": 0}
failed = 0
MatchTask.objects.filter(id=task_id).update(
status=MatchTask.STATUS_RUNNING,
total=len(item_ids),
processed=0,
updated_at=timezone.now(),
)
def save_progress(processed):
MatchTask.objects.filter(id=task_id).update(
processed=processed,
failed=failed,
updated_at=timezone.now(),
**counters,
)
try:
processed = 0
for item_id in item_ids:
if MatchTask.objects.filter(id=task_id, cancelled=True).exists():
MatchTask.objects.filter(id=task_id).update(
status=MatchTask.STATUS_CANCELLED, updated_at=timezone.now()
)
return
item = MediaItem.objects.filter(id=item_id).first()
if item is None:
processed += 1
continue
try:
status = check_item(item, task.user)
counters[status] = counters.get(status, 0) + 1
except e621.E621Error as exc:
# A broken credential or an e621 outage would fail every
# remaining item; abort instead of hammering the API.
logger.warning("Match scan %s aborted: %s", task_id, exc)
failed += 1
MatchTask.objects.filter(id=task_id).update(
status=MatchTask.STATUS_ERROR,
processed=processed,
failed=failed,
error=str(exc),
updated_at=timezone.now(),
**counters,
)
return
processed += 1
save_progress(processed)
MatchTask.objects.filter(id=task_id).update(
status=MatchTask.STATUS_COMPLETE,
processed=len(item_ids),
updated_at=timezone.now(),
**counters,
)
finally:
# Background threads hold their own DB connection; release it so
# Gunicorn workers do not leak connections when threads finish.
connection.close()
@@ -0,0 +1,42 @@
# Generated by Django 6.1.1 on 2026-09-17 16:06
import django.db.models.deletion
import uuid
from django.conf import settings
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('library', '0003_mediaitem_e621_data_mediaitem_e621_post_id'),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name='TempUpload',
fields=[
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
('file', models.FileField(blank=True, upload_to='uploads/temp/')),
('original_filename', models.CharField(max_length=255)),
('md5', models.CharField(db_index=True, max_length=32)),
('size', models.BigIntegerField(default=0)),
('status', models.CharField(choices=[('pending', 'Pending'), ('visual_match', 'Visual Similarity Detected'), ('completed', 'Completed'), ('error', 'Error')], default='pending', max_length=20)),
('resolution', models.CharField(blank=True, default='', max_length=20)),
('e621_post_id', models.IntegerField(blank=True, null=True)),
('e621_data', models.JSONField(blank=True, null=True)),
('custom_rating', models.CharField(blank=True, default='', max_length=1)),
('custom_tags', models.JSONField(blank=True, default=list)),
('custom_notes', models.TextField(blank=True, default='')),
('iqdb_data', models.JSONField(blank=True, null=True)),
('created_at', models.DateTimeField(auto_now_add=True)),
('updated_at', models.DateTimeField(auto_now=True)),
('library_item', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='temp_uploads', to='library.mediaitem')),
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='temp_uploads', to=settings.AUTH_USER_MODEL)),
],
options={
'ordering': ['-created_at'],
},
),
]
@@ -0,0 +1,41 @@
# Generated by Django 6.1.1 on 2026-09-17 17:03
import django.db.models.deletion
import uuid
from django.conf import settings
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('library', '0004_tempupload'),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name='DownloadTask',
fields=[
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
('post_id', models.IntegerField(blank=True, null=True)),
('url', models.CharField(max_length=1000)),
('filename', models.CharField(blank=True, default='', max_length=255)),
('e621_data', models.JSONField(blank=True, null=True)),
('status', models.CharField(choices=[('pending', 'Pending'), ('downloading', 'Downloading'), ('complete', 'Complete'), ('error', 'Error'), ('cancelled', 'Cancelled')], default='pending', max_length=20)),
('progress', models.IntegerField(default=0)),
('downloaded', models.BigIntegerField(default=0)),
('total', models.BigIntegerField(default=0)),
('speed', models.FloatField(blank=True, null=True)),
('error', models.TextField(blank=True, default='')),
('cancelled', models.BooleanField(default=False)),
('created_at', models.DateTimeField(auto_now_add=True)),
('updated_at', models.DateTimeField(auto_now=True)),
('library_item', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='download_tasks', to='library.mediaitem')),
('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='download_tasks', to=settings.AUTH_USER_MODEL)),
],
options={
'ordering': ['-created_at'],
},
),
]
@@ -0,0 +1,33 @@
# Generated by Django 6.1.1 on 2026-09-17 17:34
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('library', '0005_downloadtask'),
]
operations = [
migrations.AddField(
model_name='mediaitem',
name='ahash',
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
),
migrations.AddField(
model_name='mediaitem',
name='dhash',
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
),
migrations.AddField(
model_name='mediaitem',
name='phash',
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
),
migrations.AddField(
model_name='mediaitem',
name='whash',
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
),
]
@@ -0,0 +1,18 @@
# Generated by Django 6.1.1 on 2026-09-17 17:42
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('library', '0006_mediaitem_ahash_mediaitem_dhash_mediaitem_phash_and_more'),
]
operations = [
migrations.AddField(
model_name='tempupload',
name='visual_matches',
field=models.JSONField(blank=True, null=True),
),
]
@@ -0,0 +1,43 @@
# Generated by Django 6.1.1 on 2026-09-17 18:04
from django.db import migrations, models
def backfill_derived_fields(apps, schema_editor):
MediaItem = apps.get_model("library", "MediaItem")
for item in MediaItem.objects.all().iterator(chunk_size=200):
names = [str(tag) for tag in (item.tags or [])]
data = item.e621_data if isinstance(item.e621_data, dict) else {}
categories = data.get("tags")
if isinstance(categories, dict):
for values in categories.values():
if isinstance(values, list):
names.extend(str(tag) for tag in values)
search_tags = " ".join(
sorted({str(name).strip().lower() for name in names if str(name).strip()})
)
MediaItem.objects.filter(pk=item.pk).update(
search_tags=search_tags,
has_custom_data=bool(item.tags or item.notes),
)
class Migration(migrations.Migration):
dependencies = [
('library', '0007_tempupload_visual_matches'),
]
operations = [
migrations.AddField(
model_name='mediaitem',
name='has_custom_data',
field=models.BooleanField(db_index=True, default=False),
),
migrations.AddField(
model_name='mediaitem',
name='search_tags',
field=models.TextField(blank=True, default=''),
),
migrations.RunPython(backfill_derived_fields, migrations.RunPython.noop),
]
@@ -0,0 +1,67 @@
# Generated by Django 6.1.1 on 2026-09-17 18:35
import django.db.models.deletion
import uuid
from django.conf import settings
from django.db import migrations, models
def mark_existing_matches(apps, schema_editor):
"""Items downloaded/linked before the match cache existed."""
MediaItem = apps.get_model("library", "MediaItem")
MediaItem.objects.filter(e621_post_id__isnull=False).update(
e621_match_status="matched",
e621_checked_at=models.F("updated_at"),
)
def unmark_matches(apps, schema_editor):
MediaItem = apps.get_model("library", "MediaItem")
MediaItem.objects.filter(e621_post_id__isnull=False).update(
e621_match_status="unknown",
e621_checked_at=None,
)
class Migration(migrations.Migration):
dependencies = [
('library', '0008_mediaitem_has_custom_data_mediaitem_search_tags'),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.AddField(
model_name='mediaitem',
name='e621_checked_at',
field=models.DateTimeField(blank=True, null=True),
),
migrations.AddField(
model_name='mediaitem',
name='e621_match_status',
field=models.CharField(choices=[('unknown', 'Not checked'), ('matched', 'Matched'), ('not_found', 'Not found on e621'), ('deleted', 'Post deleted on e621')], db_index=True, default='unknown', max_length=20),
),
migrations.CreateModel(
name='MatchTask',
fields=[
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
('scope', models.CharField(choices=[('missing', 'Unchecked items'), ('all', 'Whole library')], default='missing', max_length=20)),
('status', models.CharField(choices=[('pending', 'Pending'), ('running', 'Running'), ('complete', 'Complete'), ('error', 'Error'), ('cancelled', 'Cancelled')], default='pending', max_length=20)),
('total', models.IntegerField(default=0)),
('processed', models.IntegerField(default=0)),
('matched', models.IntegerField(default=0)),
('not_found', models.IntegerField(default=0)),
('deleted', models.IntegerField(default=0)),
('failed', models.IntegerField(default=0)),
('error', models.TextField(blank=True, default='')),
('cancelled', models.BooleanField(default=False)),
('created_at', models.DateTimeField(auto_now_add=True)),
('updated_at', models.DateTimeField(auto_now=True)),
('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='match_tasks', to=settings.AUTH_USER_MODEL)),
],
options={
'ordering': ['-created_at'],
},
),
migrations.RunPython(mark_existing_matches, unmark_matches),
]
+225 -1
View File
@@ -1,10 +1,43 @@
from django.conf import settings
from django.db import models
import uuid
TAG_CLOUD_CACHE_KEYS = [
"j621.library.tag_cloud.auth",
"j621.library.tag_cloud.guest",
]
def build_search_tags(tags, e621_data):
"""Lowercase tag text (custom + e621) used for library tag search."""
names = [str(tag) for tag in (tags or [])]
data = e621_data if isinstance(e621_data, dict) else {}
categories = data.get("tags")
if isinstance(categories, dict):
for values in categories.values():
if isinstance(values, list):
names.extend(str(tag) for tag in values)
return " ".join(
sorted({str(name).strip().lower() for name in names if str(name).strip()})
)
class MediaItem(models.Model):
"""A logical media file, identified by its MD5 fingerprint."""
E621_UNKNOWN = "unknown"
E621_MATCHED = "matched"
E621_NOT_FOUND = "not_found"
E621_DELETED = "deleted"
E621_STATUS_CHOICES = [
(E621_UNKNOWN, "Not checked"),
(E621_MATCHED, "Matched"),
(E621_NOT_FOUND, "Not found on e621"),
(E621_DELETED, "Post deleted on e621"),
]
md5 = models.CharField(max_length=32, unique=True)
size = models.BigIntegerField(default=0)
# Custom rating: "s" (safe), "q" (questionable), "e" (explicit) or "" (unrated).
@@ -22,6 +55,22 @@ class MediaItem(models.Model):
hidden_from_guests = models.BooleanField(default=False, db_index=True)
e621_post_id = models.IntegerField(null=True, blank=True, db_index=True)
e621_data = models.JSONField(null=True, blank=True)
# Cached result of the last MD5/manual match against e621.
e621_match_status = models.CharField(
max_length=20,
choices=E621_STATUS_CHOICES,
default=E621_UNKNOWN,
db_index=True,
)
e621_checked_at = models.DateTimeField(null=True, blank=True)
# Perceptual hashes (hex strings) used by the duplicates engine.
ahash = models.CharField(max_length=32, blank=True, default="", db_index=True)
dhash = models.CharField(max_length=32, blank=True, default="", db_index=True)
phash = models.CharField(max_length=32, blank=True, default="", db_index=True)
whash = models.CharField(max_length=32, blank=True, default="", db_index=True)
# Derived search/filter helpers: flattened tag text and custom-data flag.
search_tags = models.TextField(blank=True, default="")
has_custom_data = models.BooleanField(default=False, db_index=True)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
@@ -32,13 +81,22 @@ class MediaItem(models.Model):
return f"J-{self.pk} ({self.md5})"
def save(self, *args, **kwargs):
from django.core.cache import cache
from .guest_filter import item_is_hidden_for_guests
self.hidden_from_guests = item_is_hidden_for_guests(self)
self.has_custom_data = bool(self.tags or self.notes)
self.search_tags = build_search_tags(self.tags, self.e621_data)
update_fields = kwargs.get("update_fields")
if update_fields is not None:
kwargs["update_fields"] = set(update_fields) | {"hidden_from_guests"}
kwargs["update_fields"] = set(update_fields) | {
"hidden_from_guests",
"has_custom_data",
"search_tags",
}
super().save(*args, **kwargs)
cache.delete_many(TAG_CLOUD_CACHE_KEYS)
class MediaLocation(models.Model):
@@ -57,3 +115,169 @@ class MediaLocation(models.Model):
def __str__(self):
return self.rel_path
class TempUpload(models.Model):
"""A staged upload that has not been accepted into the library yet."""
STATUS_PENDING = "pending"
STATUS_VISUAL_MATCH = "visual_match"
STATUS_COMPLETED = "completed"
STATUS_ERROR = "error"
STATUS_CHOICES = [
(STATUS_PENDING, "Pending"),
(STATUS_VISUAL_MATCH, "Visual Similarity Detected"),
(STATUS_COMPLETED, "Completed"),
(STATUS_ERROR, "Error"),
]
RESOLUTION_AUTO_MD5 = "auto_md5"
RESOLUTION_DUPLICATE = "duplicate"
RESOLUTION_LINKED = "linked"
RESOLUTION_CUSTOM = "custom"
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.CASCADE,
related_name="temp_uploads",
)
file = models.FileField(upload_to="uploads/temp/", blank=True)
original_filename = models.CharField(max_length=255)
md5 = models.CharField(max_length=32, db_index=True)
size = models.BigIntegerField(default=0)
status = models.CharField(
max_length=20, choices=STATUS_CHOICES, default=STATUS_PENDING
)
resolution = models.CharField(max_length=20, blank=True, default="")
e621_post_id = models.IntegerField(null=True, blank=True)
e621_data = models.JSONField(null=True, blank=True)
custom_rating = models.CharField(max_length=1, blank=True, default="")
custom_tags = models.JSONField(default=list, blank=True)
custom_notes = models.TextField(blank=True, default="")
iqdb_data = models.JSONField(null=True, blank=True)
visual_matches = models.JSONField(null=True, blank=True)
library_item = models.ForeignKey(
MediaItem,
null=True,
blank=True,
on_delete=models.SET_NULL,
related_name="temp_uploads",
)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class Meta:
ordering = ["-created_at"]
def __str__(self):
return f"{self.original_filename} ({self.status})"
class DownloadTask(models.Model):
"""A background 'Download to Library' job with progress tracking."""
STATUS_PENDING = "pending"
STATUS_DOWNLOADING = "downloading"
STATUS_COMPLETE = "complete"
STATUS_ERROR = "error"
STATUS_CANCELLED = "cancelled"
STATUS_CHOICES = [
(STATUS_PENDING, "Pending"),
(STATUS_DOWNLOADING, "Downloading"),
(STATUS_COMPLETE, "Complete"),
(STATUS_ERROR, "Error"),
(STATUS_CANCELLED, "Cancelled"),
]
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
null=True,
blank=True,
on_delete=models.SET_NULL,
related_name="download_tasks",
)
post_id = models.IntegerField(null=True, blank=True)
url = models.CharField(max_length=1000)
filename = models.CharField(max_length=255, blank=True, default="")
e621_data = models.JSONField(null=True, blank=True)
status = models.CharField(
max_length=20, choices=STATUS_CHOICES, default=STATUS_PENDING
)
progress = models.IntegerField(default=0)
downloaded = models.BigIntegerField(default=0)
total = models.BigIntegerField(default=0)
speed = models.FloatField(null=True, blank=True)
error = models.TextField(blank=True, default="")
cancelled = models.BooleanField(default=False)
library_item = models.ForeignKey(
MediaItem,
null=True,
blank=True,
on_delete=models.SET_NULL,
related_name="download_tasks",
)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class Meta:
ordering = ["-created_at"]
def __str__(self):
return f"Download {self.id} ({self.status})"
class MatchTask(models.Model):
"""A background library-wide e621 match scan with progress tracking."""
STATUS_PENDING = "pending"
STATUS_RUNNING = "running"
STATUS_COMPLETE = "complete"
STATUS_ERROR = "error"
STATUS_CANCELLED = "cancelled"
STATUS_CHOICES = [
(STATUS_PENDING, "Pending"),
(STATUS_RUNNING, "Running"),
(STATUS_COMPLETE, "Complete"),
(STATUS_ERROR, "Error"),
(STATUS_CANCELLED, "Cancelled"),
]
SCOPE_MISSING = "missing"
SCOPE_ALL = "all"
SCOPE_CHOICES = [
(SCOPE_MISSING, "Unchecked items"),
(SCOPE_ALL, "Whole library"),
]
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
null=True,
blank=True,
on_delete=models.SET_NULL,
related_name="match_tasks",
)
scope = models.CharField(
max_length=20, choices=SCOPE_CHOICES, default=SCOPE_MISSING
)
status = models.CharField(
max_length=20, choices=STATUS_CHOICES, default=STATUS_PENDING
)
total = models.IntegerField(default=0)
processed = models.IntegerField(default=0)
matched = models.IntegerField(default=0)
not_found = models.IntegerField(default=0)
deleted = models.IntegerField(default=0)
failed = models.IntegerField(default=0)
error = models.TextField(blank=True, default="")
cancelled = models.BooleanField(default=False)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class Meta:
ordering = ["-created_at"]
def __str__(self):
return f"Match scan {self.id} ({self.status})"
+156 -2
View File
@@ -1,9 +1,16 @@
import os
from pathlib import Path
from django.core import signing
from rest_framework import serializers
from .models import MediaItem, MediaLocation
from .services import VIDEO_EXTENSIONS
from .models import MediaItem, MediaLocation, TempUpload, DownloadTask, MatchTask
from .services import (
MEDIA_FILE_SALT,
UPLOAD_FILE_SALT,
VIDEO_EXTENSIONS,
signed_media_url,
)
class MediaLocationSerializer(serializers.ModelSerializer):
@@ -20,6 +27,8 @@ class MediaItemSerializer(serializers.ModelSerializer):
locations = MediaLocationSerializer(many=True, read_only=True)
j_id = serializers.SerializerMethodField()
display_rating = serializers.SerializerMethodField()
raw_url = serializers.SerializerMethodField()
thumbnail_url = serializers.SerializerMethodField()
filename = serializers.SerializerMethodField()
extension = serializers.SerializerMethodField()
kind = serializers.SerializerMethodField()
@@ -40,10 +49,14 @@ class MediaItemSerializer(serializers.ModelSerializer):
"filename",
"extension",
"kind",
"raw_url",
"thumbnail_url",
"uploaded_by",
"uploaded_by_id",
"e621_post_id",
"e621_data",
"e621_match_status",
"e621_checked_at",
"locations",
"created_at",
"updated_at",
@@ -57,18 +70,35 @@ class MediaItemSerializer(serializers.ModelSerializer):
"filename",
"extension",
"kind",
"raw_url",
"thumbnail_url",
"uploaded_by",
"uploaded_by_id",
"e621_post_id",
"e621_data",
"e621_match_status",
"e621_checked_at",
"locations",
"created_at",
"updated_at",
]
def _request_user(self):
request = self.context.get("request")
user = getattr(request, "user", None)
if user is None or not getattr(user, "is_authenticated", False):
return None
return user
def get_j_id(self, obj):
return f"J-{obj.id}"
def get_raw_url(self, obj):
return signed_media_url(obj, self._request_user(), "raw")
def get_thumbnail_url(self, obj):
return signed_media_url(obj, self._request_user(), "thumbnail")
def get_display_rating(self, obj):
if obj.rating:
return obj.rating
@@ -92,3 +122,127 @@ class MediaItemSerializer(serializers.ModelSerializer):
def get_kind(self, obj):
return "video" if "." + self.get_extension(obj) in VIDEO_EXTENSIONS else "image"
class TempUploadSerializer(serializers.ModelSerializer):
temp_id = serializers.UUIDField(source="id", read_only=True)
library_j_id = serializers.SerializerMethodField()
file_url = serializers.SerializerMethodField()
preview_url = serializers.SerializerMethodField()
class Meta:
model = TempUpload
fields = [
"temp_id",
"original_filename",
"md5",
"size",
"status",
"resolution",
"e621_post_id",
"e621_data",
"custom_rating",
"custom_tags",
"custom_notes",
"iqdb_data",
"visual_matches",
"library_j_id",
"file_url",
"preview_url",
"created_at",
"updated_at",
]
read_only_fields = fields
def _request_user(self):
request = self.context.get("request")
user = getattr(request, "user", None)
if user is None or not getattr(user, "is_authenticated", False):
return None
return user
def get_library_j_id(self, obj):
return f"J-{obj.library_item_id}" if obj.library_item_id else None
def get_file_url(self, obj):
"""Signed URL so <img>/<video> tags can fetch the staged file."""
if not obj.file:
return None
user = self._request_user()
if user is None:
return None
signature = signing.dumps(
{"temp": str(obj.id), "user": user.id},
salt=UPLOAD_FILE_SALT,
)
return f"/api/uploads/{obj.id}/file/?sig={signature}"
def get_preview_url(self, obj):
"""A URL an <img>/<video> tag can load: staged file or library item."""
user = self._request_user()
if user is None:
return None
if obj.file:
return self.get_file_url(obj)
item = obj.library_item
if item is None:
return None
location = item.locations.first()
is_video = (
location is not None
and Path(location.rel_path).suffix.lower() in VIDEO_EXTENSIONS
)
action = "thumbnail" if is_video else "raw"
signature = signing.dumps(
{"item": item.id, "user": user.id, "action": action},
salt=MEDIA_FILE_SALT,
)
return f"/api/files/J-{item.id}/{action}/?sig={signature}"
class DownloadTaskSerializer(serializers.ModelSerializer):
task_id = serializers.UUIDField(source="id", read_only=True)
library_j_id = serializers.SerializerMethodField()
class Meta:
model = DownloadTask
fields = [
"task_id",
"post_id",
"filename",
"status",
"progress",
"downloaded",
"total",
"speed",
"error",
"library_j_id",
"created_at",
"updated_at",
]
read_only_fields = fields
def get_library_j_id(self, obj):
return f"J-{obj.library_item_id}" if obj.library_item_id else None
class MatchTaskSerializer(serializers.ModelSerializer):
task_id = serializers.UUIDField(source="id", read_only=True)
class Meta:
model = MatchTask
fields = [
"task_id",
"scope",
"status",
"total",
"processed",
"matched",
"not_found",
"deleted",
"failed",
"error",
"created_at",
"updated_at",
]
read_only_fields = fields
+154 -4
View File
@@ -1,5 +1,6 @@
import hashlib
import json
import logging
import mimetypes
import os
import re
@@ -7,14 +8,33 @@ import shutil
import subprocess
from pathlib import Path
import imagehash
from django.conf import settings
from django.core import signing
from django.http import FileResponse, Http404, HttpResponse
from django.utils.text import get_valid_filename
from PIL import Image
from .models import MediaItem, MediaLocation
ALLOWED_EXTENSIONS = {".jpg", ".jpeg", ".png", ".gif", ".apng", ".mp4", ".webm"}
logger = logging.getLogger(__name__)
HASH_FIELDS = ("ahash", "dhash", "phash", "whash")
IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".gif", ".apng", ".webp"}
ALLOWED_EXTENSIONS = {
".jpg",
".jpeg",
".png",
".gif",
".apng",
".webp",
".mp4",
".webm",
}
VIDEO_EXTENSIONS = {".mp4", ".webm"}
UPLOAD_FILE_SALT = "j621.upload-file"
MEDIA_FILE_SALT = "j621.media-file"
CHUNK_SIZE = 1024 * 1024
RANGE_RE = re.compile(r"bytes=(\d*)-(\d*)$")
@@ -30,7 +50,7 @@ def compute_md5(path):
def index_file(path, folder):
"""Index one file into MediaItem/MediaLocation.
Returns (item, created_item, created_location).
Returns (item, created_item, location, created_location).
"""
path = Path(path)
folder = Path(folder)
@@ -50,7 +70,72 @@ def index_file(path, folder):
"mtime": stat.st_mtime,
},
)
return item, created_item, created_location
return item, created_item, location, created_location
def signed_media_url(item, user, action="raw"):
"""Media URL that <img>/<video> tags can load for a signed-in user."""
base = f"/api/files/J-{item.id}/{action}/"
if user is None or not getattr(user, "is_authenticated", False):
return base
signature = signing.dumps(
{"item": item.id, "user": user.id, "action": action},
salt=MEDIA_FILE_SALT,
)
return f"{base}?sig={signature}"
def rename_location_to_j_id(item, location):
"""Name a freshly indexed copy J-<id>.<ext> inside its own folder."""
path = Path(location.path)
if not path.exists():
return location
if path.stem == f"J-{item.id}":
return location
target = unique_destination(path.parent, f"J-{item.id}{path.suffix}")
path.rename(target)
parent = Path(location.rel_path).parent
location.path = str(target)
location.rel_path = (
str(parent / target.name) if str(parent) != "." else target.name
)
location.save(update_fields=["path", "rel_path"])
return location
def compute_visual_hashes(path):
"""Perceptual hashes for an image file (empty dict for other files)."""
path = Path(path)
if path.suffix.lower() not in IMAGE_EXTENSIONS:
return {}
try:
with Image.open(path) as image:
converted = image.convert("RGB")
return {
"ahash": str(imagehash.average_hash(converted, hash_size=8)),
"dhash": str(imagehash.dhash(converted, hash_size=8)),
"phash": str(imagehash.phash(converted, hash_size=8)),
"whash": str(imagehash.whash(converted, hash_size=8)),
}
except Exception: # noqa: BLE001 - hashing must never break indexing
logger.exception("Could not compute visual hashes for %s", path)
return {}
def ensure_visual_hashes(item):
"""Fill in missing perceptual hashes for a media item."""
if all(getattr(item, field) for field in HASH_FIELDS):
return item
location = item.locations.first()
if location is None:
return item
hashes = compute_visual_hashes(location.path)
if not hashes:
return item
for field, value in hashes.items():
setattr(item, field, value)
item.save(update_fields=[*hashes.keys(), "updated_at"])
return item
def parse_tags(raw):
@@ -160,17 +245,32 @@ def serve_file(request, path, download=False):
return response
def download_file(url, destination):
class DownloadCancelled(Exception):
"""Raised when a streamed download is cancelled by the user."""
def download_file(
url, destination, progress_callback=None, should_cancel=None
):
"""Stream a remote file into destination (used by Download to Library)."""
import requests
headers = {"User-Agent": settings.USER_AGENT}
with requests.get(url, headers=headers, stream=True, timeout=120) as response:
response.raise_for_status()
total = int(response.headers.get("content-length") or 0)
downloaded = 0
with open(destination, "wb") as handle:
for chunk in response.iter_content(chunk_size=CHUNK_SIZE):
if should_cancel is not None and should_cancel():
raise DownloadCancelled("download cancelled")
if chunk:
handle.write(chunk)
downloaded += len(chunk)
if progress_callback is not None:
progress_callback(downloaded, total)
if progress_callback is not None:
progress_callback(downloaded, total or downloaded)
E621_DESCRIPTION_LIMIT = 20000
@@ -221,11 +321,61 @@ def trim_e621_post(post):
"size": file_data.get("size"),
"width": file_data.get("width"),
"height": file_data.get("height"),
"url": file_data.get("url"),
},
"preview": {
"url": (post.get("preview") or {}).get("url"),
},
"uploader_name": post.get("uploader_name"),
}
def sanitize_iqdb_results(results):
"""Keep a compact, size-bounded copy of IQDB candidates."""
cleaned = []
for result in results[:10]:
if not isinstance(result, dict):
continue
score = result.get("score")
post_id = result.get("post_id")
tags_preview = result.get("tags_preview")
entry = {
"post_id": post_id if isinstance(post_id, int) else None,
"score": float(score) if isinstance(score, (int, float)) else None,
"preview_url": str(result.get("preview_url") or "")[:500] or None,
"rating": str(result.get("rating") or "")[:1] or None,
"md5": str(result.get("md5") or "")[:32] or None,
"score_total": (
result.get("score_total")
if isinstance(result.get("score_total"), int)
else None
),
"fav_count": (
result.get("fav_count")
if isinstance(result.get("fav_count"), int)
else None
),
"width": (
result.get("width")
if isinstance(result.get("width"), int)
else None
),
"height": (
result.get("height")
if isinstance(result.get("height"), int)
else None
),
"tags_preview": (
[str(tag)[:100] for tag in tags_preview][:8]
if isinstance(tags_preview, list)
else []
),
}
if entry["post_id"] or entry["preview_url"]:
cleaned.append(entry)
return cleaned
def generate_video_thumbnail(md5, path):
"""Extract a JPEG thumbnail from a video, cached under MEDIA_ROOT/thumbs."""
if not shutil.which("ffmpeg"):
+451
View File
@@ -0,0 +1,451 @@
"""Library tooling: duplicates, deletion and storage overview."""
import logging
import shutil
from pathlib import Path
from django.conf import settings
from django.core.cache import cache
from django.db.models import Count, Q
from rest_framework import status
from rest_framework.permissions import AllowAny, IsAuthenticated
from rest_framework.response import Response
from rest_framework.views import APIView
from .models import TAG_CLOUD_CACHE_KEYS, MediaItem, MediaLocation
from .permissions import CanUpload
from .services import signed_media_url
logger = logging.getLogger(__name__)
HASH_FIELDS = ("ahash", "dhash", "phash", "whash")
STORAGE_CACHE_KEY = "j621.library.storage"
STORAGE_CACHE_TTL = 60
GROUPS_PER_PAGE = 20
def parse_threshold(value, default=0.8):
try:
parsed = float(value)
except (TypeError, ValueError):
return default
if parsed > 1:
parsed = parsed / 100.0
return min(max(parsed, 0.0), 1.0)
def parse_algorithms(value):
if not isinstance(value, list):
return list(HASH_FIELDS)
selected = [field for field in value if field in HASH_FIELDS]
return selected or list(HASH_FIELDS)
def hashes_similarity(first_hashes, second_hashes, algorithms, threshold):
"""Best similarity between two hash mappings, or None below threshold."""
best = None
for field in algorithms:
first = first_hashes.get(field) or ""
second = second_hashes.get(field) or ""
if not first or not second:
continue
try:
distance = bin(int(first, 16) ^ int(second, 16)).count("1")
except ValueError:
continue
value = 1.0 - distance / 64.0
if best is None or value > best:
best = value
if best is None or best < threshold:
return None
return best
def similarity_between(item_a, item_b, algorithms, threshold):
"""Best similarity across the selected algorithms, or None below threshold."""
return hashes_similarity(
{field: getattr(item_a, field, "") for field in algorithms},
{field: getattr(item_b, field, "") for field in algorithms},
algorithms,
threshold,
)
def hashed_items(algorithms):
query = Q()
for field in algorithms:
query |= ~Q(**{field: ""})
return list(MediaItem.objects.filter(query).prefetch_related("locations"))
def display_rating(item):
if item.rating:
return item.rating
data = item.e621_data or {}
rating = data.get("rating") if isinstance(data, dict) else None
return rating if rating in {"s", "q", "e"} else ""
def item_brief(item, request=None):
locations = list(item.locations.all())
location = locations[0] if locations else None
user = getattr(request, "user", None)
return {
"j_id": f"J-{item.id}",
"md5": item.md5,
"filename": Path(location.rel_path).name if location else item.md5,
"size": item.size,
"rating": display_rating(item),
"location_count": len(locations),
"uploaded_by": item.uploaded_by.username if item.uploaded_by else None,
"e621_post_id": item.e621_post_id,
"thumbnail_url": signed_media_url(item, user, "thumbnail"),
}
def resolve_item(data):
j_id = str(data.get("j_id") or "").strip()
md5 = str(data.get("md5") or "").strip().lower()
if j_id:
numeric = j_id[2:] if j_id.upper().startswith("J-") else j_id
if numeric.isdigit():
item = MediaItem.objects.filter(pk=int(numeric)).first()
if item is not None:
return item
if md5:
return MediaItem.objects.filter(md5=md5).first()
return None
class ExactDuplicatesView(APIView):
"""Items whose content exists at more than one path."""
permission_classes = [IsAuthenticated]
def get(self, request):
items = (
MediaItem.objects.annotate(location_count=Count("locations"))
.filter(location_count__gt=1)
.prefetch_related("locations")
.order_by("-location_count", "id")
)
groups = []
for item in items:
brief = item_brief(item, request)
brief["locations"] = [
{"id": location.id, "rel_path": location.rel_path}
for location in item.locations.all()
]
groups.append(brief)
return Response({"count": len(groups), "groups": groups})
class VisualMatchesView(APIView):
"""Items visually similar to one library item."""
permission_classes = [IsAuthenticated]
def post(self, request):
threshold = parse_threshold(request.data.get("threshold"))
algorithms = parse_algorithms(request.data.get("algorithms"))
target = resolve_item(request.data)
if target is None:
return Response(
{"detail": "A j_id or md5 is required."},
status=status.HTTP_400_BAD_REQUEST,
)
matches = []
for item in hashed_items(algorithms):
if item.pk == target.pk:
continue
similarity = similarity_between(target, item, algorithms, threshold)
if similarity is None:
continue
brief = item_brief(item, request)
brief["similarity"] = round(similarity * 100, 1)
matches.append(brief)
matches.sort(key=lambda entry: entry["similarity"], reverse=True)
return Response(
{
"target": item_brief(target, request),
"threshold": round(threshold * 100, 1),
"algorithms": algorithms,
"count": len(matches),
"matches": matches[:200],
}
)
class VisualGroupsView(APIView):
"""Groups of visually similar items (union-find over perceptual hashes).
Pairwise comparison is O(n^2) with fast bit operations, which is fine for
a personal library. Revisit with a bucketed index if libraries grow huge.
"""
permission_classes = [IsAuthenticated]
def post(self, request):
threshold = parse_threshold(request.data.get("threshold"))
algorithms = parse_algorithms(request.data.get("algorithms"))
page = max(1, int(request.data.get("page") or 1))
items = hashed_items(algorithms)
parent = list(range(len(items)))
def find(index):
while parent[index] != index:
parent[index] = parent[parent[index]]
index = parent[index]
return index
def union(first, second):
root_a = find(first)
root_b = find(second)
if root_a != root_b:
parent[root_b] = root_a
for first in range(len(items)):
for second in range(first + 1, len(items)):
if find(first) == find(second):
continue
if (
similarity_between(
items[first], items[second], algorithms, threshold
)
is not None
):
union(first, second)
grouped = {}
for index, item in enumerate(items):
grouped.setdefault(find(index), []).append(item)
groups = [members for members in grouped.values() if len(members) >= 2]
groups.sort(key=len, reverse=True)
total = len(groups)
start = (page - 1) * GROUPS_PER_PAGE
page_groups = groups[start : start + GROUPS_PER_PAGE]
return Response(
{
"count": total,
"page": page,
"per_page": GROUPS_PER_PAGE,
"has_next": start + GROUPS_PER_PAGE < total,
"threshold": round(threshold * 100, 1),
"algorithms": algorithms,
"groups": [
{
"size": len(members),
"members": [item_brief(item, request) for item in members],
}
for members in page_groups
],
}
)
def can_delete(user, item):
if user.is_superuser or user.role == user.ROLE_STAFF:
return True
return item.uploaded_by_id == user.id
def remove_watched_file(path):
"""Delete a file only when it lives inside the watched folder."""
watched = Path(settings.WATCHED_FOLDER).resolve()
try:
resolved = Path(path).resolve()
resolved.relative_to(watched)
except (ValueError, OSError):
return False
if resolved.is_file():
resolved.unlink()
return True
return False
class DeleteFilesView(APIView):
"""Delete items (with every copy) or individual duplicate locations."""
permission_classes = [CanUpload]
def post(self, request):
j_ids = request.data.get("j_ids") or []
location_ids = request.data.get("location_ids") or []
if not isinstance(j_ids, list) or not isinstance(location_ids, list):
return Response(
{"detail": "j_ids and location_ids must be lists."},
status=status.HTTP_400_BAD_REQUEST,
)
deleted = []
errors = []
numeric_ids = []
for value in j_ids:
text = str(value).strip()
numeric = text[2:] if text.upper().startswith("J-") else text
if numeric.isdigit():
numeric_ids.append(int(numeric))
for item in MediaItem.objects.filter(pk__in=numeric_ids).prefetch_related(
"locations"
):
if not can_delete(request.user, item):
errors.append({"j_id": f"J-{item.id}", "error": "permission denied"})
continue
for location in item.locations.all():
remove_watched_file(location.path)
deleted.append(f"J-{item.id}")
item.delete()
cache.delete(STORAGE_CACHE_KEY)
cache.delete_many(TAG_CLOUD_CACHE_KEYS)
for location in MediaLocation.objects.filter(
id__in=location_ids
).select_related("item"):
if not can_delete(request.user, location.item):
errors.append(
{"location": location.id, "error": "permission denied"}
)
continue
location_id = location.id
item = location.item
remove_watched_file(location.path)
location.delete()
if item.locations.exists():
deleted.append(f"location {location_id}")
else:
deleted.append(f"J-{item.id}")
item.delete()
return Response({"deleted": deleted, "errors": errors})
class ClearTempView(APIView):
"""Remove staged files from the temp upload folder."""
permission_classes = [CanUpload]
def post(self, request):
temp_dir = Path(settings.MEDIA_ROOT) / "uploads" / "temp"
removed = 0
if temp_dir.exists():
for entry in temp_dir.iterdir():
if entry.is_file():
entry.unlink()
removed += 1
cache.delete(STORAGE_CACHE_KEY)
return Response({"removed": removed})
class TagCloudView(APIView):
"""Most-used tags across the library (custom + e621 tags)."""
permission_classes = [AllowAny]
def get(self, request):
cache_key = (
"j621.library.tag_cloud.auth"
if request.user.is_authenticated
else "j621.library.tag_cloud.guest"
)
cached = cache.get(cache_key)
if cached is not None:
return Response(cached)
queryset = MediaItem.objects.all()
if not request.user.is_authenticated:
queryset = queryset.filter(hidden_from_guests=False)
counts = {}
for item in queryset.only("tags", "e621_data").iterator(chunk_size=500):
names = set()
for tag in item.tags or []:
name = str(tag).strip().lower()
if name:
names.add(name)
data = item.e621_data or {}
categories = data.get("tags") if isinstance(data, dict) else None
if isinstance(categories, dict):
for values in categories.values():
if isinstance(values, list):
names.update(
str(tag).strip().lower()
for tag in values
if str(tag).strip()
)
for name in names:
counts[name] = counts.get(name, 0) + 1
ranked = sorted(counts.items(), key=lambda entry: (-entry[1], entry[0]))
payload = {
"count": len(counts),
"tags": [
{"tag": tag, "count": count} for tag, count in ranked[:120]
],
}
cache.set(cache_key, payload, 120)
return Response(payload)
class StorageView(APIView):
"""Disk usage for the watched folder, media root and temp uploads."""
permission_classes = [IsAuthenticated]
def get(self, request):
cached = cache.get(STORAGE_CACHE_KEY)
if cached is not None:
return Response(cached)
watched = Path(settings.WATCHED_FOLDER)
media_root = Path(settings.MEDIA_ROOT)
temp_dir = media_root / "uploads" / "temp"
usage = shutil.disk_usage(
str(watched) if watched.exists() else str(Path(settings.BASE_DIR))
)
library_size = (
sum(file.stat().st_size for file in watched.rglob("*") if file.is_file())
if watched.exists()
else 0
)
media_size = (
sum(file.stat().st_size for file in media_root.rglob("*") if file.is_file())
if media_root.exists()
else 0
)
temp_files = (
[file for file in temp_dir.rglob("*") if file.is_file()]
if temp_dir.exists()
else []
)
payload = {
"watched_folder": {
"path": str(watched),
"total": usage.total,
"used": usage.used,
"free": usage.free,
"library_size": library_size,
"percent_used": (
round(usage.used / usage.total * 100, 1) if usage.total else 0
),
},
"media": {"path": str(media_root), "size": media_size},
"temp": {
"path": str(temp_dir),
"size": sum(file.stat().st_size for file in temp_files),
"files": len(temp_files),
},
"library_items": MediaItem.objects.count(),
}
cache.set(STORAGE_CACHE_KEY, payload, STORAGE_CACHE_TTL)
return Response(payload)
+329
View File
@@ -0,0 +1,329 @@
"""Staged uploads: files land in a temp folder until they are resolved.
Flow:
- POST /api/uploads/ stage a file (computes MD5, detects duplicates)
- GET /api/uploads/ list my staged uploads
- GET /api/uploads/<id>/file/ serve the staged file (previews, IQDB)
- POST /api/uploads/<id>/iqdb/ store IQDB candidates -> visual_match
- POST /api/uploads/<id>/resolve/ link to a post or apply custom metadata
- DELETE /api/uploads/<id>/ discard a staged upload / dismiss a record
"""
import logging
import shutil
from pathlib import Path
from urllib.parse import urlparse
from django.conf import settings
from django.contrib.auth import get_user_model
from django.core import signing
from django.http import Http404
from django.utils import timezone
from rest_framework import mixins, status, viewsets
from rest_framework.decorators import action
from rest_framework.parsers import FormParser, JSONParser, MultiPartParser
from rest_framework.permissions import AllowAny
from rest_framework.response import Response
from . import services
from .models import MediaItem, TempUpload
from .permissions import CanUpload
from .serializers import TempUploadSerializer
from .tools import HASH_FIELDS, hashes_similarity
logger = logging.getLogger(__name__)
def find_library_matches(path, limit=10, user=None):
"""Library items visually similar to a staged file."""
hashes = services.compute_visual_hashes(path)
if not hashes:
return []
algorithms = list(HASH_FIELDS)
threshold = settings.VISUAL_MATCH_THRESHOLD
matches = []
for item in MediaItem.objects.prefetch_related("locations"):
similarity = hashes_similarity(
hashes,
{field: getattr(item, field, "") for field in algorithms},
algorithms,
threshold,
)
if similarity is None:
continue
location = item.locations.first()
matches.append(
{
"j_id": f"J-{item.id}",
"filename": Path(location.rel_path).name if location else item.md5,
"similarity": round(similarity * 100, 1),
"thumbnail_url": services.signed_media_url(item, user, "thumbnail"),
}
)
matches.sort(key=lambda entry: entry["similarity"], reverse=True)
return matches[:limit]
def complete_temp_upload(temp, download_url=None):
"""Index the upload into the library.
With a ``download_url`` the e621 original is fetched and the staged copy
is dropped; otherwise the staged file itself moves into the library.
"""
folder = Path(settings.WATCHED_FOLDER)
if download_url:
file_data = {}
if isinstance(temp.e621_data, dict):
file_data = temp.e621_data.get("file") or {}
extension = Path(urlparse(download_url).path).suffix
if not extension:
extension = Path(temp.original_filename).suffix
md5 = file_data.get("md5")
name = f"{md5}{extension}" if md5 else (temp.original_filename or "download")
destination = services.unique_destination(folder, name)
try:
services.download_file(download_url, destination)
except Exception:
destination.unlink(missing_ok=True)
raise
item, _, location, _ = services.index_file(destination, folder)
services.rename_location_to_j_id(item, location)
services.ensure_visual_hashes(item)
if temp.file:
temp.file.delete(save=False)
else:
if not temp.file:
raise ValueError("staged file is missing")
destination = services.unique_destination(
folder, temp.original_filename or "upload"
)
with temp.file.open("rb") as source, open(destination, "wb") as target:
shutil.copyfileobj(source, target)
item, _, location, _ = services.index_file(destination, folder)
services.rename_location_to_j_id(item, location)
services.ensure_visual_hashes(item)
temp.file.delete(save=False)
temp.library_item = item
temp.status = TempUpload.STATUS_COMPLETED
update_fields = []
if item.uploaded_by_id is None:
item.uploaded_by = temp.user
update_fields.append("uploaded_by")
if temp.e621_post_id is not None:
item.e621_post_id = temp.e621_post_id
item.e621_data = temp.e621_data
item.e621_match_status = MediaItem.E621_MATCHED
item.e621_checked_at = timezone.now()
update_fields += [
"e621_post_id",
"e621_data",
"e621_match_status",
"e621_checked_at",
]
if temp.custom_rating in {"s", "q", "e"} and not item.rating:
item.rating = temp.custom_rating
update_fields.append("rating")
if temp.custom_tags:
item.tags = temp.custom_tags
update_fields.append("tags")
if temp.custom_notes:
item.notes = temp.custom_notes
update_fields.append("notes")
if update_fields:
item.save(update_fields=update_fields + ["updated_at"])
temp.save()
return item
class TempUploadViewSet(
mixins.ListModelMixin,
mixins.RetrieveModelMixin,
mixins.DestroyModelMixin,
viewsets.GenericViewSet,
):
serializer_class = TempUploadSerializer
permission_classes = [CanUpload]
parser_classes = [MultiPartParser, FormParser, JSONParser]
http_method_names = ["get", "post", "delete", "head", "options"]
def get_queryset(self):
queryset = TempUpload.objects.select_related("library_item")
user = self.request.user
if not (user.is_staff or user.is_superuser):
queryset = queryset.filter(user=user)
return queryset
def create(self, request):
upload = request.FILES.get("file")
if upload is None:
return Response(
{"detail": "A file is required."}, status=status.HTTP_400_BAD_REQUEST
)
extension = Path(upload.name).suffix.lower()
if extension not in services.ALLOWED_EXTENSIONS:
return Response(
{"detail": f"Unsupported file type: {extension or 'unknown'}"},
status=status.HTTP_400_BAD_REQUEST,
)
temp = TempUpload.objects.create(
user=request.user,
file=upload,
original_filename=upload.name,
size=upload.size,
)
temp.md5 = services.compute_md5(temp.file.path)
existing = MediaItem.objects.filter(md5=temp.md5).first()
if existing is not None:
temp.status = TempUpload.STATUS_COMPLETED
temp.resolution = TempUpload.RESOLUTION_DUPLICATE
temp.library_item = existing
temp.file.delete(save=False)
else:
matches = find_library_matches(temp.file.path, user=request.user)
if matches:
temp.visual_matches = matches
temp.status = TempUpload.STATUS_VISUAL_MATCH
temp.save()
return Response(
self.get_serializer(temp).data, status=status.HTTP_201_CREATED
)
@action(detail=True, methods=["get", "head"], permission_classes=[AllowAny])
def file(self, request, pk=None):
"""Serve the staged file; accepts a signed URL for media tags."""
user = request.user if request.user.is_authenticated else None
if user is None:
signature = request.query_params.get("sig")
if signature:
try:
payload = signing.loads(
signature,
salt=services.UPLOAD_FILE_SALT,
max_age=86400,
)
except signing.BadSignature:
payload = None
if payload and str(payload.get("temp")) == str(pk):
user = (
get_user_model()
.objects.filter(pk=payload.get("user"))
.first()
)
if user is None or not user.can_upload:
return Response(
{"detail": "Authentication required."},
status=status.HTTP_401_UNAUTHORIZED,
)
temp = TempUpload.objects.filter(pk=pk).first()
is_owner = temp is not None and temp.user_id == user.id
if temp is None or not (
is_owner or user.is_staff or user.is_superuser
):
raise Http404
if not temp.file:
raise Http404
return services.serve_file(request, temp.file.path)
@action(detail=True, methods=["post"])
def iqdb(self, request, pk=None):
temp = self.get_object()
results = request.data.get("results")
if not isinstance(results, list):
return Response(
{"detail": "results must be a list."},
status=status.HTTP_400_BAD_REQUEST,
)
temp.iqdb_data = services.sanitize_iqdb_results(results)
if temp.status == TempUpload.STATUS_PENDING and temp.iqdb_data:
temp.status = TempUpload.STATUS_VISUAL_MATCH
temp.save(update_fields=["iqdb_data", "status", "updated_at"])
return Response(self.get_serializer(temp).data)
@action(detail=True, methods=["post"])
def resolve(self, request, pk=None):
temp = self.get_object()
if temp.status == TempUpload.STATUS_COMPLETED:
return Response(
{"detail": "This upload is already in the library."},
status=status.HTTP_400_BAD_REQUEST,
)
mode = str(request.data.get("mode") or "").strip()
download_url = None
if mode == "link":
post = request.data.get("post")
post_id = request.data.get("post_id")
trimmed = services.trim_e621_post(post)
has_numeric_id = post_id is not None and str(post_id).isdigit()
if trimmed is None and not has_numeric_id:
return Response(
{"detail": "A post payload or post_id is required."},
status=status.HTTP_400_BAD_REQUEST,
)
if trimmed is not None and trimmed.get("id") is not None:
temp.e621_post_id = int(trimmed["id"])
elif has_numeric_id:
temp.e621_post_id = int(post_id)
temp.e621_data = trimmed
temp.resolution = (
TempUpload.RESOLUTION_AUTO_MD5
if request.data.get("auto")
else TempUpload.RESOLUTION_LINKED
)
rating = str(request.data.get("rating") or "")
if rating in {"s", "q", "e"}:
temp.custom_rating = rating
tags = services.parse_tags(request.data.get("tags"))
if tags is not None:
temp.custom_tags = tags
notes = request.data.get("notes")
if notes is not None:
temp.custom_notes = str(notes)
# Prefer the e621 original; fall back to the staged file when the
# post has no URL or its file is byte-identical to the upload.
file_data = (trimmed or {}).get("file") or {}
candidate_url = str(
request.data.get("file_url") or file_data.get("url") or ""
).strip()
post_md5 = str(file_data.get("md5") or "").strip().lower()
if candidate_url and post_md5 and post_md5 == temp.md5.lower():
candidate_url = ""
download_url = candidate_url or None
elif mode == "custom":
rating = str(request.data.get("rating") or "")
temp.custom_rating = rating if rating in {"s", "q", "e"} else ""
temp.custom_tags = services.parse_tags(request.data.get("tags")) or []
temp.custom_notes = str(request.data.get("notes") or "")
temp.resolution = TempUpload.RESOLUTION_CUSTOM
else:
return Response(
{"detail": "mode must be 'link' or 'custom'."},
status=status.HTTP_400_BAD_REQUEST,
)
temp.save()
try:
complete_temp_upload(temp, download_url=download_url)
except Exception as exc: # noqa: BLE001 - report completion failures
logger.exception("Could not complete staged upload %s", temp.id)
temp.status = TempUpload.STATUS_ERROR
temp.save(update_fields=["status", "updated_at"])
return Response(
{"detail": f"Could not finish the upload: {exc}"},
status=status.HTTP_400_BAD_REQUEST,
)
temp.refresh_from_db()
return Response(self.get_serializer(temp).data)
def perform_destroy(self, instance):
if instance.file:
instance.file.delete(save=False)
instance.delete()
+37 -5
View File
@@ -1,17 +1,49 @@
from django.urls import include, path
from rest_framework.routers import DefaultRouter
from .views import DownloadToLibraryView, MediaItemViewSet, UploadView
from .tools import (
ClearTempView,
DeleteFilesView,
ExactDuplicatesView,
StorageView,
TagCloudView,
VisualGroupsView,
VisualMatchesView,
)
from .uploads import TempUploadViewSet
from .views import (
ClientDownloadView,
DownloadTaskViewSet,
MatchTaskViewSet,
MediaItemViewSet,
)
router = DefaultRouter()
router.register("files", MediaItemViewSet, basename="file")
router.register("uploads", TempUploadViewSet, basename="upload")
router.register("online/downloads", DownloadTaskViewSet, basename="download")
router.register("matches", MatchTaskViewSet, basename="match")
urlpatterns = [
path("", include(router.urls)),
path("uploads/", UploadView.as_view(), name="upload"),
path("online/file/", ClientDownloadView.as_view(), name="client_download"),
path(
"online/downloads/",
DownloadToLibraryView.as_view(),
name="download_to_library",
"duplicates/md5/",
ExactDuplicatesView.as_view(),
name="duplicates_md5",
),
path(
"duplicates/visual/",
VisualMatchesView.as_view(),
name="duplicates_visual",
),
path(
"duplicates/visual_groups/",
VisualGroupsView.as_view(),
name="duplicates_groups",
),
path("delete/", DeleteFilesView.as_view(), name="delete_files"),
path("temp/clear/", ClearTempView.as_view(), name="clear_temp"),
path("tags/cloud/", TagCloudView.as_view(), name="tag_cloud"),
path("storage/", StorageView.as_view(), name="storage_info"),
]
+282 -92
View File
@@ -3,20 +3,28 @@ from pathlib import Path
from urllib.parse import urlparse
from django.conf import settings
from django.db.models import Min
from django.http import Http404
from django.core import signing
from django.db.models import Min, Q
from django.http import Http404, StreamingHttpResponse
from django.shortcuts import get_object_or_404
from django.utils import timezone
from django.utils.text import get_valid_filename
from rest_framework import mixins, status, viewsets
from rest_framework.decorators import action
from rest_framework.parsers import FormParser, MultiPartParser
from rest_framework.permissions import AllowAny, IsAuthenticatedOrReadOnly
from rest_framework.response import Response
from rest_framework.views import APIView
from . import services
from .models import MediaItem
from . import e621, matching, services
from .downloads import reap_stale_downloads, start_download_task
from .matching import reap_stale_match_tasks, start_match_task
from .models import DownloadTask, MatchTask, MediaItem
from .permissions import CanUpload, IsUploaderOrStaffOrReadOnly
from .serializers import MediaItemSerializer
from .serializers import (
DownloadTaskSerializer,
MatchTaskSerializer,
MediaItemSerializer,
)
LIST_ORDERINGS = {"name", "-name", "size", "-size", "created_at", "-created_at"}
MD5_RE = re.compile(r"[0-9a-fA-F]{32}")
@@ -38,9 +46,30 @@ class MediaItemViewSet(
)
if not self.request.user.is_authenticated:
queryset = queryset.filter(hidden_from_guests=False)
j_ids = self.request.query_params.get("j_ids", "").strip()
if j_ids:
numeric_ids = []
for value in j_ids.split(","):
text = value.strip()
number = text[2:] if text.upper().startswith("J-") else text
if number.isdigit():
numeric_ids.append(int(number))
queryset = queryset.filter(pk__in=numeric_ids)
search = self.request.query_params.get("search", "").strip()
search_type = self.request.query_params.get("search_type", "filename").strip()
if search:
queryset = queryset.filter(locations__rel_path__icontains=search)
words = [word for word in search.lower().split() if word]
if search_type == "tags":
for word in words:
queryset = queryset.filter(search_tags__icontains=word)
elif search_type == "both":
query = Q(locations__rel_path__icontains=search)
tag_query = Q()
for word in words:
tag_query &= Q(search_tags__icontains=word)
queryset = queryset.filter(query | tag_query)
else:
queryset = queryset.filter(locations__rel_path__icontains=search)
ratings = [
value
for value in self.request.query_params.get("rating", "").split(",")
@@ -48,6 +77,28 @@ class MediaItemViewSet(
]
if ratings:
queryset = queryset.filter(rating__in=ratings)
statuses = []
for value in self.request.query_params.getlist("status"):
statuses.extend(
part.strip() for part in value.split(",") if part.strip()
)
if statuses:
status_query = Q()
for status_value in set(statuses):
if status_value == "custom":
status_query |= Q(has_custom_data=True)
elif status_value == "matched":
status_query |= Q(e621_match_status=MediaItem.E621_MATCHED)
elif status_value == "not_found":
status_query |= Q(e621_match_status=MediaItem.E621_NOT_FOUND)
elif status_value == "deleted":
status_query |= Q(e621_match_status=MediaItem.E621_DELETED)
elif status_value == "unknown":
status_query |= Q(
has_custom_data=False,
e621_match_status=MediaItem.E621_UNKNOWN,
)
queryset = queryset.filter(status_query)
ordering = self.request.query_params.get("ordering", "").strip()
queryset = queryset.order_by(
ordering if ordering in LIST_ORDERINGS else "-created_at"
@@ -68,9 +119,32 @@ class MediaItemViewSet(
self.check_object_permissions(self.request, obj)
return obj
def _signed_media_item(self, request, action_name):
"""Signed media URLs let <img>/<video> tags bypass guest filtering."""
if request.user.is_authenticated:
return None
signature = request.query_params.get("sig")
if not signature:
return None
try:
payload = signing.loads(
signature, salt=services.MEDIA_FILE_SALT, max_age=86400
)
except signing.BadSignature:
return None
if payload.get("action") != action_name:
return None
return MediaItem.objects.filter(pk=payload.get("item")).first()
def _media_object(self, request, action_name):
item = self._signed_media_item(request, action_name)
if item is not None:
return item
return self.get_object()
@action(detail=True, methods=["get"])
def raw(self, request, pk=None):
item = self.get_object()
item = self._media_object(request, "raw")
location = item.locations.first()
if location is None:
return Response(
@@ -83,7 +157,7 @@ class MediaItemViewSet(
@action(detail=True, methods=["get"])
def thumbnail(self, request, pk=None):
item = self.get_object()
item = self._media_object(request, "thumbnail")
location = item.locations.first()
if location is None:
return Response(
@@ -164,108 +238,224 @@ class MediaItemViewSet(
}
)
class UploadView(APIView):
permission_classes = [CanUpload]
parser_classes = [MultiPartParser, FormParser]
def post(self, request):
upload = request.FILES.get("file")
if upload is None:
return Response(
{"detail": "A file is required."}, status=status.HTTP_400_BAD_REQUEST
)
folder = Path(settings.WATCHED_FOLDER)
destination = services.unique_destination(folder, upload.name)
with open(destination, "wb") as output:
for chunk in upload.chunks():
output.write(chunk)
try:
item, created_item, _ = services.index_file(destination, folder)
except Exception as exc: # noqa: BLE001 - report indexing failures
destination.unlink(missing_ok=True)
return Response(
{"detail": f"Could not index upload: {exc}"},
status=status.HTTP_400_BAD_REQUEST,
)
update_fields = []
if created_item and item.uploaded_by_id is None:
item.uploaded_by = request.user
update_fields.append("uploaded_by")
rating = request.data.get("rating") or ""
if rating in {"s", "q", "e"} and item.rating != rating:
item.rating = rating
update_fields.append("rating")
tags = services.parse_tags(request.data.get("tags"))
if tags is not None:
item.tags = tags
update_fields.append("tags")
notes = request.data.get("notes")
if notes is not None:
item.notes = str(notes)
update_fields.append("notes")
if update_fields:
item.save(update_fields=update_fields + ["updated_at"])
return Response(
MediaItemSerializer(item, context={"request": request}).data,
status=status.HTTP_201_CREATED,
def _can_match(self, request, item):
user = request.user
return bool(
user.is_superuser
or user.role == user.ROLE_STAFF
or item.uploaded_by_id == user.id
)
@action(detail=True, methods=["post"], permission_classes=[CanUpload])
def match(self, request, pk=None):
"""Link this item to e621: MD5 lookup, or an explicit post id."""
item = self.get_object()
if not self._can_match(request, item):
return Response(
{"detail": "Only the uploader or staff can match this item."},
status=status.HTTP_403_FORBIDDEN,
)
extra = {}
post_id = request.data.get("post_id")
try:
if post_id not in (None, ""):
text = str(post_id).strip()
if not text.isdigit():
return Response(
{"detail": "Post ID must be a number."},
status=status.HTTP_400_BAD_REQUEST,
)
extra["md5_match"] = matching.link_item_to_post(
item, int(text), request.user
)
else:
matching.check_item(item, request.user)
except e621.E621NotFound:
return Response(
{"detail": "That post does not exist on e621."},
status=status.HTTP_404_NOT_FOUND,
)
except e621.E621Error as exc:
return Response(
{"detail": str(exc)}, status=status.HTTP_502_BAD_GATEWAY
)
item.refresh_from_db()
data = dict(self.get_serializer(item).data)
data.update(extra)
return Response(data)
class DownloadToLibraryView(APIView):
"""Fetch an e621 file into the watched folder and index it."""
@action(detail=True, methods=["post"], permission_classes=[CanUpload])
def unlink(self, request, pk=None):
"""Forget this item's e621 link so it can be matched again."""
item = self.get_object()
if not self._can_match(request, item):
return Response(
{"detail": "Only the uploader or staff can unlink this item."},
status=status.HTTP_403_FORBIDDEN,
)
matching.unlink_item(item)
return Response(self.get_serializer(item).data)
class DownloadTaskViewSet(
mixins.ListModelMixin,
mixins.RetrieveModelMixin,
viewsets.GenericViewSet,
):
"""Background 'Download to Library' jobs."""
serializer_class = DownloadTaskSerializer
permission_classes = [CanUpload]
http_method_names = ["get", "post", "head", "options"]
def post(self, request):
def get_queryset(self):
reap_stale_downloads()
queryset = DownloadTask.objects.select_related("library_item")
user = self.request.user
if not (user.is_staff or user.is_superuser):
queryset = queryset.filter(user=user)
return queryset
def create(self, request):
url = str(request.data.get("url") or "").strip()
post_id = request.data.get("post_id")
filename = str(request.data.get("filename") or "").strip()
post_payload = request.data.get("post")
if not url.startswith(("http://", "https://")):
return Response(
{"detail": "A valid file URL is required."},
status=status.HTTP_400_BAD_REQUEST,
)
folder = Path(settings.WATCHED_FOLDER)
name = (
filename
or Path(urlparse(url).path).name
or f"post-{post_id or 'download'}"
trimmed = services.trim_e621_post(request.data.get("post"))
resolved_post_id = None
if trimmed is not None and trimmed.get("id") is not None:
resolved_post_id = int(trimmed["id"])
elif post_id is not None and str(post_id).isdigit():
resolved_post_id = int(post_id)
task = DownloadTask.objects.create(
user=request.user,
post_id=resolved_post_id,
url=url,
filename=filename,
e621_data=trimmed,
)
destination = services.unique_destination(folder, name)
try:
services.download_file(url, destination)
item, _, _ = services.index_file(destination, folder)
except Exception as exc: # noqa: BLE001 - report download failures
destination.unlink(missing_ok=True)
start_download_task(task.id)
return Response(
self.get_serializer(task).data, status=status.HTTP_202_ACCEPTED
)
@action(detail=True, methods=["post"])
def cancel(self, request, pk=None):
task = self.get_object()
if task.status not in {
DownloadTask.STATUS_PENDING,
DownloadTask.STATUS_DOWNLOADING,
}:
return Response(
{"detail": f"Download failed: {exc}"},
{"detail": "Task is not running."},
status=status.HTTP_400_BAD_REQUEST,
)
DownloadTask.objects.filter(id=task.id).update(
cancelled=True, updated_at=timezone.now()
)
return Response({"success": True})
class MatchTaskViewSet(
mixins.ListModelMixin,
mixins.RetrieveModelMixin,
viewsets.GenericViewSet,
):
"""Background library-wide e621 match scans."""
serializer_class = MatchTaskSerializer
permission_classes = [CanUpload]
http_method_names = ["get", "post", "head", "options"]
def get_queryset(self):
reap_stale_match_tasks()
queryset = MatchTask.objects.all()
user = self.request.user
if not (user.is_staff or user.is_superuser):
queryset = queryset.filter(user=user)
return queryset
def create(self, request):
scope = str(request.data.get("scope") or MatchTask.SCOPE_MISSING).strip()
if scope not in {MatchTask.SCOPE_MISSING, MatchTask.SCOPE_ALL}:
scope = MatchTask.SCOPE_MISSING
task = MatchTask.objects.create(user=request.user, scope=scope)
start_match_task(task.id)
return Response(
self.get_serializer(task).data, status=status.HTTP_202_ACCEPTED
)
@action(detail=True, methods=["post"])
def cancel(self, request, pk=None):
task = self.get_object()
if task.status not in {MatchTask.STATUS_PENDING, MatchTask.STATUS_RUNNING}:
return Response(
{"detail": "Task is not running."},
status=status.HTTP_400_BAD_REQUEST,
)
MatchTask.objects.filter(id=task.id).update(
cancelled=True, updated_at=timezone.now()
)
return Response({"success": True})
class ClientDownloadView(APIView):
"""Stream an e621 file straight to the browser (no library write)."""
permission_classes = [AllowAny]
def get(self, request):
url = str(request.query_params.get("url") or "").strip()
filename = str(request.query_params.get("filename") or "").strip()
parsed = urlparse(url)
if (
parsed.scheme not in {"http", "https"}
or parsed.hostname not in settings.E621_MEDIA_HOSTS
):
return Response(
{"detail": "URL not allowed."},
status=status.HTTP_400_BAD_REQUEST,
)
update_fields = []
if item.uploaded_by_id is None:
item.uploaded_by = request.user
update_fields.append("uploaded_by")
import requests
trimmed = services.trim_e621_post(post_payload)
if trimmed is not None:
if trimmed.get("id") is not None:
item.e621_post_id = int(trimmed["id"])
elif post_id and str(post_id).isdigit():
item.e621_post_id = int(post_id)
item.e621_data = trimmed
update_fields += ["e621_post_id", "e621_data"]
if not item.rating and trimmed.get("rating") in {"s", "q", "e"}:
item.rating = trimmed["rating"]
update_fields.append("rating")
try:
upstream = requests.get(
url,
headers={"User-Agent": settings.USER_AGENT},
stream=True,
timeout=60,
)
upstream.raise_for_status()
except requests.RequestException as exc:
return Response(
{"detail": f"Could not fetch the file: {exc}"},
status=status.HTTP_502_BAD_GATEWAY,
)
if update_fields:
item.save(update_fields=update_fields + ["updated_at"])
return Response(
MediaItemSerializer(item, context={"request": request}).data,
status=status.HTTP_201_CREATED,
content_type = upstream.headers.get(
"Content-Type", "application/octet-stream"
)
name = get_valid_filename(
filename or Path(parsed.path).name or "download"
)
def stream():
try:
for chunk in upstream.iter_content(chunk_size=64 * 1024):
if chunk:
yield chunk
finally:
upstream.close()
response = StreamingHttpResponse(stream(), content_type=content_type)
response["Content-Disposition"] = f'attachment; filename="{name}"'
length = upstream.headers.get("Content-Length")
if length:
response["Content-Length"] = length
return response
+13
View File
@@ -52,6 +52,7 @@ INSTALLED_APPS = [
"django_filters",
"apps.accounts",
"apps.library",
"apps.follows",
"apps.core",
]
@@ -144,6 +145,15 @@ WATCHED_FOLDER = str(WATCHED_FOLDER)
E621_BASE_URL = os.getenv("E621_BASE_URL", "https://e621.net").rstrip("/")
USER_AGENT = os.getenv("USER_AGENT", "J621/0.1 (by J621 on e621)")
# Hosts the client-download proxy is allowed to stream from.
E621_MEDIA_HOSTS = [
host.strip()
for host in os.getenv(
"E621_MEDIA_HOSTS",
"static1.e621.net,static2.e621.net,static3.e621.net",
).split(",")
if host.strip()
]
# Guest visibility: e621's anonymous default blacklist is mirrored into the
# cache by `manage.py refresh_guest_blacklist`. This fallback is used until
@@ -157,6 +167,9 @@ GUEST_BLACKLIST_FALLBACK = [
]
GUEST_BLACKLIST_TTL = int(os.getenv("GUEST_BLACKLIST_TTL", "3600"))
# Similarity threshold for flagging staged uploads that match library items.
VISUAL_MATCH_THRESHOLD = float(os.getenv("VISUAL_MATCH_THRESHOLD", "0.9"))
# Redis cache (run via docker compose at the repo root), shared by web
# workers and management commands (e.g. the mirrored guest blacklist).
CACHES = {
+2
View File
@@ -6,8 +6,10 @@ from django.urls import include, path
urlpatterns = [
path("admin/", admin.site.urls),
path("api/auth/", include("apps.accounts.urls")),
path("api/", include("apps.accounts.api_urls")),
path("api/", include("apps.core.urls")),
path("api/", include("apps.library.urls")),
path("api/", include("apps.follows.urls")),
]
if settings.DEBUG:
+1
View File
@@ -4,5 +4,6 @@ django-filter>=25.1
Pillow>=11.0
python-dotenv>=1.0
requests>=2.32
imagehash>=4.3
PyMySQL>=1.1
redis>=5.0
+55 -1
View File
@@ -1,4 +1,4 @@
import { useEffect, type ReactNode } from "react";
import { useEffect, useState, type ReactNode } from "react";
import {
Navigate,
Route,
@@ -10,13 +10,21 @@ import {
import { AppShell } from "@/components/AppShell";
import { FullPageSpinner } from "@/components/ui";
import AccountPage from "@/features/account/AccountPage";
import { AgeGate } from "@/features/age/AgeGate";
import LoginPage from "@/features/auth/LoginPage";
import RegisterPage from "@/features/auth/RegisterPage";
import DetailPage from "@/features/detail/DetailPage";
import DeletePage from "@/features/library/DeletePage";
import DuplicatesPage from "@/features/library/DuplicatesPage";
import FollowedPage from "@/features/follows/FollowedPage";
import LibraryPage from "@/features/library/LibraryPage";
import Md5Redirect from "@/features/library/Md5Redirect";
import OnlinePage from "@/features/online/OnlinePage";
import PoolDetailPage from "@/features/pools/PoolDetailPage";
import PoolsPage from "@/features/pools/PoolsPage";
import UploadPage from "@/features/upload/UploadPage";
import UsersPage from "@/features/users/UsersPage";
import { isAgeVerified, markAgeVerified } from "@/lib/age";
import { useAuth } from "@/store/auth";
function RequireAuth({ children }: { children: ReactNode }) {
@@ -36,11 +44,23 @@ function PostRedirect() {
export default function App() {
const ready = useAuth((state) => state.ready);
const loadUser = useAuth((state) => state.loadUser);
const [ageVerified, setAgeVerified] = useState(isAgeVerified);
useEffect(() => {
void loadUser();
}, [loadUser]);
if (!ageVerified) {
return (
<AgeGate
onEnter={() => {
markAgeVerified();
setAgeVerified(true);
}}
/>
);
}
if (!ready) {
return <FullPageSpinner />;
}
@@ -53,6 +73,40 @@ export default function App() {
<Route path="/view/:md5" element={<Md5Redirect />} />
<Route path="/online" element={<OnlinePage />} />
<Route path="/online/view/:postId" element={<PostRedirect />} />
<Route path="/pools" element={<PoolsPage />} />
<Route path="/pools/:poolId" element={<PoolDetailPage />} />
<Route
path="/followed"
element={
<RequireAuth>
<FollowedPage />
</RequireAuth>
}
/>
<Route
path="/duplicates"
element={
<RequireAuth>
<DuplicatesPage />
</RequireAuth>
}
/>
<Route
path="/delete"
element={
<RequireAuth>
<DeletePage />
</RequireAuth>
}
/>
<Route
path="/users"
element={
<RequireAuth>
<UsersPage />
</RequireAuth>
}
/>
<Route
path="/upload"
element={
+25 -6
View File
@@ -28,8 +28,8 @@ export function AppShell() {
const statusQuery = useQuery({
queryKey: ["status"],
queryFn: () => api<SystemStatus>("/api/status/"),
refetchInterval: 30_000,
staleTime: 15_000,
refetchInterval: 5_000,
staleTime: 4_000,
});
const status = statusQuery.data;
@@ -60,7 +60,18 @@ export function AppShell() {
const navItems = [
{ to: "/", label: "Library" },
{ to: "/online", label: "Online" },
...(user?.can_upload ? [{ to: "/upload", label: "Upload" }] : []),
{ to: "/pools", label: "Pools" },
...(user ? [{ to: "/followed", label: "Followed" }] : []),
...(user?.can_upload
? [
{ to: "/upload", label: "Upload" },
{ to: "/duplicates", label: "Duplicates" },
{ to: "/delete", label: "Delete" },
]
: []),
...(user && (user.is_staff || user.is_superuser || user.role === "staff")
? [{ to: "/users", label: "Users" }]
: []),
];
const navigationType = useNavigationType();
@@ -120,9 +131,17 @@ export function AppShell() {
title="Account settings"
className="flex items-center gap-2 transition hover:opacity-80"
>
<span className="flex h-7 w-7 items-center justify-center rounded-full bg-ctp-surface1 font-mono text-xs font-semibold text-ctp-lavender">
{user.username.slice(0, 1).toUpperCase()}
</span>
{user.avatar_url ? (
<img
src={user.avatar_url}
alt={user.username}
className="h-7 w-7 rounded-full border border-ctp-surface1 object-cover"
/>
) : (
<span className="flex h-7 w-7 items-center justify-center rounded-full bg-ctp-surface1 font-mono text-xs font-semibold text-ctp-lavender">
{user.username.slice(0, 1).toUpperCase()}
</span>
)}
<span className="hidden text-sm text-ctp-subtext1 sm:block">
{user.username}
</span>
@@ -1,4 +1,5 @@
import {
Copy,
FolderOpen,
Globe,
LogIn,
@@ -6,7 +7,9 @@ import {
Menu,
Search,
Settings,
Trash2,
Upload,
Users,
type LucideIcon,
} from "lucide-react";
import { useMemo, useState } from "react";
@@ -59,6 +62,18 @@ function CommandPaletteDialog({ onClose }: { onClose: () => void }) {
icon: Upload,
run: () => navigate("/upload"),
},
{
id: "duplicates",
label: "Find duplicates",
icon: Copy,
run: () => navigate("/duplicates"),
},
{
id: "delete",
label: "Delete files & storage",
icon: Trash2,
run: () => navigate("/delete"),
},
]
: []),
{
@@ -94,6 +109,14 @@ function CommandPaletteDialog({ onClose }: { onClose: () => void }) {
icon: Settings,
run: () => navigate("/account"),
});
if (user.is_staff || user.is_superuser || user.role === "staff") {
list.push({
id: "users",
label: "Manage users",
icon: Users,
run: () => navigate("/users"),
});
}
list.push({
id: "logout",
label: `Log out (${user.username})`,
+1 -3
View File
@@ -18,9 +18,7 @@ const ratingLabels: Record<string, string> = {
export function MediaCard({ item }: { item: MediaItem }) {
const preview =
item.kind === "video"
? `/api/files/${item.j_id}/thumbnail/`
: `/api/files/${item.j_id}/raw/`;
item.kind === "video" ? item.thumbnail_url : item.raw_url;
const rating = item.display_rating;
return (
+81 -6
View File
@@ -1,4 +1,5 @@
import { Laptop, Monitor, Terminal, type LucideIcon } from "lucide-react";
import { useEffect, useRef, useState } from "react";
import { cn } from "@/lib/cn";
import type { SystemStatus } from "@/lib/types";
@@ -16,17 +17,45 @@ function formatMs(value: number): string {
return `${value.toFixed(1)}ms`;
}
function formatClock(at: number): string {
return new Date(at).toLocaleTimeString();
}
export function StatusPill({ status }: { status?: SystemStatus }) {
const requestCount = useE621Metrics((state) => state.requestCount);
const totalMs = useE621Metrics((state) => state.totalMs);
const last = useE621Metrics((state) => state.last);
const history = useE621Metrics((state) => state.history);
const [open, setOpen] = useState(false);
const containerRef = useRef<HTMLSpanElement>(null);
useEffect(() => {
if (!open) return;
function handlePointerDown(event: MouseEvent) {
if (
containerRef.current &&
!containerRef.current.contains(event.target as Node)
) {
setOpen(false);
}
}
function handleKeyDown(event: KeyboardEvent) {
if (event.key === "Escape") setOpen(false);
}
document.addEventListener("mousedown", handlePointerDown);
document.addEventListener("keydown", handleKeyDown);
return () => {
document.removeEventListener("mousedown", handlePointerDown);
document.removeEventListener("keydown", handleKeyDown);
};
}, [open]);
if (!status) return null;
const OsIcon = osIcons[status.os.icon] ?? Monitor;
return (
<span
className="hidden select-none items-center gap-1.5 rounded-full border border-ctp-surface0 bg-ctp-mantle px-2.5 py-1 font-mono text-[11px] text-ctp-overlay1 lg:flex"
ref={containerRef}
className="relative hidden select-none items-center gap-1.5 rounded-full border border-ctp-surface0 bg-ctp-mantle px-2.5 py-1 font-mono text-[11px] text-ctp-overlay1 lg:flex"
title={`${status.os.name} ${status.os.release} · built from ${status.git_hash}`}
>
<span
@@ -47,14 +76,19 @@ export function StatusPill({ status }: { status?: SystemStatus }) {
{requestCount > 0 ? (
<>
<span className="text-ctp-surface2">·</span>
<span
className="text-ctp-teal"
<button
type="button"
onClick={() => setOpen((value) => !value)}
title={`${requestCount} e621 request${
requestCount === 1 ? "" : "s"
} this session · last ${last ? `${formatMs(last.ms)} (${last.path})` : "—"}`}
} this session — click for details`}
className={cn(
"text-ctp-teal transition hover:underline",
open && "underline",
)}
>
e621: {formatMs(totalMs)}
</span>
</button>
</>
) : null}
{status.e621_time_ms !== null ? (
@@ -65,6 +99,47 @@ export function StatusPill({ status }: { status?: SystemStatus }) {
</span>
</>
) : null}
{open && requestCount > 0 ? (
<span className="absolute right-0 top-full z-50 mt-2 block w-80 cursor-default overflow-hidden rounded-lg border border-ctp-surface1 bg-ctp-surface0 text-left shadow-[0_4px_12px_rgba(17,17,27,0.6)]">
<span className="flex items-center justify-between gap-2 border-b border-ctp-surface1 px-3 py-2 text-[11px] text-ctp-overlay0">
<span className="font-sans">e621 requests this session</span>
<span className="font-mono">
{requestCount} · {formatMs(totalMs)}
</span>
</span>
<span className="block max-h-72 overflow-y-auto py-1">
{history.map((sample, index) => (
<span
key={`${sample.at}-${index}`}
className="flex items-center gap-2 px-3 py-1 font-mono text-[11px]"
>
<span className="shrink-0 text-ctp-overlay0">
{formatClock(sample.at)}
</span>
<span
className="min-w-0 flex-1 truncate text-ctp-subtext0"
title={sample.path}
>
{sample.path}
</span>
<span
className={cn(
"shrink-0",
sample.ms >= 2000
? "text-ctp-peach"
: sample.ms >= 800
? "text-ctp-yellow"
: "text-ctp-teal",
)}
>
{formatMs(sample.ms)}
</span>
</span>
))}
</span>
</span>
) : null}
</span>
);
}
+18
View File
@@ -1,4 +1,5 @@
import { cn } from "@/lib/cn";
import { TagFollowToggle } from "@/features/follows/TagFollowToggle";
const TAG_STYLES: Record<string, string> = {
general:
@@ -26,11 +27,13 @@ export function TagChip({
category,
count,
onClick,
followable = false,
}: {
tag: string;
category?: string;
count?: number;
onClick?: () => void;
followable?: boolean;
}) {
const style = TAG_STYLES[category ?? "unknown"] ?? TAG_STYLES.unknown;
const className = cn(
@@ -39,6 +42,21 @@ export function TagChip({
);
const label = count !== undefined ? `${tag} (${count})` : tag;
if (followable) {
return (
<span className={cn(className, "inline-flex items-center gap-1")}>
{onClick ? (
<button type="button" onClick={onClick} className="text-left">
{label}
</button>
) : (
<span>{label}</span>
)}
<TagFollowToggle tag={tag} />
</span>
);
}
if (!onClick) {
return <span className={className}>{label}</span>;
}
+75
View File
@@ -0,0 +1,75 @@
import { Ban, ShieldAlert } from "lucide-react";
import { useState } from "react";
import { Button } from "@/components/ui";
export function AgeGate({ onEnter }: { onEnter: () => void }) {
const [declined, setDeclined] = useState(false);
if (declined) {
return (
<div className="flex min-h-screen items-center justify-center bg-ctp-crust px-4">
<div className="w-full max-w-md rounded-lg border border-ctp-red/40 bg-ctp-base p-6 text-center">
<Ban className="mx-auto h-7 w-7 text-ctp-red" />
<h1 className="mt-3 text-lg font-semibold text-ctp-text">
Access restricted
</h1>
<p className="mt-2 text-sm leading-relaxed text-ctp-subtext0">
You have to be 18 years or older to view this archive. Come back
when that applies.
</p>
<Button
variant="secondary"
className="mt-5"
onClick={() => setDeclined(false)}
>
Go back
</Button>
</div>
</div>
);
}
return (
<div className="flex min-h-screen items-center justify-center bg-ctp-crust px-4">
<div className="w-full max-w-lg">
<div className="flex items-center gap-3">
<span className="rounded bg-ctp-mauve px-2.5 py-1.5 font-mono text-sm font-bold tracking-wide text-ctp-crust">
J621
</span>
<span className="font-mono text-[11px] uppercase tracking-widest text-ctp-overlay0">
Self-hosted media archive
</span>
</div>
<div className="mt-6 rounded-lg border border-ctp-surface0 bg-ctp-base p-6">
<div className="flex items-start gap-3">
<ShieldAlert className="mt-0.5 h-5 w-5 shrink-0 text-ctp-peach" />
<h1 className="text-lg font-semibold leading-snug text-ctp-text">
You have to be 18+ to access this site.
</h1>
</div>
<p className="mt-3 text-sm leading-relaxed text-ctp-subtext0">
This archive contains adult material. By entering you confirm that
you are at least 18 years old and that viewing adult content is
legal where you live.
</p>
<div className="mt-6 flex flex-col gap-3 sm:flex-row">
<Button className="flex-1" autoFocus onClick={onEnter}>
I am 18 or older — Enter
</Button>
<Button variant="secondary" onClick={() => setDeclined(true)}>
Leave
</Button>
</div>
</div>
<p className="mt-4 text-center font-mono text-[11px] text-ctp-overlay0">
Local-first library · e621 integration · nothing leaves this server
</p>
</div>
</div>
);
}
@@ -0,0 +1,527 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Eye, StarOff } from "lucide-react";
import { useState } from "react";
import { Link } from "react-router-dom";
import { Button, EmptyState, Spinner } from "@/components/ui";
import { api } from "@/lib/api";
import { cn } from "@/lib/cn";
import { formatDate } from "@/lib/format";
import type {
FollowCloud,
FollowCoverData,
FollowedPool,
FollowedTag,
FollowFeed,
Paginated,
} from "@/lib/types";
const ratingPill: Record<string, string> = {
s: "bg-ctp-green text-ctp-crust",
q: "bg-ctp-peach text-ctp-crust",
e: "bg-ctp-red text-ctp-crust",
};
function invalidateFollows(queryClient: ReturnType<typeof useQueryClient>) {
void queryClient.invalidateQueries({ queryKey: ["follows-tags"] });
void queryClient.invalidateQueries({ queryKey: ["follows-pools"] });
void queryClient.invalidateQueries({ queryKey: ["follows-feed"] });
}
function unseenLabel(count: number): string {
return count > 99 ? "99+ unseen" : `${count} unseen`;
}
function FollowCard({
title,
meta,
cover,
unseen,
syncedAt,
selected,
onSelect,
onSeen,
onUnfollow,
}: {
title: string;
meta: string;
cover: FollowCoverData | null;
unseen: number;
syncedAt: string | null;
selected: boolean;
onSelect: () => void;
onSeen: () => void;
onUnfollow: () => void;
}) {
const metaLine = [meta, syncedAt ? `synced ${formatDate(syncedAt)}` : null]
.filter(Boolean)
.join(" · ");
return (
<div
className={cn(
"overflow-hidden rounded-lg border bg-ctp-base transition",
selected
? "border-ctp-mauve ring-2 ring-ctp-mauve/30"
: "border-ctp-surface0 hover:border-ctp-surface2",
)}
>
<button
type="button"
onClick={onSelect}
className="relative block aspect-[4/3] w-full overflow-hidden bg-ctp-mantle"
title={selected ? "Clear the feed filter" : "Show only this in the feed"}
>
{cover?.preview_url ? (
<img
src={cover.preview_url}
alt={title}
loading="lazy"
className="h-full w-full object-cover"
/>
) : (
<span className="flex h-full w-full items-center justify-center text-xs text-ctp-overlay0">
No cover yet — run a sync
</span>
)}
{unseen > 0 ? (
<span className="absolute right-2 top-2 rounded-full bg-ctp-red px-2 py-0.5 font-mono text-[10px] font-bold text-ctp-crust">
{unseenLabel(unseen)}
</span>
) : null}
{cover?.rating ? (
<span
className={cn(
"absolute left-2 top-2 rounded-full px-1.5 py-0.5 font-mono text-[10px] font-semibold uppercase",
ratingPill[cover.rating] ?? "bg-ctp-surface1 text-ctp-text",
)}
>
{cover.rating}
</span>
) : null}
</button>
<div className="p-3">
<div className="flex items-start justify-between gap-2">
<div className="min-w-0">
<p
className="truncate font-mono text-sm text-ctp-text"
title={title}
>
{title}
</p>
{metaLine ? (
<p className="truncate text-[11px] text-ctp-overlay0">
{metaLine}
</p>
) : null}
</div>
<button
type="button"
onClick={onUnfollow}
title="Unfollow"
className="shrink-0 rounded p-1 text-ctp-overlay0 transition hover:bg-ctp-red/15 hover:text-ctp-red"
>
<StarOff className="h-4 w-4" />
</button>
</div>
<div className="mt-2 flex items-center gap-2">
<Button
variant="secondary"
className="px-2 py-1 text-xs"
disabled={unseen === 0}
onClick={onSeen}
>
<Eye className="h-3.5 w-3.5" /> Mark seen
</Button>
</div>
</div>
</div>
);
}
function BlacklistCloudPanel() {
const query = useQuery({
queryKey: ["follows-cloud"],
queryFn: () => api<FollowCloud>("/api/follows/cloud/"),
refetchInterval: (q) =>
q.state.data?.status === "building" ? 2000 : false,
});
const cloud = query.data;
return (
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
<div className="flex flex-wrap items-center justify-between gap-2">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Blacklisted tags
</h2>
<p className="text-xs text-ctp-overlay0">
{cloud?.source === "user"
? "From your e621 blacklist"
: "From e621's anonymous default blacklist"}
{cloud ? ` · ${cloud.blacklist_count} entries` : ""}
{cloud ? ` · ${cloud.posts} feed post(s) scanned` : ""}
{cloud?.computed_at ? ` · computed ${formatDate(cloud.computed_at)}` : ""}
{cloud?.status === "building" ? (
<span className="ml-2 inline-flex items-center gap-1.5">
<Spinner className="h-3 w-3" /> building…
</span>
) : null}
</p>
</div>
{query.isPending ? (
<div className="mt-3 flex justify-center py-4">
<Spinner className="h-4 w-4" />
</div>
) : cloud && cloud.tags.length > 0 ? (
<div className="mt-3 flex flex-wrap gap-1.5">
{cloud.tags.map(([tag, count]) => (
<span
key={tag}
className="rounded border border-ctp-red/30 bg-ctp-red/10 px-1.5 py-0.5 font-mono text-[11px] text-ctp-red"
>
{tag} ({count})
</span>
))}
</div>
) : (
<p className="mt-3 text-xs text-ctp-overlay0">
No blacklisted tags in your feeds.
</p>
)}
</section>
);
}
export default function FollowedPage() {
const queryClient = useQueryClient();
const [feedSource, setFeedSource] = useState<{
kind: "all" | "tag" | "pool";
id?: number;
label?: string;
}>({ kind: "all" });
const [unseenOnly, setUnseenOnly] = useState(false);
const [page, setPage] = useState(1);
const tagsQuery = useQuery({
queryKey: ["follows-tags"],
queryFn: () => api<Paginated<FollowedTag>>("/api/follows/tags/"),
});
const poolsQuery = useQuery({
queryKey: ["follows-pools"],
queryFn: () => api<Paginated<FollowedPool>>("/api/follows/pools/"),
});
const feedQuery = useQuery({
queryKey: [
"follows-feed",
feedSource.kind,
feedSource.id ?? null,
unseenOnly,
page,
],
queryFn: () => {
const params = new URLSearchParams({
page: String(page),
per_page: "48",
});
if (feedSource.kind === "tag") params.set("tag", String(feedSource.id));
if (feedSource.kind === "pool") params.set("pool", String(feedSource.id));
if (unseenOnly) params.set("unseen", "1");
return api<FollowFeed>(`/api/follows/feed/?${params.toString()}`);
},
});
const seenFollow = useMutation({
mutationFn: ({ kind, id }: { kind: "tag" | "pool"; id: number }) =>
api(`/api/follows/${kind === "tag" ? "tags" : "pools"}/${id}/seen/`, {
method: "POST",
}),
onSuccess: () => invalidateFollows(queryClient),
});
const seenPost = useMutation({
mutationFn: (postId: number) =>
api(`/api/follows/posts/${postId}/seen/`, { method: "POST" }),
onSuccess: () => invalidateFollows(queryClient),
});
const unfollow = useMutation({
mutationFn: ({ kind, id }: { kind: "tag" | "pool"; id: number }) =>
api(`/api/follows/${kind === "tag" ? "tags" : "pools"}/${id}/`, {
method: "DELETE",
}),
onSuccess: () => invalidateFollows(queryClient),
});
function selectSource(source: {
kind: "all" | "tag" | "pool";
id?: number;
label?: string;
}) {
setPage(1);
setFeedSource((current) =>
current.kind === source.kind && current.id === source.id
? { kind: "all" }
: source,
);
}
const tags = tagsQuery.data?.results ?? [];
const pools = poolsQuery.data?.results ?? [];
const feed = feedQuery.data;
return (
<div className="mx-auto flex w-full max-w-6xl flex-col gap-6">
<header>
<h1 className="text-lg font-semibold">Followed</h1>
<p className="mt-1 text-sm text-ctp-overlay0">
New posts from your followed e621 tags and pools. Follow tags with
the + on any tag chip in the online or library detail views, and
pools from a pool page. The periodic commands refresh these feeds
and their covers.
</p>
</header>
<section className="flex flex-col gap-3">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Tags{" "}
<span className="font-mono text-[11px] text-ctp-overlay0">
{tags.length}
</span>
</h2>
{tagsQuery.isPending ? (
<div className="flex justify-center py-8">
<Spinner className="h-5 w-5" />
</div>
) : tags.length === 0 ? (
<p className="text-xs text-ctp-overlay0">
No followed tags yet — use the + on any tag chip.
</p>
) : (
<div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
{tags.map((follow) => (
<FollowCard
key={follow.id}
title={follow.tag}
meta={
follow.cover_data?.post_id
? `latest #${follow.cover_data.post_id}`
: ""
}
cover={follow.cover_data}
unseen={follow.unseen_count}
syncedAt={follow.last_synced_at}
selected={
feedSource.kind === "tag" && feedSource.id === follow.id
}
onSelect={() =>
selectSource({
kind: "tag",
id: follow.id,
label: `tag:${follow.tag}`,
})
}
onSeen={() => seenFollow.mutate({ kind: "tag", id: follow.id })}
onUnfollow={() => unfollow.mutate({ kind: "tag", id: follow.id })}
/>
))}
</div>
)}
</section>
<section className="flex flex-col gap-3">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Pools{" "}
<span className="font-mono text-[11px] text-ctp-overlay0">
{pools.length}
</span>
</h2>
{poolsQuery.isPending ? (
<div className="flex justify-center py-8">
<Spinner className="h-5 w-5" />
</div>
) : pools.length === 0 ? (
<p className="text-xs text-ctp-overlay0">
No followed pools yet — open a pool and use "Follow pool".
</p>
) : (
<div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
{pools.map((follow) => (
<FollowCard
key={follow.id}
title={follow.name || `Pool #${follow.pool_id}`}
meta={`#${follow.pool_id}${
follow.cover_data?.post_id
? ` · latest #${follow.cover_data.post_id}`
: ""
}`}
cover={follow.cover_data}
unseen={follow.unseen_count}
syncedAt={follow.last_synced_at}
selected={
feedSource.kind === "pool" && feedSource.id === follow.id
}
onSelect={() =>
selectSource({
kind: "pool",
id: follow.id,
label: follow.name || `Pool #${follow.pool_id}`,
})
}
onSeen={() =>
seenFollow.mutate({ kind: "pool", id: follow.id })
}
onUnfollow={() =>
unfollow.mutate({ kind: "pool", id: follow.id })
}
/>
))}
</div>
)}
</section>
<section className="flex flex-col gap-3">
<div className="flex flex-wrap items-center justify-between gap-2">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Feed{" "}
<span className="font-mono text-[11px] text-ctp-overlay0">
{feed ? `${feed.count}` : "…"}
{feedSource.kind !== "all" ? ` · showing ${feedSource.label}` : ""}
</span>
</h2>
<div className="flex items-center gap-2">
<button
type="button"
onClick={() => {
setPage(1);
setUnseenOnly((value) => !value);
}}
className={cn(
"rounded-full border px-2.5 py-1 text-xs font-medium transition",
unseenOnly
? "border-ctp-blue/40 bg-ctp-blue/15 text-ctp-blue"
: "border-ctp-surface1 text-ctp-subtext0 hover:border-ctp-surface2 hover:text-ctp-text",
)}
>
Unseen only
</button>
{feedSource.kind !== "all" ? (
<button
type="button"
onClick={() => {
setPage(1);
setFeedSource({ kind: "all" });
}}
className="text-xs text-ctp-blue hover:underline"
>
Clear filter
</button>
) : null}
</div>
</div>
{feedQuery.isPending ? (
<div className="flex justify-center py-12">
<Spinner className="h-5 w-5" />
</div>
) : !feed || feed.results.length === 0 ? (
<EmptyState
title="Nothing in the feed"
description={
unseenOnly
? "Everything here is marked seen."
: "Follow tags or pools, or run a sync command to fetch posts."
}
/>
) : (
<div className="grid grid-cols-2 gap-3 sm:grid-cols-3 xl:grid-cols-4">
{feed.results.map((entry) => (
<Link
key={entry.post_id}
to={`/detail/${entry.post_id}`}
onClick={() => {
if (!entry.seen) seenPost.mutate(entry.post_id);
}}
className="group block overflow-hidden rounded-lg border border-ctp-surface0 bg-ctp-base transition hover:border-ctp-surface2"
>
<span className="relative block aspect-square overflow-hidden bg-ctp-mantle">
{entry.data.preview?.url ? (
<img
src={entry.data.preview.url}
alt={`Post #${entry.post_id}`}
loading="lazy"
className="h-full w-full object-cover transition group-hover:scale-[1.03]"
/>
) : (
<span className="flex h-full w-full items-center justify-center text-[10px] text-ctp-overlay0">
No preview
</span>
)}
{!entry.seen ? (
<span
className="absolute left-2 top-2 h-2.5 w-2.5 rounded-full bg-ctp-blue ring-2 ring-ctp-crust"
title="Unseen"
/>
) : null}
{entry.data.rating ? (
<span
className={cn(
"absolute right-2 top-2 rounded-full px-1.5 py-0.5 font-mono text-[10px] font-semibold uppercase",
ratingPill[entry.data.rating] ??
"bg-ctp-surface1 text-ctp-text",
)}
>
{entry.data.rating}
</span>
) : null}
</span>
<span className="block px-2 py-1.5">
<span className="block font-mono text-[11px] text-ctp-subtext0">
#{entry.post_id}
<span className="text-ctp-overlay0">
{" "}
▲ {entry.data.score?.total ?? 0}
{entry.data.fav_count
? ` · ${entry.data.fav_count} ★`
: ""}
</span>
</span>
<span className="block truncate text-[10px] text-ctp-overlay0">
{entry.sources.map((source) => source.label).join(", ")}
</span>
</span>
</Link>
))}
</div>
)}
{feed && (page > 1 || feed.has_next) ? (
<div className="flex items-center justify-center gap-3">
<Button
variant="secondary"
disabled={page <= 1 || feedQuery.isFetching}
onClick={() => setPage((current) => Math.max(1, current - 1))}
>
Previous
</Button>
<span className="font-mono text-xs text-ctp-subtext0">
Page {page}
</span>
<Button
variant="secondary"
disabled={!feed.has_next || feedQuery.isFetching}
onClick={() => setPage((current) => current + 1)}
>
Next
</Button>
</div>
) : null}
</section>
<BlacklistCloudPanel />
</div>
);
}
@@ -0,0 +1,82 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Check, Star } from "lucide-react";
import { useState } from "react";
import { Button, Spinner } from "@/components/ui";
import { api, errorMessage } from "@/lib/api";
import type { FollowedPool, Paginated } from "@/lib/types";
import { useAuth } from "@/store/auth";
export function PoolFollowButton({
poolId,
name,
}: {
poolId: number;
name?: string;
}) {
const user = useAuth((state) => state.user);
const queryClient = useQueryClient();
const [error, setError] = useState<string | null>(null);
// What the button should show once the request and refetch settle.
const [target, setTarget] = useState<boolean | null>(null);
const query = useQuery({
queryKey: ["follows-pools"],
queryFn: () => api<Paginated<FollowedPool>>("/api/follows/pools/"),
enabled: Boolean(user),
staleTime: 30_000,
});
const follow = query.data?.results.find((entry) => entry.pool_id === poolId);
const followed = Boolean(follow);
const mutation = useMutation({
mutationFn: () =>
follow
? api(`/api/follows/pools/${follow.id}/`, { method: "DELETE" })
: api<FollowedPool>("/api/follows/pools/", {
method: "POST",
json: { pool_id: poolId },
}),
onSuccess: () => {
setError(null);
void queryClient.invalidateQueries({ queryKey: ["follows-pools"] });
void queryClient.invalidateQueries({ queryKey: ["follows-feed"] });
void queryClient.invalidateQueries({ queryKey: ["follows-cloud"] });
},
onError: (err) => {
setTarget(null);
setError(errorMessage(err));
},
});
if (!user) return null;
const busy = mutation.isPending || (target !== null && followed !== target);
const label = name ? `"${name}"` : `#${poolId}`;
return (
<div className="flex flex-col items-end gap-1">
<Button
variant={followed ? "secondary" : "primary"}
disabled={busy || query.isPending}
onClick={() => {
if (busy) return;
setTarget(!followed);
mutation.mutate();
}}
title={followed ? `Unfollow pool ${label}` : `Follow pool ${label}`}
>
{busy ? (
<Spinner className="h-3.5 w-3.5" />
) : followed ? (
<Check className="h-4 w-4" />
) : (
<Star className="h-4 w-4" />
)}
{followed ? "Following" : "Follow pool"}
</Button>
{error ? <p className="text-xs text-ctp-red">{error}</p> : null}
</div>
);
}
@@ -0,0 +1,90 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Check, Plus } from "lucide-react";
import { useState, type MouseEvent } from "react";
import { api, errorMessage } from "@/lib/api";
import { cn } from "@/lib/cn";
import type { FollowedTag, Paginated } from "@/lib/types";
import { useAuth } from "@/store/auth";
export function TagFollowToggle({ tag }: { tag: string }) {
const user = useAuth((state) => state.user);
const queryClient = useQueryClient();
const [error, setError] = useState<string | null>(null);
// What the chip should show once the request and refetch settle.
const [target, setTarget] = useState<boolean | null>(null);
const query = useQuery({
queryKey: ["follows-tags"],
queryFn: () => api<Paginated<FollowedTag>>("/api/follows/tags/"),
enabled: Boolean(user),
staleTime: 30_000,
});
const normalized = tag.trim().toLowerCase();
const follow = query.data?.results.find((entry) => entry.tag === normalized);
const followed = Boolean(follow);
const mutation = useMutation({
mutationFn: () =>
follow
? api(`/api/follows/tags/${follow.id}/`, { method: "DELETE" })
: api<FollowedTag>("/api/follows/tags/", {
method: "POST",
json: { tag: normalized },
}),
onSuccess: () => {
setError(null);
void queryClient.invalidateQueries({ queryKey: ["follows-tags"] });
void queryClient.invalidateQueries({ queryKey: ["follows-feed"] });
void queryClient.invalidateQueries({ queryKey: ["follows-cloud"] });
},
onError: (err) => {
setTarget(null);
setError(errorMessage(err));
},
});
// Keep the spinner up until the refetched list reflects the new state.
// A stale target that matches `followed` is harmless: the next click
// recomputes it before mutating.
const busy = mutation.isPending || (target !== null && followed !== target);
if (!user) return null;
function handleClick(event: MouseEvent<HTMLButtonElement>) {
// Chips often sit inside clickable labels/links; the toggle is its own
// action and must not trigger them.
event.preventDefault();
event.stopPropagation();
if (busy) return;
setTarget(!followed);
mutation.mutate();
}
return (
<button
type="button"
onClick={handleClick}
disabled={busy || query.isPending}
title={error ?? (followed ? `Unfollow ${tag}` : `Follow ${tag}`)}
aria-label={followed ? `Unfollow ${tag}` : `Follow ${tag}`}
className={cn(
"-mr-0.5 inline-flex h-3.5 w-3.5 shrink-0 items-center justify-center rounded-sm transition",
error ? "text-ctp-red" : "opacity-70 hover:opacity-100",
query.isPending && "opacity-40",
)}
>
{busy ? (
<span
aria-hidden="true"
className="h-3 w-3 animate-spin rounded-full border-[1.5px] border-current border-t-transparent"
/>
) : followed ? (
<Check className="h-3 w-3" strokeWidth={3} />
) : (
<Plus className="h-3 w-3" strokeWidth={3} />
)}
</button>
);
}
@@ -0,0 +1,271 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { useState } from "react";
import { Link } from "react-router-dom";
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
import { api, errorMessage } from "@/lib/api";
import { cn } from "@/lib/cn";
import { formatBytes } from "@/lib/format";
import type { MediaItem, Paginated, StorageInfo } from "@/lib/types";
function parseJIds(text: string): string[] {
return text
.split(/[\s,]+/)
.map((value) => value.trim().toUpperCase())
.filter((value) => /^J-\d+$/.test(value));
}
export default function DeletePage() {
const queryClient = useQueryClient();
const [jIdsInput, setJIdsInput] = useState("");
const [loadedIds, setLoadedIds] = useState<string[]>([]);
const [selected, setSelected] = useState<string[]>([]);
const [status, setStatus] = useState<string | null>(null);
const storageQuery = useQuery({
queryKey: ["storage"],
queryFn: () => api<StorageInfo>("/api/storage/"),
});
const filesQuery = useQuery({
queryKey: ["delete-files", loadedIds],
queryFn: () =>
api<Paginated<MediaItem>>(`/api/files/?j_ids=${loadedIds.join(",")}`),
enabled: loadedIds.length > 0,
});
const deleteMutation = useMutation({
mutationFn: (jIds: string[]) =>
api<{ deleted: string[]; errors: { j_id?: string; error: string }[] }>(
"/api/delete/",
{ method: "POST", json: { j_ids: jIds } },
),
onSuccess: (result) => {
setSelected([]);
setLoadedIds([]);
setStatus(
`Deleted ${result.deleted.length} item(s)${
result.errors.length > 0 ? ` · ${result.errors.length} failed` : ""
}.`,
);
void queryClient.invalidateQueries({ queryKey: ["storage"] });
void queryClient.invalidateQueries({ queryKey: ["files"] });
void queryClient.invalidateQueries({ queryKey: ["duplicates-md5"] });
},
onError: (error) => setStatus(errorMessage(error)),
});
const tempMutation = useMutation({
mutationFn: () =>
api<{ removed: number }>("/api/temp/clear/", { method: "POST" }),
onSuccess: (result) => {
setStatus(`Removed ${result.removed} temp file(s).`);
void queryClient.invalidateQueries({ queryKey: ["storage"] });
},
onError: (error) => setStatus(errorMessage(error)),
});
function toggleSelected(jId: string) {
setSelected((current) =>
current.includes(jId)
? current.filter((value) => value !== jId)
: [...current, jId],
);
}
const storage = storageQuery.data;
return (
<div className="mx-auto flex w-full max-w-6xl flex-col gap-6">
<header>
<h1 className="text-lg font-semibold">Delete &amp; storage</h1>
<p className="mt-1 text-sm text-ctp-overlay0">
Inspect disk usage and remove files from the library.
</p>
</header>
{status ? (
<p className="rounded-md border border-ctp-surface1 bg-ctp-base px-3 py-2 text-xs text-ctp-subtext0">
{status}
</p>
) : null}
<section className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
{storageQuery.isPending ? (
<Spinner className="h-5 w-5" />
) : storage ? (
<>
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
Library size
</p>
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
{formatBytes(storage.watched_folder.library_size)}
</p>
<p className="mt-1 truncate font-mono text-[10px] text-ctp-overlay0" title={storage.watched_folder.path}>
{storage.watched_folder.path}
</p>
<p className="font-mono text-[10px] text-ctp-overlay0">
{storage.library_items} items
</p>
</div>
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
Disk
</p>
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
{formatBytes(storage.watched_folder.used)} /{" "}
{formatBytes(storage.watched_folder.total)}
</p>
<div className="mt-2 h-1.5 w-full overflow-hidden rounded-full bg-ctp-surface0">
<div
className={cn(
"h-full",
storage.watched_folder.percent_used >= 95
? "bg-ctp-red"
: storage.watched_folder.percent_used >= 80
? "bg-ctp-peach"
: "bg-ctp-blue",
)}
style={{ width: `${storage.watched_folder.percent_used}%` }}
/>
</div>
<p className="mt-1 font-mono text-[10px] text-ctp-overlay0">
{storage.watched_folder.percent_used}% used ·{" "}
{formatBytes(storage.watched_folder.free)} free
</p>
</div>
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
Media folder
</p>
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
{formatBytes(storage.media.size)}
</p>
<p className="mt-1 truncate font-mono text-[10px] text-ctp-overlay0" title={storage.media.path}>
{storage.media.path}
</p>
</div>
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
Temp uploads
</p>
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
{formatBytes(storage.temp.size)} · {storage.temp.files} file(s)
</p>
<Button
variant="secondary"
className="mt-2 px-2 py-1 text-xs"
disabled={tempMutation.isPending || storage.temp.files === 0}
onClick={() => tempMutation.mutate()}
>
{tempMutation.isPending ? "Clearing…" : "Clear temp folder"}
</Button>
</div>
</>
) : (
<EmptyState title="Could not load storage info" />
)}
</section>
<section className="flex flex-col gap-3">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Delete files by J-ID
</h2>
<textarea
className={cn(inputClass, "min-h-20 resize-y font-mono")}
placeholder="J-12, J-34 J-56&#10;comma, space or newline separated"
value={jIdsInput}
onChange={(event) => setJIdsInput(event.target.value)}
/>
<div className="flex items-center gap-3">
<Button
variant="secondary"
onClick={() => {
setStatus(null);
setSelected([]);
setLoadedIds(parseJIds(jIdsInput));
}}
>
Load files
</Button>
{selected.length > 0 ? (
<Button
variant="danger"
disabled={deleteMutation.isPending}
onClick={() => deleteMutation.mutate(selected)}
>
{deleteMutation.isPending
? "Deleting…"
: `Delete ${selected.length} selected`}
</Button>
) : null}
</div>
{loadedIds.length === 0 ? (
<p className="text-xs text-ctp-overlay0">
Enter one or more J-IDs to preview them here.
</p>
) : filesQuery.isPending ? (
<Spinner className="h-5 w-5" />
) : (filesQuery.data?.results.length ?? 0) === 0 ? (
<EmptyState title="No matching library items" />
) : (
<div className="grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
{filesQuery.data?.results.map((item) => {
const isSelected = selected.includes(item.j_id);
return (
<div
key={item.j_id}
className={cn(
"overflow-hidden rounded-lg border bg-ctp-base",
isSelected ? "border-ctp-mauve" : "border-ctp-surface0",
)}
>
<button
type="button"
onClick={() => toggleSelected(item.j_id)}
className="block w-full text-left"
>
<div className="relative aspect-square overflow-hidden bg-ctp-mantle">
<img
src={item.thumbnail_url}
alt={item.filename}
loading="lazy"
className="h-full w-full object-cover"
/>
<span
className={cn(
"absolute left-2 top-2 flex h-5 w-5 items-center justify-center rounded border text-[11px]",
isSelected
? "border-ctp-mauve bg-ctp-mauve text-ctp-crust"
: "border-ctp-surface1 bg-ctp-crust/70 text-transparent",
)}
>
✓
</span>
</div>
</button>
<div className="px-2 py-1.5">
<Link
to={`/detail/${item.j_id}`}
className="font-mono text-[10px] text-ctp-blue hover:underline"
>
{item.j_id}
</Link>
<p className="truncate font-mono text-[11px] text-ctp-subtext1" title={item.filename}>
{item.filename}
</p>
<p className="font-mono text-[10px] text-ctp-overlay0">
{formatBytes(item.size)}
</p>
</div>
</div>
);
})}
</div>
)}
</section>
</div>
);
}
@@ -0,0 +1,473 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { useMemo, useState } from "react";
import { Link } from "react-router-dom";
import { Button, Spinner, inputClass } from "@/components/ui";
import { api, errorMessage } from "@/lib/api";
import { cn } from "@/lib/cn";
import { formatBytes } from "@/lib/format";
import type {
DuplicateEntry,
VisualGroupsResponse,
VisualMatchesResponse,
} from "@/lib/types";
const ALGORITHMS = [
{ value: "ahash", label: "aHash" },
{ value: "dhash", label: "dHash" },
{ value: "phash", label: "pHash" },
{ value: "whash", label: "wHash" },
];
interface DeleteResult {
deleted: string[];
errors: { j_id?: string; location?: number; error: string }[];
}
function MemberCard({
entry,
selected,
onToggle,
}: {
entry: DuplicateEntry;
selected: boolean;
onToggle: () => void;
}) {
const preview =
entry.thumbnail_url ?? `/api/files/${entry.j_id}/thumbnail/`;
return (
<div
className={cn(
"overflow-hidden rounded-lg border bg-ctp-base",
selected ? "border-ctp-mauve" : "border-ctp-surface0",
)}
>
<button type="button" onClick={onToggle} className="block w-full text-left">
<div className="relative aspect-square overflow-hidden bg-ctp-mantle">
<img
src={preview}
alt={entry.j_id}
loading="lazy"
className="h-full w-full object-cover"
/>
<span
className={cn(
"absolute left-2 top-2 flex h-5 w-5 items-center justify-center rounded border text-[11px]",
selected
? "border-ctp-mauve bg-ctp-mauve text-ctp-crust"
: "border-ctp-surface1 bg-ctp-crust/70 text-transparent",
)}
>
✓
</span>
{entry.similarity !== undefined ? (
<span className="absolute right-2 top-2 rounded-sm bg-ctp-crust/80 px-1.5 py-0.5 font-mono text-[10px] text-ctp-peach">
{entry.similarity}%
</span>
) : null}
</div>
</button>
<div className="px-2 py-1.5">
<Link
to={`/detail/${entry.j_id}`}
className="font-mono text-[10px] text-ctp-blue hover:underline"
>
{entry.j_id}
</Link>
<p
className="truncate font-mono text-[11px] text-ctp-subtext1"
title={entry.filename}
>
{entry.filename}
</p>
<p className="font-mono text-[10px] text-ctp-overlay0">
{formatBytes(entry.size)}
{entry.location_count > 1 ? ` · ${entry.location_count} copies` : ""}
</p>
</div>
</div>
);
}
export default function DuplicatesPage() {
const queryClient = useQueryClient();
const [threshold, setThreshold] = useState(80);
const [algorithms, setAlgorithms] = useState<string[]>(
ALGORITHMS.map((algorithm) => algorithm.value),
);
const [groupsPage, setGroupsPage] = useState(1);
const [groupsEnabled, setGroupsEnabled] = useState(false);
const [selected, setSelected] = useState<string[]>([]);
const [dismissed, setDismissed] = useState<string[]>([]);
const [similarTarget, setSimilarTarget] = useState("");
const [status, setStatus] = useState<string | null>(null);
const exactQuery = useQuery({
queryKey: ["duplicates-md5"],
queryFn: () =>
api<{ count: number; groups: DuplicateEntry[] }>("/api/duplicates/md5/"),
});
const groupsQuery = useQuery({
queryKey: [
"duplicate-groups",
groupsEnabled,
threshold,
algorithms,
groupsPage,
],
queryFn: () =>
api<VisualGroupsResponse>("/api/duplicates/visual_groups/", {
method: "POST",
json: { threshold, algorithms, page: groupsPage },
}),
enabled: groupsEnabled && algorithms.length > 0,
});
const similarMutation = useMutation({
mutationFn: (jId: string) =>
api<VisualMatchesResponse>("/api/duplicates/visual/", {
method: "POST",
json: { j_id: jId, threshold, algorithms },
}),
onError: (error) => setStatus(errorMessage(error)),
});
const deleteMutation = useMutation({
mutationFn: (payload: { j_ids?: string[]; location_ids?: number[] }) =>
api<DeleteResult>("/api/delete/", { method: "POST", json: payload }),
onSuccess: (result) => {
setSelected([]);
setStatus(
`Deleted ${result.deleted.length} entr${
result.deleted.length === 1 ? "y" : "ies"
}${result.errors.length > 0 ? ` · ${result.errors.length} failed` : ""}.`,
);
void queryClient.invalidateQueries({ queryKey: ["duplicates-md5"] });
void queryClient.invalidateQueries({ queryKey: ["duplicate-groups"] });
void queryClient.invalidateQueries({ queryKey: ["files"] });
},
onError: (error) => setStatus(errorMessage(error)),
});
function toggleSelected(jId: string) {
setSelected((current) =>
current.includes(jId)
? current.filter((value) => value !== jId)
: [...current, jId],
);
}
function toggleAlgorithm(value: string) {
setGroupsPage(1);
setAlgorithms((current) =>
current.includes(value)
? current.filter((algorithm) => algorithm !== value)
: [...current, value],
);
}
const groups = useMemo(
() =>
(groupsQuery.data?.groups ?? []).filter(
(group) =>
!dismissed.includes(
group.members
.map((member) => member.j_id)
.sort()
.join("|"),
),
),
[groupsQuery.data, dismissed],
);
return (
<div className="mx-auto flex w-full max-w-6xl flex-col gap-8">
<header>
<h1 className="text-lg font-semibold">Duplicates</h1>
<p className="mt-1 text-sm text-ctp-overlay0">
Clean up exact copies and visually similar files.
</p>
</header>
{status ? (
<p className="rounded-md border border-ctp-surface1 bg-ctp-base px-3 py-2 text-xs text-ctp-subtext0">
{status}
</p>
) : null}
<section className="flex flex-col gap-3">
<header className="flex items-center justify-between gap-2">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Exact duplicates
</h2>
<span className="font-mono text-[11px] text-ctp-overlay0">
{exactQuery.data?.count ?? 0} item(s)
</span>
</header>
{exactQuery.isPending ? (
<Spinner className="h-5 w-5" />
) : (exactQuery.data?.count ?? 0) === 0 ? (
<p className="rounded-lg border border-dashed border-ctp-surface1 px-3 py-6 text-center text-xs text-ctp-overlay0">
No items exist at multiple paths.
</p>
) : (
<div className="flex flex-col gap-3">
{exactQuery.data?.groups.map((group) => (
<div
key={group.j_id}
className="flex flex-col gap-3 rounded-lg border border-ctp-surface0 bg-ctp-base p-3 lg:flex-row"
>
<div className="w-32 shrink-0">
<div className="aspect-square overflow-hidden rounded-md bg-ctp-mantle">
<img
src={
group.thumbnail_url ??
`/api/files/${group.j_id}/thumbnail/`
}
alt={group.j_id}
loading="lazy"
className="h-full w-full object-cover"
/>
</div>
<Link
to={`/detail/${group.j_id}`}
className="font-mono text-[10px] text-ctp-blue hover:underline"
>
{group.j_id}
</Link>
</div>
<div className="min-w-0 flex-1">
<p className="truncate font-mono text-xs text-ctp-subtext1">
{group.filename}
</p>
<ul className="mt-2 flex flex-col gap-1.5">
{(group.locations ?? []).map((location) => (
<li
key={location.id}
className="flex items-center justify-between gap-3 rounded-md border border-ctp-surface0 px-2 py-1"
>
<span className="truncate font-mono text-[11px] text-ctp-overlay1" title={location.rel_path}>
{location.rel_path}
</span>
<button
type="button"
onClick={() =>
deleteMutation.mutate({ location_ids: [location.id] })
}
className="shrink-0 rounded-md px-1.5 py-0.5 font-mono text-[10px] text-ctp-red transition hover:bg-ctp-red/15"
>
delete copy
</button>
</li>
))}
</ul>
</div>
<div className="shrink-0">
<Button
variant="danger"
className="px-2 py-1 text-xs"
onClick={() => deleteMutation.mutate({ j_ids: [group.j_id] })}
>
Delete item
</Button>
</div>
</div>
))}
</div>
)}
</section>
<section className="flex flex-col gap-3">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Visual similarity
</h2>
<div className="flex flex-col gap-3 rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
<div className="flex flex-wrap items-center gap-4">
<label className="flex flex-1 items-center gap-3 text-xs text-ctp-subtext0">
Similarity {threshold}%
<input
type="range"
min={50}
max={100}
value={threshold}
onChange={(event) => {
setGroupsPage(1);
setThreshold(Number(event.target.value));
}}
className="flex-1"
/>
</label>
<div className="flex flex-wrap items-center gap-2">
{ALGORITHMS.map((algorithm) => (
<label
key={algorithm.value}
className={cn(
"cursor-pointer rounded-full border px-2.5 py-1 font-mono text-[11px] transition",
algorithms.includes(algorithm.value)
? "border-ctp-mauve/40 bg-ctp-mauve/15 text-ctp-mauve"
: "border-ctp-surface1 text-ctp-subtext0 hover:border-ctp-surface2",
)}
>
<input
type="checkbox"
className="hidden"
checked={algorithms.includes(algorithm.value)}
onChange={() => toggleAlgorithm(algorithm.value)}
/>
{algorithm.label}
</label>
))}
</div>
<Button
onClick={() => {
setGroupsEnabled(true);
void groupsQuery.refetch();
}}
disabled={algorithms.length === 0 || groupsQuery.isFetching}
>
{groupsQuery.isFetching ? "Comparing…" : "Find groups"}
</Button>
</div>
<div className="flex flex-wrap items-end gap-2 border-t border-ctp-surface0 pt-3">
<label className="flex flex-col gap-1 text-xs text-ctp-subtext0">
Find files similar to
<input
className={cn(inputClass, "w-32 font-mono")}
placeholder="J-123"
value={similarTarget}
onChange={(event) => setSimilarTarget(event.target.value)}
/>
</label>
<Button
variant="secondary"
onClick={() => similarMutation.mutate(similarTarget.trim())}
disabled={!similarTarget.trim() || similarMutation.isPending}
>
{similarMutation.isPending ? "Searching…" : "Search"}
</Button>
{similarMutation.isError ? (
<p className="text-xs text-ctp-red">
{errorMessage(similarMutation.error)}
</p>
) : null}
</div>
{similarMutation.data ? (
similarMutation.data.count === 0 ? (
<p className="text-xs text-ctp-overlay0">
No matches at {similarMutation.data.threshold}% similarity.
</p>
) : (
<div className="grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
{similarMutation.data.matches.map((entry) => (
<MemberCard
key={entry.j_id}
entry={entry}
selected={selected.includes(entry.j_id)}
onToggle={() => toggleSelected(entry.j_id)}
/>
))}
</div>
)
) : null}
</div>
{groupsEnabled ? (
groupsQuery.isPending ? (
<Spinner className="h-5 w-5" />
) : groups.length === 0 ? (
<p className="rounded-lg border border-dashed border-ctp-surface1 px-3 py-6 text-center text-xs text-ctp-overlay0">
No groups at {threshold}% similarity.
</p>
) : (
<div className="flex flex-col gap-4">
{groups.map((group) => {
const groupKey = group.members
.map((member) => member.j_id)
.sort()
.join("|");
return (
<div
key={groupKey}
className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3"
>
<header className="flex items-center justify-between gap-2">
<span className="font-mono text-[11px] text-ctp-overlay0">
{group.size} similar items
</span>
<button
type="button"
onClick={() => setDismissed((current) => [...current, groupKey])}
className="rounded-md px-1.5 py-0.5 font-mono text-[10px] text-ctp-overlay0 transition hover:bg-ctp-surface0 hover:text-ctp-text"
>
dismiss
</button>
</header>
<div className="mt-3 grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
{group.members.map((member) => (
<MemberCard
key={member.j_id}
entry={member}
selected={selected.includes(member.j_id)}
onToggle={() => toggleSelected(member.j_id)}
/>
))}
</div>
</div>
);
})}
<div className="flex items-center justify-between gap-3">
<Button
variant="secondary"
disabled={groupsPage <= 1 || groupsQuery.isFetching}
onClick={() => setGroupsPage((page) => Math.max(1, page - 1))}
>
Previous
</Button>
<span className="font-mono text-xs text-ctp-overlay0">
Page {groupsQuery.data?.page ?? groupsPage} ·{" "}
{groupsQuery.data?.count ?? 0} group(s)
</span>
<Button
variant="secondary"
disabled={!groupsQuery.data?.has_next || groupsQuery.isFetching}
onClick={() => setGroupsPage((page) => page + 1)}
>
Next
</Button>
</div>
</div>
)
) : null}
{selected.length > 0 ? (
<div className="sticky bottom-16 z-20 flex items-center justify-between gap-3 rounded-lg border border-ctp-mauve/40 bg-ctp-surface0 px-3 py-2 shadow-[0_4px_12px_rgba(17,17,27,0.6)]">
<span className="text-xs text-ctp-subtext1">
{selected.length} selected
</span>
<div className="flex items-center gap-2">
<Button
variant="ghost"
className="px-2 py-1 text-xs"
onClick={() => setSelected([])}
>
Clear
</Button>
<Button
variant="danger"
className="px-2 py-1 text-xs"
disabled={deleteMutation.isPending}
onClick={() => deleteMutation.mutate({ j_ids: selected })}
>
{deleteMutation.isPending ? "Deleting…" : "Delete selected"}
</Button>
</div>
</div>
) : null}
</section>
</div>
);
}
+282
View File
@@ -0,0 +1,282 @@
import { useMutation, useQueryClient } from "@tanstack/react-query";
import { ScanSearch } from "lucide-react";
import { useState } from "react";
import { Link } from "react-router-dom";
import { Button, Spinner } from "@/components/ui";
import { api, errorMessage } from "@/lib/api";
import { cn } from "@/lib/cn";
import {
e621ErrorMessage,
effectiveCredentials,
iqdbSearch,
RATING_LABELS,
type E621IqdbResult,
} from "@/lib/e621";
import type { MediaItem } from "@/lib/types";
import { useE621 } from "@/store/e621";
const ratingPill: Record<string, string> = {
s: "bg-ctp-green text-ctp-crust",
q: "bg-ctp-peach text-ctp-crust",
e: "bg-ctp-red text-ctp-crust",
};
type Notice = { kind: "ok" | "warn" | "error"; text: string } | null;
export function IqdbCard({
item,
canLink,
}: {
item: MediaItem;
canLink: boolean;
}) {
const queryClient = useQueryClient();
const credentials = useE621((state) => state.credentials);
const [busy, setBusy] = useState(false);
const [candidates, setCandidates] = useState<E621IqdbResult[] | null>(null);
const [selected, setSelected] = useState<E621IqdbResult | null>(null);
const [notice, setNotice] = useState<Notice>(null);
const isImage = item.kind === "image";
const configured = Boolean(credentials?.configured);
async function search() {
setBusy(true);
setNotice(null);
setSelected(null);
setCandidates(null);
try {
const response = await fetch(item.raw_url);
if (!response.ok) {
throw new Error(`Could not load the file (HTTP ${response.status}).`);
}
const blob = await response.blob();
const results = await iqdbSearch(
effectiveCredentials(credentials),
blob,
item.filename,
);
setCandidates(results);
if (results.length === 0) {
setNotice({ kind: "warn", text: "No similar posts found on e621." });
}
} catch (error) {
setNotice({ kind: "error", text: e621ErrorMessage(error) });
} finally {
setBusy(false);
}
}
const linkMutation = useMutation({
mutationFn: (postId: number) =>
api<MediaItem & { md5_match?: boolean }>(
`/api/files/${item.j_id}/match/`,
{ method: "POST", json: { post_id: postId } },
),
onSuccess: (data) => {
void queryClient.invalidateQueries({ queryKey: ["file", item.j_id] });
void queryClient.invalidateQueries({ queryKey: ["files"] });
void queryClient.invalidateQueries({ queryKey: ["tag-cloud"] });
setSelected(null);
setNotice(
data.md5_match
? {
kind: "ok",
text: `Linked to post #${data.e621_post_id} — MD5 matches.`,
}
: {
kind: "warn",
text: `Linked to post #${data.e621_post_id}, but its MD5 differs from this file.`,
},
);
},
onError: (error) =>
setNotice({ kind: "error", text: errorMessage(error) }),
});
return (
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
<div className="flex items-center justify-between gap-2">
<h2 className="text-sm font-semibold text-ctp-subtext1">IQDB search</h2>
{candidates && candidates.length > 0 ? (
<span className="font-mono text-[11px] text-ctp-overlay0">
{candidates.length} candidates
</span>
) : null}
</div>
<p className="mt-2 text-xs text-ctp-overlay0">
Reverse-search e621 with this file to find its original post.
</p>
<Button
variant="secondary"
className="mt-3 w-full"
disabled={busy || !isImage || !configured}
onClick={() => void search()}
>
{busy ? (
<Spinner className="h-3.5 w-3.5" />
) : (
<ScanSearch className="h-4 w-4" />
)}
{busy ? "Searching…" : "Search e621"}
</Button>
{!configured ? (
<p className="mt-2 text-[11px] text-ctp-overlay0">
Configure e621 credentials in Account to use IQDB.
</p>
) : !isImage ? (
<p className="mt-2 text-[11px] text-ctp-overlay0">
IQDB works on images only.
</p>
) : null}
{notice ? (
<p
className={cn(
"mt-2 text-xs",
notice.kind === "ok"
? "text-ctp-green"
: notice.kind === "warn"
? "text-ctp-yellow"
: "text-ctp-red",
)}
>
{notice.text}
</p>
) : null}
{candidates && candidates.length > 0 ? (
<div className="mt-3 grid grid-cols-3 gap-1.5">
{candidates.map((candidate) => {
const exact = candidate.md5 !== null && candidate.md5 === item.md5;
return (
<button
key={`${candidate.post_id}-${candidate.md5}`}
type="button"
disabled={linkMutation.isPending || candidate.post_id === null}
onClick={() => setSelected(candidate)}
className={cn(
"overflow-hidden rounded-md border bg-ctp-mantle text-left transition",
selected?.post_id === candidate.post_id
? "border-ctp-mauve ring-2 ring-ctp-mauve/40"
: "border-ctp-surface0 hover:border-ctp-surface2",
)}
>
<span className="relative block aspect-square overflow-hidden">
{candidate.preview_url ? (
<img
src={candidate.preview_url}
alt={`Post #${candidate.post_id}`}
loading="lazy"
className="h-full w-full object-cover"
/>
) : (
<span className="flex h-full w-full items-center justify-center text-[10px] text-ctp-overlay0">
No preview
</span>
)}
{candidate.rating ? (
<span
className={cn(
"absolute left-1 top-1 rounded-full px-1 py-0.5 font-mono text-[9px] font-semibold uppercase",
ratingPill[candidate.rating] ??
"bg-ctp-surface1 text-ctp-text",
)}
>
{candidate.rating}
</span>
) : null}
{exact ? (
<span className="absolute right-1 top-1 rounded-full bg-ctp-green/90 px-1 py-0.5 font-mono text-[9px] font-semibold uppercase text-ctp-crust">
MD5
</span>
) : null}
</span>
<span className="block px-1 py-0.5 font-mono text-[10px] text-ctp-overlay0">
#{candidate.post_id}
{candidate.score !== null
? ` · ${Math.round(candidate.score)}%`
: ""}
</span>
</button>
);
})}
</div>
) : null}
{selected ? (
<div className="mt-3 rounded-md border border-ctp-surface0 bg-ctp-surface0/40 p-3">
<div className="flex items-center justify-between gap-2">
<p className="font-mono text-xs text-ctp-subtext1">
Post #{selected.post_id}
</p>
{selected.md5 === item.md5 ? (
<span className="rounded-full bg-ctp-green/15 px-1.5 py-0.5 font-mono text-[9px] font-semibold uppercase text-ctp-green">
MD5 match
</span>
) : null}
</div>
<p className="mt-1 text-xs text-ctp-overlay0">
{selected.rating
? (RATING_LABELS[selected.rating] ?? selected.rating)
: "Unknown rating"}
{selected.score_total !== null
? ` · ▲ ${selected.score_total}`
: ""}
{selected.fav_count !== null ? ` · ${selected.fav_count} favs` : ""}
{selected.width && selected.height
? ` · ${selected.width}×${selected.height}`
: ""}
</p>
{selected.tags_preview.length > 0 ? (
<div className="mt-2 flex flex-wrap gap-1">
{selected.tags_preview.map((tag) => (
<span
key={tag}
className="rounded border border-ctp-surface1 px-1 py-0.5 font-mono text-[10px] text-ctp-subtext0"
>
{tag}
</span>
))}
</div>
) : null}
<div className="mt-3 flex items-center gap-3">
<Button
disabled={
!canLink || linkMutation.isPending || selected.post_id === null
}
onClick={() =>
selected.post_id !== null &&
linkMutation.mutate(selected.post_id)
}
>
{linkMutation.isPending ? "Linking…" : "Link selected post"}
</Button>
<Link
to={`/detail/${selected.post_id}`}
target="_blank"
className="text-xs text-ctp-blue hover:underline"
>
View post
</Link>
</div>
<p className="mt-2 text-[11px] text-ctp-overlay0">
{canLink
? "Linking stores this post's metadata on the item."
: "Only the uploader or staff can link posts."}
</p>
</div>
) : null}
{candidates && candidates.length > 0 && !selected ? (
<p className="mt-2 text-[11px] text-ctp-overlay0">
Select a candidate first, then confirm the link.
</p>
) : null}
</section>
);
}
+270 -7
View File
@@ -1,18 +1,24 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { ArrowLeft, Download } from "lucide-react";
import { ArrowLeft, Download, Trash2 } from "lucide-react";
import { useEffect, useState } from "react";
import { Link, useNavigate } from "react-router-dom";
import { Button, EmptyState, Spinner, inputClass, linkButtonClass } from "@/components/ui";
import { TagChip } from "@/components/TagChip";
import { DetailShell } from "@/features/detail/DetailShell";
import { IqdbCard } from "@/features/library/IqdbCard";
import { api, errorMessage } from "@/lib/api";
import { cn } from "@/lib/cn";
import { isTypingTarget } from "@/lib/dom";
import { renderDText } from "@/lib/dtext";
import { RATING_LABELS, TAG_CATEGORY_LABELS } from "@/lib/e621";
import { formatBytes, formatDate } from "@/lib/format";
import type { FileNeighbors, MediaItem, Rating } from "@/lib/types";
import type {
E621MatchStatus,
FileNeighbors,
MediaItem,
Rating,
} from "@/lib/types";
import { useAuth } from "@/store/auth";
import { useUi } from "@/store/ui";
@@ -45,6 +51,188 @@ const ratingLabel: Record<string, string> = {
e: "Explicit",
};
const matchStatusMeta: Record<
E621MatchStatus,
{ label: string; className: string }
> = {
unknown: {
label: "Not checked",
className: "border-ctp-surface2 bg-ctp-surface0 text-ctp-subtext1",
},
matched: {
label: "Matched",
className: "border-ctp-green/40 bg-ctp-green/15 text-ctp-green",
},
not_found: {
label: "Not found on e621",
className: "border-ctp-yellow/40 bg-ctp-yellow/15 text-ctp-yellow",
},
deleted: {
label: "Deleted on e621",
className: "border-ctp-red/40 bg-ctp-red/15 text-ctp-red",
},
};
function MatchCard({ item, canEdit }: { item: MediaItem; canEdit: boolean }) {
const queryClient = useQueryClient();
const [postId, setPostId] = useState("");
const [notice, setNotice] = useState<{
kind: "ok" | "warn" | "error";
text: string;
} | null>(null);
function refresh() {
void queryClient.invalidateQueries({ queryKey: ["file", item.j_id] });
void queryClient.invalidateQueries({ queryKey: ["files"] });
void queryClient.invalidateQueries({ queryKey: ["tag-cloud"] });
}
const checkMutation = useMutation({
mutationFn: () =>
api<MediaItem>(`/api/files/${item.j_id}/match/`, {
method: "POST",
json: {},
}),
onSuccess: (data) => {
refresh();
if (data.e621_match_status === "matched") {
setNotice({ kind: "ok", text: `Matched to post #${data.e621_post_id}.` });
} else if (data.e621_match_status === "deleted") {
setNotice({
kind: "error",
text: "The linked post no longer exists on e621.",
});
} else {
setNotice({
kind: "warn",
text: "No e621 post with this file's MD5 was found.",
});
}
},
onError: (error) =>
setNotice({ kind: "error", text: errorMessage(error) }),
});
const linkMutation = useMutation({
mutationFn: (id: number) =>
api<MediaItem & { md5_match?: boolean }>(`/api/files/${item.j_id}/match/`, {
method: "POST",
json: { post_id: id },
}),
onSuccess: (data) => {
refresh();
setPostId("");
setNotice(
data.md5_match
? {
kind: "ok",
text: `Linked to post #${data.e621_post_id} — MD5 matches.`,
}
: {
kind: "warn",
text: `Linked to post #${data.e621_post_id}, but its MD5 differs from this file.`,
},
);
},
onError: (error) =>
setNotice({ kind: "error", text: errorMessage(error) }),
});
const unlinkMutation = useMutation({
mutationFn: () =>
api<MediaItem>(`/api/files/${item.j_id}/unlink/`, { method: "POST" }),
onSuccess: () => {
refresh();
setNotice({ kind: "ok", text: "e621 link removed." });
},
onError: (error) =>
setNotice({ kind: "error", text: errorMessage(error) }),
});
const meta = matchStatusMeta[item.e621_match_status];
return (
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
<div className="flex items-center justify-between gap-2">
<h2 className="text-sm font-semibold text-ctp-subtext1">e621 match</h2>
<span
className={cn(
"rounded-full border px-2 py-0.5 text-[11px] font-medium",
meta.className,
)}
>
{meta.label}
</span>
</div>
<p className="mt-2 text-xs text-ctp-overlay0">
{item.e621_checked_at
? `Last checked ${formatDate(item.e621_checked_at)}`
: "This item has not been checked against e621 yet."}
</p>
{canEdit ? (
<div className="mt-3 flex flex-col gap-2">
<div className="flex gap-2">
<Button
variant="secondary"
disabled={checkMutation.isPending}
onClick={() => checkMutation.mutate()}
>
{checkMutation.isPending
? "Checking…"
: item.e621_post_id
? "Recheck"
: "Check MD5"}
</Button>
{item.e621_post_id ? (
<Button
variant="ghost"
disabled={unlinkMutation.isPending}
onClick={() => unlinkMutation.mutate()}
>
Unlink
</Button>
) : null}
</div>
<div className="flex gap-2">
<input
className={inputClass}
placeholder="Post ID"
inputMode="numeric"
value={postId}
onChange={(event) =>
setPostId(event.target.value.replace(/[^0-9]/g, ""))
}
/>
<Button
disabled={!postId || linkMutation.isPending}
onClick={() => linkMutation.mutate(Number(postId))}
>
Link
</Button>
</div>
</div>
) : null}
{notice ? (
<p
className={cn(
"mt-2 text-xs",
notice.kind === "ok"
? "text-ctp-green"
: notice.kind === "warn"
? "text-ctp-yellow"
: "text-ctp-red",
)}
>
{notice.text}
</p>
) : null}
</section>
);
}
function E621MetadataCard({ item }: { item: MediaItem }) {
const data = item.e621_data;
if (!item.e621_post_id || !data) return null;
@@ -77,6 +265,13 @@ function E621MetadataCard({ item }: { item: MediaItem }) {
</span>
</div>
{item.e621_match_status === "deleted" ? (
<p className="mt-3 rounded border border-ctp-red/40 bg-ctp-red/10 px-2.5 py-1.5 text-xs text-ctp-red">
This post has been deleted on e621 — the metadata below is a cached
copy.
</p>
) : null}
<dl className="mt-3 grid grid-cols-2 gap-x-4 gap-y-2 text-xs sm:grid-cols-4">
<div>
<dt className="text-ctp-overlay0">Rating</dt>
@@ -112,7 +307,7 @@ function E621MetadataCard({ item }: { item: MediaItem }) {
</p>
<div className="flex flex-wrap gap-1.5">
{tags.map((tag) => (
<TagChip key={tag} tag={tag} category={category} />
<TagChip key={tag} tag={tag} category={category} followable />
))}
</div>
</div>
@@ -296,15 +491,21 @@ function CustomDataCard({ item, canEdit }: { item: MediaItem; canEdit: boolean }
export function LibraryDetail({ jId }: { jId: string }) {
const navigate = useNavigate();
const queryClient = useQueryClient();
const ordering = useUi((state) => state.ordering);
const user = useAuth((state) => state.user);
const rawUrl = `/api/files/${jId}/raw/`;
const [confirmDelete, setConfirmDelete] = useState(false);
const query = useQuery({
queryKey: ["file", jId],
queryFn: () => api<MediaItem>(`/api/files/${jId}/`),
});
const rawUrl = query.data?.raw_url ?? `/api/files/${jId}/raw/`;
const downloadUrl = rawUrl.includes("?")
? `${rawUrl}&download=1`
: `${rawUrl}?download=1`;
const neighborsQuery = useQuery({
queryKey: ["neighbors", jId, ordering],
queryFn: () =>
@@ -313,13 +514,27 @@ export function LibraryDetail({ jId }: { jId: string }) {
),
});
const deleteMutation = useMutation({
mutationFn: () =>
api<{ deleted: string[]; errors: { j_id?: string; error: string }[] }>(
"/api/delete/",
{ method: "POST", json: { j_ids: [jId] } },
),
onSuccess: () => {
void queryClient.invalidateQueries({ queryKey: ["files"] });
void queryClient.invalidateQueries({ queryKey: ["duplicates-md5"] });
void queryClient.invalidateQueries({ queryKey: ["storage"] });
navigate("/");
},
});
useEffect(() => {
function handleKeyDown(event: KeyboardEvent) {
if (event.ctrlKey || event.metaKey || event.altKey) return;
if (isTypingTarget(event.target)) return;
if (event.key === "d" || event.key === "D") {
event.preventDefault();
window.location.href = `${rawUrl}?download=1`;
window.location.href = downloadUrl;
return;
}
const neighbors = neighborsQuery.data;
@@ -331,7 +546,7 @@ export function LibraryDetail({ jId }: { jId: string }) {
}
window.addEventListener("keydown", handleKeyDown);
return () => window.removeEventListener("keydown", handleKeyDown);
}, [neighborsQuery.data, navigate, rawUrl]);
}, [neighborsQuery.data, navigate, downloadUrl]);
if (query.isPending) {
return (
@@ -361,6 +576,12 @@ export function LibraryDetail({ jId }: { jId: string }) {
user.is_staff ||
user.role === "staff"),
);
const canDelete = Boolean(
user &&
(user.is_superuser ||
user.role === "staff" ||
user.id === item.uploaded_by_id),
);
return (
<DetailShell
@@ -378,9 +599,45 @@ export function LibraryDetail({ jId }: { jId: string }) {
}
actions={
<div className="flex flex-wrap items-center gap-2">
<a href={`${rawUrl}?download=1`} className={linkButtonClass}>
<a href={downloadUrl} className={linkButtonClass}>
<Download className="h-4 w-4" /> Download
</a>
{canDelete ? (
confirmDelete ? (
<>
<span className="text-xs text-ctp-subtext0">
Delete this item and all its copies?
</span>
<Button
variant="danger"
disabled={deleteMutation.isPending}
onClick={() => deleteMutation.mutate()}
>
{deleteMutation.isPending ? "Deleting…" : "Confirm delete"}
</Button>
<Button
variant="ghost"
onClick={() => setConfirmDelete(false)}
disabled={deleteMutation.isPending}
>
Cancel
</Button>
</>
) : (
<Button
variant="danger"
onClick={() => setConfirmDelete(true)}
title="Delete this item from the library"
>
<Trash2 className="h-4 w-4" /> Delete
</Button>
)
) : null}
{deleteMutation.isError ? (
<p className="text-sm text-ctp-red">
{errorMessage(deleteMutation.error)}
</p>
) : null}
</div>
}
media={
@@ -439,6 +696,12 @@ export function LibraryDetail({ jId }: { jId: string }) {
</p>
</section>
<MatchCard key={item.j_id} item={item} canEdit={canEdit} />
{user ? (
<IqdbCard key={item.j_id} item={item} canLink={canEdit} />
) : null}
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Locations ({item.locations.length})
+136 -8
View File
@@ -5,6 +5,7 @@ import { useState, useEffect, type FormEvent } from "react";
import { CollapsibleSidebar } from "@/components/CollapsibleSidebar";
import { MediaCard } from "@/components/MediaCard";
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
import { MatchPanel } from "@/features/library/MatchPanel";
import { api } from "@/lib/api";
import { cn } from "@/lib/cn";
import { isTypingTarget } from "@/lib/dom";
@@ -41,13 +42,44 @@ const ratingFilters = [
},
];
const statusFilters = [
{
value: "matched",
label: "Matched",
active: "border-ctp-green/40 bg-ctp-green/15 text-ctp-green",
},
{
value: "not_found",
label: "Not found",
active: "border-ctp-yellow/40 bg-ctp-yellow/15 text-ctp-yellow",
},
{
value: "deleted",
label: "Deleted",
active: "border-ctp-red/40 bg-ctp-red/15 text-ctp-red",
},
{
value: "custom",
label: "Custom",
active: "border-ctp-mauve/40 bg-ctp-mauve/15 text-ctp-mauve",
},
{
value: "unknown",
label: "Unknown",
active: "border-ctp-surface2 bg-ctp-surface0 text-ctp-subtext1",
},
];
export default function LibraryPage() {
const [searchInput, setSearchInput] = useState("");
const [search, setSearch] = useState("");
const [searchType, setSearchType] = useState("filename");
const [page, setPage] = useState(1);
const ratings = useUi((state) => state.ratings);
const setRatings = useUi((state) => state.setRatings);
const statuses = useUi((state) => state.statuses);
const setStatuses = useUi((state) => state.setStatuses);
const perPage = useUi((state) => state.perPage);
const setPerPage = useUi((state) => state.setPerPage);
const ordering = useUi((state) => state.ordering);
@@ -68,19 +100,35 @@ export default function LibraryPage() {
}, []);
const query = useQuery({
queryKey: ["files", { search, ratings, ordering, perPage, page }],
queryKey: [
"files",
{ search, searchType, ratings, statuses, ordering, perPage, page },
],
queryFn: () => {
const params = new URLSearchParams({
page: String(page),
per_page: String(perPage),
ordering,
});
if (search) params.set("search", search);
if (search) {
params.set("search", search);
params.set("search_type", searchType);
}
if (ratings.length) params.set("rating", ratings.join(","));
if (statuses.length) params.set("status", statuses.join(","));
return api<Paginated<MediaItem>>(`/api/files/?${params.toString()}`);
},
});
const tagCloudQuery = useQuery({
queryKey: ["tag-cloud"],
queryFn: () =>
api<{ count: number; tags: { tag: string; count: number }[] }>(
"/api/tags/cloud/",
),
staleTime: 60_000,
});
const pageCount = query.data
? Math.max(1, Math.ceil(query.data.count / perPage))
: 1;
@@ -100,10 +148,28 @@ export default function LibraryPage() {
);
}
function toggleStatus(value: string) {
setPage(1);
setStatuses(
statuses.includes(value)
? statuses.filter((status) => status !== value)
: [...statuses, value],
);
}
function searchTag(tag: string) {
setSearchInput(tag);
setSearch(tag);
setSearchType("tags");
setPage(1);
}
function clearFilters() {
setSearchInput("");
setSearch("");
setSearchType("filename");
setRatings([]);
setStatuses([]);
setPage(1);
}
@@ -118,11 +184,22 @@ export default function LibraryPage() {
<input
id="library-search"
className={cn(inputClass, "pl-8")}
placeholder="Filename…"
placeholder={
searchType === "tags" ? "Tags…" : "Filename…"
}
value={searchInput}
onChange={(event) => setSearchInput(event.target.value)}
/>
</div>
<select
className={inputClass}
value={searchType}
onChange={(event) => setSearchType(event.target.value)}
>
<option value="filename">Filename</option>
<option value="tags">Tags</option>
<option value="both">Filename or tags</option>
</select>
<div className="flex gap-2">
<Button type="submit" className="flex-1">
Search
@@ -133,6 +210,29 @@ export default function LibraryPage() {
</div>
</form>
<div className="flex flex-col gap-2">
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Status
</span>
<div className="flex flex-wrap gap-1.5">
{statusFilters.map((option) => (
<button
key={option.value}
type="button"
onClick={() => toggleStatus(option.value)}
className={cn(
"rounded-full border px-2.5 py-1 text-xs font-medium transition",
statuses.includes(option.value)
? option.active
: "border-ctp-surface1 text-ctp-subtext0 hover:border-ctp-surface2 hover:text-ctp-text",
)}
>
{option.label}
</button>
))}
</div>
</div>
<div className="flex flex-col gap-2">
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Rating
@@ -156,6 +256,31 @@ export default function LibraryPage() {
</div>
</div>
<div className="flex flex-col gap-2">
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Tag cloud
</span>
<div className="flex flex-wrap gap-1.5">
{(tagCloudQuery.data?.tags ?? []).map((entry) => (
<button
key={entry.tag}
type="button"
onClick={() => searchTag(entry.tag)}
title={`Search ${entry.tag}`}
className="rounded border border-ctp-blue/30 bg-ctp-blue/10 px-1.5 py-0.5 font-mono text-[11px] text-ctp-blue transition hover:bg-ctp-blue/20"
>
{entry.tag} ({entry.count})
</button>
))}
{tagCloudQuery.isPending ? (
<Spinner className="h-3.5 w-3.5" />
) : null}
{tagCloudQuery.data && tagCloudQuery.data.tags.length === 0 ? (
<p className="text-xs text-ctp-overlay0">No tags yet.</p>
) : null}
</div>
</div>
<label className="flex flex-col gap-2">
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Sort
@@ -209,11 +334,14 @@ export default function LibraryPage() {
? `${query.data.count} item${query.data.count === 1 ? "" : "s"}`
: "…"}
</p>
{pageCount > 1 ? (
<p className="font-mono text-xs text-ctp-overlay0">
Page {page} / {pageCount}
</p>
) : null}
<div className="flex items-center gap-3">
{pageCount > 1 ? (
<p className="font-mono text-xs text-ctp-overlay0">
Page {page} / {pageCount}
</p>
) : null}
{user?.can_upload ? <MatchPanel /> : null}
</div>
</div>
{!user ? (
@@ -0,0 +1,158 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { RefreshCw, X } from "lucide-react";
import { useEffect, useState } from "react";
import { Button, Spinner } from "@/components/ui";
import { api, errorMessage } from "@/lib/api";
import type { MatchTask } from "@/lib/types";
type Scope = "missing" | "all";
function summary(task: MatchTask): string {
const parts = [
`${task.matched} matched`,
`${task.not_found} not found`,
`${task.deleted} deleted`,
];
if (task.failed > 0) parts.push(`${task.failed} failed`);
return parts.join(" · ");
}
export function MatchPanel() {
const queryClient = useQueryClient();
const [taskId, setTaskId] = useState<string | null>(null);
const [menuOpen, setMenuOpen] = useState(false);
const taskQuery = useQuery({
queryKey: ["match-task", taskId],
queryFn: () => api<MatchTask>(`/api/matches/${taskId}/`),
enabled: Boolean(taskId),
refetchInterval: (query) => {
const status = query.state.data?.status;
return status === "pending" || status === "running" ? 1000 : false;
},
});
const task = taskQuery.data;
const running = task?.status === "pending" || task?.status === "running";
const finishedTaskId = task && !running ? task.task_id : null;
useEffect(() => {
if (!finishedTaskId) return;
void queryClient.invalidateQueries({ queryKey: ["files"] });
void queryClient.invalidateQueries({ queryKey: ["tag-cloud"] });
}, [finishedTaskId, queryClient]);
const startMutation = useMutation({
mutationFn: (scope: Scope) =>
api<MatchTask>("/api/matches/", { method: "POST", json: { scope } }),
onSuccess: (data) => {
setTaskId(data.task_id);
setMenuOpen(false);
},
});
const cancelMutation = useMutation({
mutationFn: () =>
api<{ success: boolean }>(`/api/matches/${taskId}/cancel/`, {
method: "POST",
}),
});
if (running && task) {
const percent = task.total
? Math.round((task.processed * 100) / task.total)
: 0;
return (
<div className="flex items-center gap-3">
<div className="w-36">
<div className="h-1.5 overflow-hidden rounded-full bg-ctp-surface0">
<div
className="h-full bg-ctp-mauve transition-all"
style={{ width: `${percent}%` }}
/>
</div>
</div>
<span className="font-mono text-xs text-ctp-subtext0">
{task.processed}/{task.total} · {task.matched} matched
</span>
<Button
variant="ghost"
disabled={cancelMutation.isPending}
onClick={() => cancelMutation.mutate()}
>
Cancel
</Button>
</div>
);
}
if (task) {
return (
<div className="flex items-center gap-2 text-xs">
{task.status === "complete" ? (
<span className="text-ctp-green">Scan complete — {summary(task)}</span>
) : task.status === "cancelled" ? (
<span className="text-ctp-subtext0">Scan cancelled.</span>
) : (
<span className="text-ctp-red">
{task.error || "The scan failed."}
</span>
)}
<button
type="button"
className="text-ctp-overlay0 transition hover:text-ctp-text"
onClick={() => setTaskId(null)}
title="Dismiss"
>
<X className="h-3.5 w-3.5" />
</button>
</div>
);
}
return (
<div className="relative">
<Button variant="secondary" onClick={() => setMenuOpen((open) => !open)}>
{startMutation.isPending ? (
<Spinner className="h-3.5 w-3.5" />
) : (
<RefreshCw className="h-4 w-4" />
)}
Match e621
</Button>
{menuOpen ? (
<div className="absolute right-0 top-full z-20 mt-2 w-72 rounded-lg border border-ctp-surface1 bg-ctp-base p-1.5 shadow-lg">
<button
type="button"
className="flex w-full flex-col rounded-md px-2.5 py-2 text-left transition hover:bg-ctp-surface0"
onClick={() => startMutation.mutate("missing")}
>
<span className="text-sm text-ctp-text">
Check unchecked items
</span>
<span className="text-xs text-ctp-overlay0">
MD5 lookups for items never checked against e621.
</span>
</button>
<button
type="button"
className="flex w-full flex-col rounded-md px-2.5 py-2 text-left transition hover:bg-ctp-surface0"
onClick={() => startMutation.mutate("all")}
>
<span className="text-sm text-ctp-text">Recheck the library</span>
<span className="text-xs text-ctp-overlay0">
Refreshes matched metadata and flags deleted posts too.
</span>
</button>
{startMutation.isError ? (
<p className="px-2.5 py-1.5 text-xs text-ctp-red">
{errorMessage(startMutation.error)}
</p>
) : null}
</div>
) : null}
</div>
);
}
+123 -11
View File
@@ -7,13 +7,13 @@ import {
Heart,
HeartOff,
} from "lucide-react";
import { useMemo } from "react";
import { useMemo, useEffect, useState } from "react";
import { Link, useLocation, useNavigate } from "react-router-dom";
import { Button, EmptyState, Spinner, linkButtonClass } from "@/components/ui";
import { TagChip } from "@/components/TagChip";
import { DetailShell } from "@/features/detail/DetailShell";
import { api } from "@/lib/api";
import { api, errorMessage } from "@/lib/api";
import { cn } from "@/lib/cn";
import { renderDText } from "@/lib/dtext";
import {
@@ -27,7 +27,7 @@ import {
TAG_CATEGORY_ORDER,
} from "@/lib/e621";
import { formatBytes, formatDate } from "@/lib/format";
import type { MediaItem } from "@/lib/types";
import type { DownloadTask } from "@/lib/types";
import { useAuth } from "@/store/auth";
import { useE621 } from "@/store/e621";
@@ -116,9 +116,11 @@ export function OnlineDetail({ postId }: { postId: number }) {
},
});
const [downloadTaskId, setDownloadTaskId] = useState<string | null>(null);
const downloadMutation = useMutation({
mutationFn: (target: NonNullable<typeof post>) =>
api<MediaItem>("/api/online/downloads/", {
api<DownloadTask>("/api/online/downloads/", {
method: "POST",
json: {
post_id: target.id,
@@ -127,13 +129,51 @@ export function OnlineDetail({ postId }: { postId: number }) {
post: target,
},
}),
onSuccess: (item) => {
void queryClient.invalidateQueries({ queryKey: ["library-lookup"] });
void queryClient.invalidateQueries({ queryKey: ["files"] });
navigate(`/detail/${item.j_id}`);
onSuccess: (task) => {
setDownloadTaskId(task.task_id);
void queryClient.invalidateQueries({ queryKey: ["status"] });
},
});
const taskQuery = useQuery({
queryKey: ["download-task", downloadTaskId],
queryFn: () =>
api<DownloadTask>(`/api/online/downloads/${downloadTaskId}/`),
enabled: Boolean(downloadTaskId),
refetchInterval: (query) => {
const status = query.state.data?.status;
return status === "pending" || status === "downloading" ? 700 : false;
},
});
const downloadTask = taskQuery.data;
const downloadStatus = downloadTask?.status;
const downloadActive =
downloadStatus === "pending" || downloadStatus === "downloading";
useEffect(() => {
if (!downloadStatus) return;
if (
downloadStatus === "complete" ||
downloadStatus === "error" ||
downloadStatus === "cancelled"
) {
void queryClient.invalidateQueries({ queryKey: ["status"] });
}
if (downloadStatus === "complete") {
void queryClient.invalidateQueries({ queryKey: ["library-lookup"] });
void queryClient.invalidateQueries({ queryKey: ["files"] });
}
}, [downloadStatus, queryClient]);
function cancelDownload() {
if (!downloadTaskId) return;
void api(`/api/online/downloads/${downloadTaskId}/cancel/`, {
method: "POST",
}).then(() => {
void queryClient.invalidateQueries({ queryKey: ["status"] });
});
}
if (query.isPending) {
return (
<div className="flex justify-center py-24">
@@ -163,6 +203,11 @@ export function OnlineDetail({ postId }: { postId: number }) {
const imageSrc =
post.sample.has && post.sample.url ? post.sample.url : post.file.url;
const canUpload = Boolean(user?.can_upload);
const clientDownloadUrl = post.file.url
? `/api/online/file/?url=${encodeURIComponent(post.file.url)}&filename=${encodeURIComponent(
`${post.file.md5}.${post.file.ext}`,
)}`
: null;
return (
<DetailShell
@@ -213,7 +258,10 @@ export function OnlineDetail({ postId }: { postId: number }) {
<Button
variant="secondary"
disabled={
!canUpload || downloadMutation.isPending || !post.file.url
!canUpload ||
downloadMutation.isPending ||
downloadActive ||
!post.file.url
}
onClick={() => downloadMutation.mutate(post)}
title={
@@ -223,12 +271,22 @@ export function OnlineDetail({ postId }: { postId: number }) {
}
>
<Download className="h-4 w-4" />
{downloadMutation.isPending
{downloadMutation.isPending || downloadActive
? "Downloading…"
: "Download to Library"}
</Button>
)}
{clientDownloadUrl ? (
<a
href={clientDownloadUrl}
className={linkButtonClass}
title="Save the original file straight to your computer"
>
<Download className="h-4 w-4" /> Download to client
</a>
) : null}
<a
href={`https://e621.net/posts/${post.id}`}
target="_blank"
@@ -239,6 +297,59 @@ export function OnlineDetail({ postId }: { postId: number }) {
</a>
</div>
{downloadTask &&
(downloadTask.status === "pending" ||
downloadTask.status === "downloading") ? (
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
<div className="flex flex-wrap items-center justify-between gap-3 text-xs">
<span className="text-ctp-subtext1">
Downloading to the library… {downloadTask.progress}%
</span>
<span className="font-mono text-ctp-overlay0">
{formatBytes(downloadTask.downloaded)}
{downloadTask.total > 0
? ` / ${formatBytes(downloadTask.total)}`
: ""}
{downloadTask.speed
? ` · ${(downloadTask.speed / 1024 / 1024).toFixed(1)} MB/s`
: ""}
</span>
</div>
<div className="mt-2 h-1.5 w-full overflow-hidden rounded-full bg-ctp-surface0">
<div
className="h-full bg-ctp-mauve transition-[width]"
style={{ width: `${downloadTask.progress}%` }}
/>
</div>
<button
type="button"
onClick={cancelDownload}
className="mt-2 rounded-md px-1.5 py-0.5 font-mono text-[10px] text-ctp-overlay0 transition hover:bg-ctp-surface0 hover:text-ctp-text"
>
cancel
</button>
</div>
) : downloadTask?.status === "complete" ? (
<p className="flex items-center gap-2 rounded-md border border-ctp-green/30 bg-ctp-green/10 px-3 py-2 text-sm text-ctp-green">
<CheckCircle2 className="h-4 w-4 shrink-0" />
Downloaded to the library
{downloadTask.library_j_id ? (
<Link
to={`/detail/${downloadTask.library_j_id}`}
className="font-mono text-ctp-blue hover:underline"
>
{downloadTask.library_j_id}
</Link>
) : null}
</p>
) : downloadTask?.status === "cancelled" ? (
<p className="text-xs text-ctp-overlay0">Download cancelled.</p>
) : downloadTask?.status === "error" ? (
<p className="rounded-md border border-ctp-red/30 bg-ctp-red/10 px-3 py-2 text-sm text-ctp-red">
{downloadTask.error || "Download failed."}
</p>
) : null}
{favoriteMutation.isError ? (
<p className="rounded-md border border-ctp-red/30 bg-ctp-red/10 px-3 py-2 text-sm text-ctp-red">
{e621ErrorMessage(favoriteMutation.error)}
@@ -246,7 +357,7 @@ export function OnlineDetail({ postId }: { postId: number }) {
) : null}
{downloadMutation.isError ? (
<p className="rounded-md border border-ctp-red/30 bg-ctp-red/10 px-3 py-2 text-sm text-ctp-red">
{e621ErrorMessage(downloadMutation.error)}
{errorMessage(downloadMutation.error)}
</p>
) : null}
</>
@@ -292,6 +403,7 @@ export function OnlineDetail({ postId }: { postId: number }) {
key={tag}
tag={tag}
category={category}
followable
onClick={() =>
navigate(`/online?tags=${encodeURIComponent(tag)}`)
}
+1 -1
View File
@@ -31,7 +31,7 @@ export function PostCard({
to={`/detail/${post.id}`}
state={from ? { from } : undefined}
className={cn(
"group overflow-hidden rounded-lg border bg-ctp-base transition",
"group block overflow-hidden rounded-lg border bg-ctp-base transition",
ratingBorder[post.rating] ?? "border-ctp-surface0",
)}
>
@@ -0,0 +1,314 @@
import { useQuery } from "@tanstack/react-query";
import { ArrowLeft, Archive, Eye, EyeOff } from "lucide-react";
import { useMemo, useState } from "react";
import { Link, useLocation, useParams } from "react-router-dom";
import { Button, EmptyState, Spinner } from "@/components/ui";
import { PostCard } from "@/features/online/PostCard";
import { PoolFollowButton } from "@/features/follows/PoolFollowButton";
import { api } from "@/lib/api";
import { cn } from "@/lib/cn";
import { renderDText } from "@/lib/dtext";
import {
e621ErrorMessage,
effectiveCredentials,
fetchPool,
fetchPostsByIds,
postIsBlacklisted,
type E621Pool,
type E621Post,
} from "@/lib/e621";
import { formatDate } from "@/lib/format";
import { useE621 } from "@/store/e621";
const CHUNK = 100;
async function fetchPostsChunked(
credentials: ReturnType<typeof effectiveCredentials>,
ids: number[],
signal?: AbortSignal,
): Promise<E621Post[]> {
const posts: E621Post[] = [];
for (let index = 0; index < ids.length; index += CHUNK) {
const chunk = ids.slice(index, index + CHUNK);
posts.push(...(await fetchPostsByIds(credentials, chunk, signal)));
}
return posts;
}
const categoryBadge: Record<string, string> = {
series: "border-ctp-mauve/40 bg-ctp-mauve/15 text-ctp-mauve",
collection: "border-ctp-blue/40 bg-ctp-blue/15 text-ctp-blue",
};
export default function PoolDetailPage() {
const { poolId = "" } = useParams();
const id = Number(poolId);
const location = useLocation();
const credentials = useE621((state) => state.credentials);
const blacklist = useE621((state) => state.blacklist);
const blacklistSet = useMemo(() => new Set(blacklist), [blacklist]);
const [visibleCount, setVisibleCount] = useState(CHUNK);
const [showBlacklisted, setShowBlacklisted] = useState(false);
const poolQuery = useQuery({
queryKey: ["e621", "pool", id, credentials?.username ?? ""],
queryFn: ({ signal }) =>
fetchPool(effectiveCredentials(credentials), id, signal),
enabled: Number.isFinite(id) && id > 0,
staleTime: 5 * 60_000,
});
const pool: E621Pool | undefined = poolQuery.data;
const postIds = useMemo(
() => (pool?.post_ids ?? []).slice(0, visibleCount),
[pool, visibleCount],
);
const postsQuery = useQuery({
queryKey: [
"e621",
"pool-posts",
id,
postIds.join(","),
credentials?.username ?? "",
],
queryFn: ({ signal }) =>
fetchPostsChunked(
effectiveCredentials(credentials),
postIds,
signal,
),
enabled: postIds.length > 0,
staleTime: 5 * 60_000,
});
// Keep the pool's own post order, dropping ids e621 no longer serves.
const orderedPosts = useMemo(() => {
const byId = new Map<number, E621Post>();
for (const post of postsQuery.data ?? []) byId.set(post.id, post);
return postIds
.map((postIdValue) => byId.get(postIdValue))
.filter((post): post is E621Post => Boolean(post));
}, [postsQuery.data, postIds]);
const md5s = useMemo(
() => orderedPosts.map((post) => post.file.md5),
[orderedPosts],
);
const lookupQuery = useQuery({
queryKey: ["library-lookup", md5s.join(",")],
queryFn: () =>
api<{ found: string[] }>("/api/files/lookup/", {
method: "POST",
json: { md5s },
}),
enabled: md5s.length > 0,
staleTime: 60_000,
});
const inLibrary = useMemo(
() => new Set(lookupQuery.data?.found ?? []),
[lookupQuery.data],
);
const { shownPosts, hiddenCount } = useMemo(() => {
if (blacklistSet.size === 0 || showBlacklisted) {
return { shownPosts: orderedPosts, hiddenCount: 0 };
}
const shown: E621Post[] = [];
let hidden = 0;
for (const post of orderedPosts) {
if (postIsBlacklisted(post, blacklistSet).length > 0) {
hidden += 1;
} else {
shown.push(post);
}
}
return { shownPosts: shown, hiddenCount: hidden };
}, [orderedPosts, blacklistSet, showBlacklisted]);
if (!Number.isFinite(id) || id <= 0) {
return (
<EmptyState
title="Pool not found"
description={
<Link to="/pools" className="text-ctp-blue hover:underline">
Back to pools
</Link>
}
/>
);
}
if (poolQuery.isPending) {
return (
<div className="flex justify-center py-24">
<Spinner className="h-6 w-6" />
</div>
);
}
if (poolQuery.isError || !pool) {
return (
<EmptyState
title="Could not load this pool"
description={
<>
{e621ErrorMessage(poolQuery.error)}{" "}
<Link to="/pools" className="text-ctp-blue hover:underline">
Back to pools
</Link>
</>
}
/>
);
}
const remaining = pool.post_ids.length - postIds.length;
return (
<div className="mx-auto flex w-full max-w-6xl flex-col gap-6">
<header className="flex flex-wrap items-start justify-between gap-4">
<div className="min-w-0">
<Link
to="/pools"
className="inline-flex items-center gap-1 text-sm text-ctp-subtext0 transition hover:text-ctp-text"
>
<ArrowLeft className="h-4 w-4" /> Pools
</Link>
<h1 className="mt-2 break-words text-lg font-semibold">
{pool.name}
</h1>
<p className="mt-1 flex flex-wrap items-center gap-2 text-xs text-ctp-overlay0">
<span
className={cn(
"rounded-full border px-1.5 py-0.5 font-mono text-[10px] font-medium capitalize",
categoryBadge[pool.category] ??
"border-ctp-surface1 text-ctp-subtext0",
)}
>
{pool.category}
</span>
<span className="font-mono">#{pool.id}</span>
<span>{pool.post_count} posts</span>
<span>by {pool.creator_name || `#${pool.creator_id}`}</span>
<span>· updated {formatDate(pool.updated_at)}</span>
{!pool.is_active ? (
<span className="flex items-center gap-1 text-ctp-red">
<Archive className="h-3 w-3" /> deleted
</span>
) : null}
</p>
</div>
<PoolFollowButton poolId={pool.id} name={pool.name} />
</header>
{pool.description.trim() ? (
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Description
</h2>
<div className="mt-2 text-sm leading-relaxed text-ctp-subtext0">
{renderDText(pool.description)}
</div>
</section>
) : null}
<section className="flex flex-col gap-3">
<div className="flex flex-wrap items-center justify-between gap-2">
<h2 className="text-sm font-semibold text-ctp-subtext1">
Posts{" "}
<span className="font-mono text-[11px] text-ctp-overlay0">
{shownPosts.length}
{hiddenCount > 0 ? ` of ${orderedPosts.length}` : ""}
</span>
</h2>
{hiddenCount > 0 || showBlacklisted ? (
<button
type="button"
onClick={() => setShowBlacklisted((value) => !value)}
className="inline-flex items-center gap-1.5 text-xs text-ctp-overlay0 transition hover:text-ctp-text"
>
{showBlacklisted ? (
<>
<EyeOff className="h-3.5 w-3.5" /> Hide blacklisted
</>
) : (
<>
<Eye className="h-3.5 w-3.5" /> Reveal {hiddenCount}{" "}
blacklisted
</>
)}
</button>
) : null}
</div>
{postsQuery.isPending ? (
<div className="flex justify-center py-24">
<Spinner className="h-6 w-6" />
</div>
) : postsQuery.isError ? (
<EmptyState
title="Could not load the pool's posts"
description={e621ErrorMessage(postsQuery.error)}
/>
) : shownPosts.length === 0 ? (
<EmptyState
title="No posts to show"
description={
hiddenCount > 0
? "Everything here is blacklisted."
: "This pool has no available posts."
}
/>
) : (
<div className="grid grid-cols-[repeat(auto-fill,minmax(190px,1fr))] gap-3">
{shownPosts.map((post) => {
const blacklisted =
blacklistSet.size > 0 &&
postIsBlacklisted(post, blacklistSet).length > 0;
if (!blacklisted) {
return (
<PostCard
key={post.id}
post={post}
inLibrary={inLibrary.has(post.file.md5)}
from={`${location.pathname}${location.search}`}
/>
);
}
return (
<div
key={post.id}
className="min-w-0 rounded-lg ring-1 ring-ctp-red/50"
title="Matches your blacklist"
>
<PostCard
post={post}
inLibrary={inLibrary.has(post.file.md5)}
from={`${location.pathname}${location.search}`}
/>
</div>
);
})}
</div>
)}
{remaining > 0 ? (
<div className="mt-2 flex justify-center">
<Button
variant="secondary"
disabled={postsQuery.isFetching}
onClick={() => setVisibleCount((count) => count + CHUNK)}
>
{postsQuery.isFetching
? "Loading…"
: `Load more posts (${remaining} remaining)`}
</Button>
</div>
) : null}
</section>
</div>
);
}
+352
View File
@@ -0,0 +1,352 @@
import { keepPreviousData, useQuery } from "@tanstack/react-query";
import { Archive, Layers, Search } from "lucide-react";
import { useMemo, useState } from "react";
import { Link, useSearchParams } from "react-router-dom";
import { CollapsibleSidebar } from "@/components/CollapsibleSidebar";
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
import { cn } from "@/lib/cn";
import {
e621ErrorMessage,
effectiveCredentials,
fetchPools,
fetchPostsByIds,
postIsBlacklisted,
type E621Pool,
type E621PoolOrder,
type E621Post,
} from "@/lib/e621";
import { formatDate } from "@/lib/format";
import { useE621 } from "@/store/e621";
const PAGE_SIZE = 48;
const EMPTY_POOLS: E621Pool[] = [];
const categoryBadge: Record<string, string> = {
series: "border-ctp-mauve/40 bg-ctp-mauve/15 text-ctp-mauve",
collection: "border-ctp-blue/40 bg-ctp-blue/15 text-ctp-blue",
};
const orderOptions: { value: E621PoolOrder; label: string }[] = [
{ value: "id_desc", label: "Newest first" },
{ value: "id_asc", label: "Oldest first" },
{ value: "name", label: "Name A–Z" },
{ value: "post_count", label: "Most posts" },
{ value: "created_at", label: "Recently active" },
];
export default function PoolsPage() {
const [searchParams, setSearchParams] = useSearchParams();
const credentials = useE621((state) => state.credentials);
const blacklist = useE621((state) => state.blacklist);
const blacklistSet = useMemo(() => new Set(blacklist), [blacklist]);
const q = searchParams.get("q") ?? "";
const category = searchParams.get("category") ?? "";
const status = searchParams.get("status") ?? "";
const order = (searchParams.get("order") ?? "id_desc") as E621PoolOrder;
const page = Math.max(1, Number(searchParams.get("page") ?? "1") || 1);
const [draft, setDraft] = useState<string | null>(null);
const nameInput = draft ?? q;
const poolsQuery = useQuery({
queryKey: [
"e621",
"pools",
q,
category,
status,
order,
page,
credentials?.username ?? "",
],
queryFn: ({ signal }) =>
fetchPools(effectiveCredentials(credentials), {
name: q,
category,
isActive: status,
order,
page,
limit: PAGE_SIZE,
signal,
}),
placeholderData: keepPreviousData,
staleTime: 5 * 60_000,
});
const pools = poolsQuery.data ?? EMPTY_POOLS;
// Covers come from each pool's first post, fetched in one batch.
const coverIds = useMemo(
() =>
[
...new Set(
pools
.map((pool) => pool.post_ids[0])
.filter((id): id is number => typeof id === "number"),
),
],
[pools],
);
const coversQuery = useQuery({
queryKey: ["e621", "pool-covers", coverIds.join(","), credentials?.username ?? ""],
queryFn: ({ signal }) =>
fetchPostsByIds(effectiveCredentials(credentials), coverIds, signal),
enabled: coverIds.length > 0,
staleTime: 30 * 60_000,
});
const covers = useMemo(() => {
const map = new Map<number, E621Post>();
for (const post of coversQuery.data ?? []) map.set(post.id, post);
return map;
}, [coversQuery.data]);
function buildParams(values: {
q: string;
category: string;
status: string;
order: string;
page: number;
}): URLSearchParams {
const params = new URLSearchParams();
if (values.q.trim()) params.set("q", values.q.trim());
if (values.category) params.set("category", values.category);
if (values.status) params.set("status", values.status);
if (values.order && values.order !== "id_desc")
params.set("order", values.order);
if (values.page > 1) params.set("page", String(values.page));
return params;
}
function update(patch: Partial<{ q: string; category: string; status: string; order: string }>) {
setDraft(null);
setSearchParams(
buildParams({
q: patch.q ?? q,
category: patch.category ?? category,
status: patch.status ?? status,
order: patch.order ?? order,
page: 1,
}),
);
}
const hasNext = pools.length >= PAGE_SIZE;
return (
<div className="flex flex-col gap-6 lg:flex-row lg:items-start">
<CollapsibleSidebar>
<form
onSubmit={(event) => {
event.preventDefault();
update({ q: nameInput });
}}
className="flex flex-col gap-2"
>
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Pool name
</span>
<div className="relative">
<Search className="pointer-events-none absolute left-2.5 top-1/2 h-4 w-4 -translate-y-1/2 text-ctp-overlay0" />
<input
className={cn(inputClass, "pl-8 font-mono")}
placeholder="Name contains…"
value={nameInput}
onChange={(event) => setDraft(event.target.value)}
/>
</div>
<div className="flex gap-2">
<Button type="submit" className="flex-1">
Search
</Button>
<Button
type="button"
variant="secondary"
onClick={() => update({ q: "" })}
>
Clear
</Button>
</div>
</form>
<label className="flex flex-col gap-2">
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Category
</span>
<select
className={inputClass}
value={category}
onChange={(event) => update({ category: event.target.value })}
>
<option value="">Any category</option>
<option value="series">Series</option>
<option value="collection">Collection</option>
</select>
</label>
<label className="flex flex-col gap-2">
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Status
</span>
<select
className={inputClass}
value={status}
onChange={(event) => update({ status: event.target.value })}
>
<option value="">Any status</option>
<option value="true">Active</option>
<option value="false">Deleted</option>
</select>
</label>
<label className="flex flex-col gap-2">
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
Sort
</span>
<select
className={inputClass}
value={order}
onChange={(event) => update({ order: event.target.value })}
>
{orderOptions.map((option) => (
<option key={option.value} value={option.value}>
{option.label}
</option>
))}
</select>
</label>
</CollapsibleSidebar>
<section className="min-w-0 flex-1">
<div className="mb-4 flex items-center justify-between gap-3">
<p className="text-sm text-ctp-overlay0">
{poolsQuery.data
? `${pools.length} pool${pools.length === 1 ? "" : "s"}`
: "…"}
</p>
<p className="font-mono text-xs text-ctp-overlay0">Page {page}</p>
</div>
{poolsQuery.isPending ? (
<div className="flex justify-center py-24">
<Spinner className="h-6 w-6" />
</div>
) : poolsQuery.isError ? (
<EmptyState
title="Could not load pools from e621"
description={e621ErrorMessage(poolsQuery.error)}
/>
) : pools.length === 0 ? (
<EmptyState
title="No pools found"
description="Try a different name, category or status."
/>
) : (
<div
className={cn(
"grid grid-cols-[repeat(auto-fill,minmax(220px,1fr))] gap-3 transition-opacity",
poolsQuery.isPlaceholderData && "opacity-50",
)}
>
{pools.map((pool) => {
const cover = covers.get(pool.post_ids[0] ?? -1);
const coverBlacklisted = Boolean(
cover &&
blacklistSet.size > 0 &&
postIsBlacklisted(cover, blacklistSet).length > 0,
);
return (
<Link
key={pool.id}
to={`/pools/${pool.id}`}
className="group overflow-hidden rounded-lg border border-ctp-surface0 bg-ctp-base transition hover:border-ctp-surface2"
>
<div className="relative aspect-[4/3] overflow-hidden bg-ctp-mantle">
{cover && cover.preview.url && !coverBlacklisted ? (
<img
src={cover.preview.url}
alt={pool.name}
loading="lazy"
className="h-full w-full object-cover transition duration-200 group-hover:scale-[1.03]"
/>
) : (
<div className="flex h-full w-full flex-col items-center justify-center gap-1 text-ctp-overlay0">
<Layers className="h-5 w-5" />
<span className="text-[10px]">
{coverBlacklisted ? "Blacklisted cover" : "No cover"}
</span>
</div>
)}
<span
className={cn(
"absolute left-2 top-2 rounded-full border px-1.5 py-0.5 font-mono text-[10px] font-medium capitalize",
categoryBadge[pool.category] ??
"border-ctp-surface1 bg-ctp-base/80 text-ctp-subtext0",
)}
>
{pool.category}
</span>
<span className="absolute right-2 top-2 rounded-sm bg-ctp-crust/80 px-1.5 py-0.5 font-mono text-[10px] uppercase text-ctp-overlay1">
{pool.post_count} posts
</span>
{!pool.is_active ? (
<span className="absolute bottom-2 left-2 flex items-center gap-1 rounded-full bg-ctp-red/90 px-1.5 py-0.5 font-mono text-[10px] font-medium text-ctp-crust">
<Archive className="h-3 w-3" /> deleted
</span>
) : null}
</div>
<div className="px-2.5 py-2">
<p
className="truncate font-mono text-xs text-ctp-subtext1"
title={pool.name}
>
{pool.name}
</p>
<p className="mt-0.5 truncate text-[11px] text-ctp-overlay0">
#{pool.id} · {pool.creator_name || "unknown"} · updated{" "}
{formatDate(pool.updated_at)}
</p>
</div>
</Link>
);
})}
</div>
)}
{page > 1 || hasNext ? (
<div className="mt-6 flex items-center justify-center gap-3">
<Button
variant="secondary"
disabled={page <= 1 || poolsQuery.isFetching}
onClick={() =>
setSearchParams(
buildParams({ q, category, status, order, page: page - 1 }),
)
}
>
Previous
</Button>
<span className="font-mono text-xs text-ctp-subtext0">
Page {page}
</span>
<Button
variant="secondary"
disabled={!hasNext || poolsQuery.isFetching}
onClick={() =>
setSearchParams(
buildParams({ q, category, status, order, page: page + 1 }),
)
}
>
Next
</Button>
</div>
) : null}
</section>
</div>
);
}
File diff suppressed because it is too large Load Diff
+180
View File
@@ -0,0 +1,180 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { useState } from "react";
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
import { api, errorMessage } from "@/lib/api";
import { cn } from "@/lib/cn";
import { formatDate } from "@/lib/format";
import type { AdminUser, Paginated } from "@/lib/types";
import { useAuth } from "@/store/auth";
const ROLE_OPTIONS = [
{ value: "user", label: "User" },
{ value: "uploader", label: "Uploader" },
{ value: "staff", label: "Staff" },
];
function UserRow({ profile }: { profile: AdminUser }) {
const queryClient = useQueryClient();
const [avatarJId, setAvatarJId] = useState(profile.avatar_j_id ?? "");
const [message, setMessage] = useState<string | null>(null);
const [error, setError] = useState<string | null>(null);
const mutation = useMutation({
mutationFn: (payload: Record<string, string | null>) =>
api<AdminUser>(`/api/users/${profile.id}/`, {
method: "PATCH",
json: payload,
}),
onSuccess: (updated) => {
setMessage("Saved.");
setError(null);
setAvatarJId(updated.avatar_j_id ?? "");
void queryClient.invalidateQueries({ queryKey: ["users"] });
},
onError: (err) => {
setError(errorMessage(err));
setMessage(null);
},
});
return (
<div className="flex flex-col gap-3 rounded-lg border border-ctp-surface0 bg-ctp-base p-3 lg:flex-row lg:items-center">
<div className="flex min-w-0 flex-1 items-center gap-3">
{profile.avatar_url ? (
<img
src={profile.avatar_url}
alt={profile.username}
className="h-10 w-10 shrink-0 rounded-full border border-ctp-surface1 object-cover"
/>
) : (
<span className="flex h-10 w-10 shrink-0 items-center justify-center rounded-full bg-ctp-surface1 font-mono text-sm font-semibold text-ctp-lavender">
{profile.username.slice(0, 1).toUpperCase()}
</span>
)}
<div className="min-w-0">
<p className="flex flex-wrap items-center gap-2 text-sm text-ctp-subtext1">
<span className="font-medium">{profile.username}</span>
{profile.is_superuser ? (
<span className="rounded-full border border-ctp-red/40 bg-ctp-red/10 px-2 py-0.5 font-mono text-[10px] text-ctp-red">
admin
</span>
) : null}
{profile.is_staff && !profile.is_superuser ? (
<span className="rounded-full border border-ctp-surface1 px-2 py-0.5 font-mono text-[10px] text-ctp-overlay0">
django staff
</span>
) : null}
</p>
<p className="truncate font-mono text-[11px] text-ctp-overlay0">
{profile.email || "no email"} · joined{" "}
{formatDate(profile.date_joined)} · {profile.uploads_count} upload
{profile.uploads_count === 1 ? "" : "s"}
</p>
</div>
</div>
<div className="flex flex-wrap items-center gap-2">
<select
className={cn(inputClass, "w-32")}
value={profile.role}
onChange={(event) =>
mutation.mutate({ role: event.target.value })
}
disabled={mutation.isPending}
title="Role"
>
{ROLE_OPTIONS.map((option) => (
<option key={option.value} value={option.value}>
{option.label}
</option>
))}
</select>
<input
className={cn(inputClass, "w-28 font-mono")}
placeholder="J-ID"
value={avatarJId}
onChange={(event) => setAvatarJId(event.target.value)}
title="Avatar library item"
/>
<Button
variant="secondary"
disabled={mutation.isPending}
onClick={() => mutation.mutate({ avatar_j_id: avatarJId.trim() })}
>
Save avatar
</Button>
</div>
{message ? (
<p className="text-xs text-ctp-green lg:w-20">{message}</p>
) : null}
{error ? (
<p className="text-xs text-ctp-red lg:w-40">{error}</p>
) : null}
</div>
);
}
export default function UsersPage() {
const user = useAuth((state) => state.user);
const [search, setSearch] = useState("");
const isStaff = Boolean(
user &&
(user.is_staff || user.is_superuser || user.role === "staff"),
);
const query = useQuery({
queryKey: ["users", search],
queryFn: () =>
api<Paginated<AdminUser>>(
`/api/users/${search ? `?search=${encodeURIComponent(search)}` : ""}`,
),
enabled: isStaff,
});
if (!isStaff) {
return (
<EmptyState
title="Staff only"
description="This page is limited to staff accounts."
/>
);
}
return (
<div className="mx-auto flex w-full max-w-5xl flex-col gap-4">
<header className="flex flex-wrap items-end justify-between gap-3">
<div>
<h1 className="text-lg font-semibold">Users</h1>
<p className="mt-1 text-sm text-ctp-overlay0">
Manage roles and set profile pictures from library J-IDs.
</p>
</div>
<input
className={cn(inputClass, "w-56")}
placeholder="Search username…"
value={search}
onChange={(event) => setSearch(event.target.value)}
/>
</header>
{query.isPending ? (
<div className="flex justify-center py-16">
<Spinner className="h-6 w-6" />
</div>
) : query.isError || !query.data ? (
<EmptyState title="Could not load users" />
) : query.data.results.length === 0 ? (
<EmptyState title="No users found" />
) : (
<div className="flex flex-col gap-3">
{query.data.results.map((profile) => (
<UserRow key={profile.id} profile={profile} />
))}
</div>
)}
</div>
);
}
+17
View File
@@ -0,0 +1,17 @@
const AGE_KEY = "j621.age-verified";
export function isAgeVerified(): boolean {
try {
return window.localStorage.getItem(AGE_KEY) === "1";
} catch {
return false;
}
}
export function markAgeVerified(): void {
try {
window.localStorage.setItem(AGE_KEY, "1");
} catch {
// Private mode with storage disabled: the gate simply shows again.
}
}
+10 -3
View File
@@ -1,4 +1,4 @@
import type { MediaItem } from "@/lib/types";
import type { TempUpload } from "@/lib/types";
const TOKEN_KEY = "j621.token";
@@ -6,6 +6,13 @@ export function getToken(): string | null {
return window.localStorage.getItem(TOKEN_KEY);
}
export function authHeaders(extra?: HeadersInit): Headers {
const headers = new Headers(extra);
const token = getToken();
if (token) headers.set("Authorization", `Token ${token}`);
return headers;
}
export function setToken(token: string | null): void {
if (token) {
window.localStorage.setItem(TOKEN_KEY, token);
@@ -82,7 +89,7 @@ export function uploadFile(
file: File,
fields: Record<string, string> = {},
onProgress?: (percent: number) => void,
): Promise<MediaItem> {
): Promise<TempUpload> {
return new Promise((resolve, reject) => {
const form = new FormData();
form.append("file", file);
@@ -107,7 +114,7 @@ export function uploadFile(
data = null;
}
if (request.status >= 200 && request.status < 300) {
resolve(data as MediaItem);
resolve(data as TempUpload);
} else {
reject(new ApiError(request.status, data));
}
+203
View File
@@ -296,6 +296,209 @@ export async function setFavorite(
});
}
export async function fetchPostsByMd5(
credentials: E621Credentials,
md5s: string[],
signal?: AbortSignal,
): Promise<E621Post[]> {
if (md5s.length === 0) return [];
const data = await e621Request<{ posts: E621Post[] }>(
credentials,
"/posts.json",
{
params: {
tags: `md5:${md5s.join(",")}`,
limit: Math.min(md5s.length, 100),
},
signal,
},
);
return data.posts;
}
export interface E621Pool {
id: number;
name: string;
created_at: string;
updated_at: string;
creator_id: number;
creator_name: string;
description: string;
is_active: boolean;
category: "collection" | "series";
post_ids: number[];
post_count: number;
}
export type E621PoolOrder =
| "id_desc"
| "id_asc"
| "name"
| "created_at"
| "post_count";
export async function fetchPools(
credentials: E621Credentials,
options: {
name?: string;
category?: string;
isActive?: string;
order?: E621PoolOrder;
page?: number;
limit?: number;
signal?: AbortSignal;
} = {},
): Promise<E621Pool[]> {
const data = await e621Request<unknown>(credentials, "/pools.json", {
params: {
limit: options.limit ?? 48,
page: options.page,
"search[order]": options.order ?? "id_desc",
"search[name_matches]": options.name || undefined,
"search[category]": options.category || undefined,
"search[is_active]": options.isActive || undefined,
},
signal: options.signal,
});
// The spec says a bare array; tolerate a wrapped payload too.
if (Array.isArray(data)) return data as E621Pool[];
const wrapped = (data as { pools?: E621Pool[] })?.pools;
return Array.isArray(wrapped) ? wrapped : [];
}
export async function fetchPool(
credentials: E621Credentials,
poolId: number,
signal?: AbortSignal,
): Promise<E621Pool> {
const data = await e621Request<unknown>(credentials, `/pools/${poolId}.json`, {
signal,
});
const wrapped = (data as { pool?: E621Pool })?.pool;
return (wrapped ?? (data as E621Pool)) as E621Pool;
}
export interface E621IqdbResult {
post_id: number | null;
score: number | null;
preview_url: string | null;
rating: string | null;
md5: string | null;
score_total: number | null;
fav_count: number | null;
width: number | null;
height: number | null;
tags_preview: string[];
}
function flattenTagPreview(tags: unknown, limit = 8): string[] {
if (!tags || typeof tags !== "object") return [];
const out: string[] = [];
for (const value of Object.values(tags as Record<string, unknown>)) {
if (!Array.isArray(value)) continue;
for (const tag of value) {
if (typeof tag === "string" && !out.includes(tag)) {
out.push(tag);
if (out.length >= limit) return out;
}
}
}
return out;
}
function normalizeIqdbEntry(entry: unknown): E621IqdbResult | null {
if (!entry || typeof entry !== "object") return null;
const record = entry as Record<string, unknown>;
const postWrap = record.post;
let post: Record<string, unknown> = {};
if (postWrap && typeof postWrap === "object") {
const wrap = postWrap as Record<string, unknown>;
post =
wrap.posts && typeof wrap.posts === "object"
? (wrap.posts as Record<string, unknown>)
: wrap;
}
const file = (post.file ?? {}) as Record<string, unknown>;
const preview = (post.preview ?? {}) as Record<string, unknown>;
const postId =
typeof record.post_id === "number"
? record.post_id
: typeof post.id === "number"
? post.id
: null;
const scoreTotal = post.score;
return {
post_id: postId,
score: typeof record.score === "number" ? record.score : null,
preview_url: typeof preview.url === "string" ? preview.url : null,
rating: typeof post.rating === "string" ? post.rating : null,
md5: typeof file.md5 === "string" ? file.md5 : null,
score_total: typeof scoreTotal === "number" ? scoreTotal : null,
fav_count: typeof post.fav_count === "number" ? post.fav_count : null,
width: typeof file.width === "number" ? file.width : null,
height: typeof file.height === "number" ? file.height : null,
tags_preview: flattenTagPreview(post.tags),
};
}
export async function iqdbSearch(
credentials: E621Credentials,
file: Blob,
filename: string,
signal?: AbortSignal,
): Promise<E621IqdbResult[]> {
const form = new FormData();
form.append("file", file, filename);
const data = await e621Request<unknown>(credentials, "/iqdb_queries.json", {
method: "POST",
body: form,
signal,
});
const list = Array.isArray(data)
? data
: Array.isArray((data as { matches?: unknown[] })?.matches)
? (data as { matches: unknown[] }).matches
: [];
const candidates = list
.map(normalizeIqdbEntry)
.filter(
(entry): entry is E621IqdbResult =>
entry !== null && (entry.post_id !== null || entry.preview_url !== null),
)
.slice(0, 10);
// The IQDB payload carries no preview/file data; enrich with one batched
// post lookup so candidates can show what they actually are.
const ids = candidates
.map((candidate) => candidate.post_id)
.filter((id): id is number => typeof id === "number");
if (ids.length > 0) {
try {
const posts = await fetchPostsByIds(credentials, ids, signal);
const byId = new Map(posts.map((post) => [post.id, post]));
return candidates.map((candidate) => {
const post =
candidate.post_id !== null ? byId.get(candidate.post_id) : undefined;
if (!post) return candidate;
return {
...candidate,
preview_url: post.preview.url ?? candidate.preview_url,
rating: post.rating ?? candidate.rating,
md5: post.file.md5 ?? candidate.md5,
score_total: post.score.total,
fav_count: post.fav_count,
width: post.file.width,
height: post.file.height,
tags_preview: flattenTagPreview(post.tags),
};
});
} catch {
// Keep the base candidates when the enrichment request fails.
}
}
return candidates;
}
export function postIsBlacklisted(
post: E621Post,
blacklist: Set<string>,
+12
View File
@@ -0,0 +1,12 @@
import { QueryClient } from "@tanstack/react-query";
export const queryClient = new QueryClient({
defaultOptions: {
queries: {
retry: 1,
staleTime: 30_000,
gcTime: 30 * 60_000,
refetchOnWindowFocus: false,
},
},
});
+209
View File
@@ -7,10 +7,71 @@ export interface User {
date_joined: string;
role: "user" | "uploader" | "staff";
can_upload: boolean;
avatar_j_id: string | null;
avatar_url: string | null;
e621_username: string;
e621_configured: boolean;
}
export interface AdminUser {
id: number;
username: string;
email: string;
role: "user" | "uploader" | "staff";
is_staff: boolean;
is_superuser: boolean;
date_joined: string;
avatar_j_id: string | null;
avatar_url: string | null;
uploads_count: number;
}
export interface DuplicateEntry {
j_id: string;
md5: string;
filename: string;
size: number;
rating: string;
location_count: number;
uploaded_by: string | null;
e621_post_id: number | null;
locations?: { id: number; rel_path: string }[];
thumbnail_url?: string;
similarity?: number;
}
export interface VisualGroupsResponse {
count: number;
page: number;
per_page: number;
has_next: boolean;
threshold: number;
algorithms: string[];
groups: { size: number; members: DuplicateEntry[] }[];
}
export interface VisualMatchesResponse {
target: DuplicateEntry;
threshold: number;
algorithms: string[];
count: number;
matches: DuplicateEntry[];
}
export interface StorageInfo {
watched_folder: {
path: string;
total: number;
used: number;
free: number;
library_size: number;
percent_used: number;
};
media: { path: string; size: number };
temp: { path: string; size: number; files: number };
library_items: number;
}
export interface MediaLocation {
id: number;
path: string;
@@ -20,6 +81,8 @@ export interface MediaLocation {
export type Rating = "" | "s" | "q" | "e";
export type E621MatchStatus = "unknown" | "matched" | "not_found" | "deleted";
export interface E621StoredPost {
id: number | null;
created_at?: string | null;
@@ -65,7 +128,11 @@ export interface MediaItem {
uploaded_by_id: number | null;
e621_post_id: number | null;
e621_data: E621StoredPost | null;
e621_match_status: E621MatchStatus;
e621_checked_at: string | null;
locations: MediaLocation[];
raw_url: string;
thumbnail_url: string;
created_at: string;
updated_at: string;
}
@@ -106,3 +173,145 @@ export interface FileNeighbors {
previous: FileNeighbor | null;
next: FileNeighbor | null;
}
export interface DownloadTask {
task_id: string;
post_id: number | null;
filename: string;
status: "pending" | "downloading" | "complete" | "error" | "cancelled";
progress: number;
downloaded: number;
total: number;
speed: number | null;
error: string;
library_j_id: string | null;
created_at: string;
updated_at: string;
}
export interface MatchTask {
task_id: string;
scope: "missing" | "all";
status: "pending" | "running" | "complete" | "error" | "cancelled";
total: number;
processed: number;
matched: number;
not_found: number;
deleted: number;
failed: number;
error: string;
created_at: string;
updated_at: string;
}
export interface FollowCoverData {
post_id: number | null;
rating: string | null;
preview_url: string | null;
score: number | null;
fav_count: number | null;
}
export interface FollowedTag {
id: number;
tag: string;
unseen_count: number;
cover_data: FollowCoverData | null;
last_synced_at: string | null;
created_at: string;
}
export interface FollowedPool {
id: number;
pool_id: number;
name: string;
post_count: number;
unseen_count: number;
cover_data: FollowCoverData | null;
last_synced_at: string | null;
created_at: string;
}
export interface FollowFeedSource {
kind: "tag" | "pool";
id: number;
label: string;
}
export interface FollowFeedEntry {
post_id: number;
seen: boolean;
data: {
id?: number | null;
rating?: string | null;
tags?: Record<string, string[]>;
score?: { total?: number | null } | null;
fav_count?: number | null;
file?: {
width?: number | null;
height?: number | null;
ext?: string | null;
} | null;
preview?: { url?: string | null } | null;
};
sources: FollowFeedSource[];
}
export interface FollowFeed {
count: number;
page: number;
per_page: number;
has_next: boolean;
results: FollowFeedEntry[];
}
export interface FollowCloud {
status: "ready" | "building";
stale: boolean;
tags: [string, number][];
posts: number;
source: "user" | "default" | null;
blacklist_count: number;
computed_at: string | null;
}
export interface E621IqdbCandidate {
post_id: number | null;
score: number | null;
preview_url: string | null;
rating: string | null;
md5: string | null;
score_total?: number | null;
fav_count?: number | null;
width?: number | null;
height?: number | null;
tags_preview?: string[];
}
export interface TempUpload {
temp_id: string;
original_filename: string;
md5: string;
size: number;
status: "pending" | "visual_match" | "completed" | "error";
resolution: "" | "auto_md5" | "duplicate" | "linked" | "custom";
e621_post_id: number | null;
e621_data: E621StoredPost | null;
custom_rating: Rating;
custom_tags: string[];
custom_notes: string;
iqdb_data: E621IqdbCandidate[] | null;
visual_matches:
| {
j_id: string;
filename: string;
similarity: number;
thumbnail_url?: string;
}[]
| null;
library_j_id: string | null;
file_url: string | null;
preview_url: string | null;
created_at: string;
updated_at: string;
}
+2 -12
View File
@@ -1,21 +1,11 @@
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { QueryClientProvider } from "@tanstack/react-query";
import { StrictMode } from "react";
import { createRoot } from "react-dom/client";
import { BrowserRouter } from "react-router-dom";
import App from "./App";
import "./index.css";
const queryClient = new QueryClient({
defaultOptions: {
queries: {
retry: 1,
staleTime: 30_000,
gcTime: 30 * 60_000,
refetchOnWindowFocus: false,
},
},
});
import { queryClient } from "@/lib/queryClient";
createRoot(document.getElementById("root")!).render(
<StrictMode>
+7
View File
@@ -1,6 +1,7 @@
import { create } from "zustand";
import { api, getToken, setToken } from "@/lib/api";
import { queryClient } from "@/lib/queryClient";
import type { User } from "@/lib/types";
import { useE621 } from "@/store/e621";
@@ -24,6 +25,9 @@ export const useAuth = create<AuthState>((set) => ({
});
setToken(data.token);
const user = await api<User>("/api/auth/me/");
// Never show the previous account's cached data to this one.
queryClient.clear();
useE621.getState().clear();
set({ user, ready: true });
},
@@ -33,6 +37,8 @@ export const useAuth = create<AuthState>((set) => ({
json: { username, password },
});
setToken(data.token);
queryClient.clear();
useE621.getState().clear();
set({ user: data.user, ready: true });
},
@@ -43,6 +49,7 @@ export const useAuth = create<AuthState>((set) => ({
// The token may already be invalid; clearing it locally is enough.
}
setToken(null);
queryClient.clear();
useE621.getState().clear();
set({ user: null, ready: true });
},
+15 -6
View File
@@ -6,10 +6,13 @@ export interface E621RequestSample {
at: number;
}
const HISTORY_LIMIT = 50;
interface E621MetricsState {
requestCount: number;
totalMs: number;
last: E621RequestSample | null;
history: E621RequestSample[];
record: (path: string, ms: number) => void;
reset: () => void;
}
@@ -19,13 +22,19 @@ export const useE621Metrics = create<E621MetricsState>((set) => ({
requestCount: 0,
totalMs: 0,
last: null,
history: [],
record: (path, ms) =>
set((state) => ({
requestCount: state.requestCount + 1,
totalMs: state.totalMs + ms,
last: { path, ms, at: Date.now() },
})),
set((state) => {
const sample: E621RequestSample = { path, ms, at: Date.now() };
return {
requestCount: state.requestCount + 1,
totalMs: state.totalMs + ms,
last: sample,
history: [sample, ...state.history].slice(0, HISTORY_LIMIT),
};
}),
reset: () => set({ requestCount: 0, totalMs: 0, last: null }),
reset: () =>
set({ requestCount: 0, totalMs: 0, last: null, history: [] }),
}));
+5
View File
@@ -7,6 +7,8 @@ interface UiState {
toggleSidebar: () => void;
ratings: string[];
setRatings: (ratings: string[]) => void;
statuses: string[];
setStatuses: (statuses: string[]) => void;
perPage: number;
setPerPage: (perPage: number) => void;
ordering: string;
@@ -23,6 +25,9 @@ export const useUi = create<UiState>()(
ratings: [],
setRatings: (ratings) => set({ ratings }),
statuses: [],
setStatuses: (statuses) => set({ statuses }),
perPage: 48,
setPerPage: (perPage) => set({ perPage }),