Compare commits
26
Commits
afca89de8f
...
cea34422ec
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cea34422ec | ||
|
|
e95dc5f265 | ||
|
|
62561a13f3 | ||
|
|
4c0b8cb1d8 | ||
|
|
26f5069aa2 | ||
|
|
0a7dc5a991 | ||
|
|
c396c368dc | ||
|
|
4585a8ac43 | ||
|
|
03dd235f3a | ||
|
|
c55fa1fdca | ||
|
|
a0d36911e7 | ||
|
|
09405d1a0f | ||
|
|
db74f7ab18 | ||
|
|
4df573da43 | ||
|
|
7ec8ee974e | ||
|
|
cd490b0a23 | ||
|
|
75b7ed35eb | ||
|
|
6962e483fc | ||
|
|
bf00cf36a2 | ||
|
|
d4f5df2e4e | ||
|
|
a3d1063946 | ||
|
|
1086beb974 | ||
|
|
b71ec729e0 | ||
|
|
7deb6084b6 | ||
|
|
d0e2901c92 | ||
|
|
60e1113231 |
@@ -0,0 +1,81 @@
|
||||
Jake Labs Non-Commercial Software Licence
|
||||
|
||||
Copyright (c) 2026 JakeBreath
|
||||
|
||||
0. Authorized Distribution and Scope
|
||||
|
||||
This Licence applies exclusively to copies of the Software that have been
|
||||
lawfully obtained from a public distribution channel explicitly designated
|
||||
by the Licensor (e.g., GitHub, GitLab, Codeberg, Itch.io, Steam, or any
|
||||
similar public repository or platform) where the Software has been published
|
||||
by the copyright holder. Any copy of the Software acquired through any other
|
||||
means - including but not limited to breach of contract, violation of a
|
||||
non-disclosure agreement, unauthorized access to private repositories,
|
||||
exploitation of security vulnerabilities, hacking, or accessing a developer's
|
||||
personal computer, local network, or development device - is not authorized,
|
||||
and this Licence does not grant any rights whatsoever with respect to such
|
||||
copies. Such copies receive no licence and remain fully protected by
|
||||
copyright. No rights under this Licence are conveyed to any recipient of such
|
||||
an unauthorized copy.
|
||||
|
||||
1. Grant of Rights and Conditions
|
||||
|
||||
Subject to the terms of this Licence, you are granted a worldwide,
|
||||
royalty-free, non-exclusive, irrevocable licence to use, copy, modify, merge,
|
||||
publish, distribute, and sublicense copies of the Software, provided that:
|
||||
|
||||
a) Attribution: You must give appropriate credit, provide a link to this
|
||||
Licence, and indicate if changes were made. You may do so in any
|
||||
reasonable manner, but not in any way that suggests the Licensor
|
||||
endorses you or your use.
|
||||
|
||||
b) Copyleft: Any modified version of the Software that you distribute must
|
||||
be licensed as a whole under this same Licence, and you must make the
|
||||
corresponding source code readily available to recipients under these
|
||||
same terms. This requirement applies to the modified work as a whole,
|
||||
including any combined or derivative works that include any portion of
|
||||
the Software.
|
||||
|
||||
c) Strictly Non-Commercial: Commercial use of the Software, in original or
|
||||
modified form, is prohibited. "Commercial use" includes, but is not
|
||||
limited to:
|
||||
|
||||
(i) selling the Software, including it in a product or service for
|
||||
which a fee is charged, or using it to provide a paid service;
|
||||
|
||||
(ii) using the Software (directly or indirectly) on any website,
|
||||
platform, application, or online service that displays
|
||||
advertisements, sponsored content, or other promotional
|
||||
material intended to generate revenue - including incidental
|
||||
or automatically injected advertisements;
|
||||
|
||||
(iii) using the Software as part of the backend infrastructure,
|
||||
content delivery network (CDN), or server-side processing of
|
||||
any website, application, or service that restricts access to
|
||||
content behind a paywall, subscription, or other payment
|
||||
mechanism, or that otherwise monetizes content through
|
||||
advertisements, commercial data collection, or similar means.
|
||||
|
||||
Any output, data, or results generated by the Software may not be
|
||||
used in connection with any such monetized or advertisement-supported
|
||||
platform.
|
||||
|
||||
2. Term and Termination
|
||||
|
||||
Your rights under this Licence terminate automatically if you violate any of
|
||||
its terms. If you cure the violation within 30 days of it occurring, your
|
||||
rights are reinstated automatically as of the date the violation is cured. On
|
||||
termination you must stop using and distributing the Software. Termination
|
||||
does not affect the rights of recipients who obtained copies from you in
|
||||
compliance with this Licence. The Disclaimer of Warranty survives
|
||||
termination.
|
||||
|
||||
3. Disclaimer of Warranty
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
|
||||
IN THE SOFTWARE.
|
||||
@@ -36,3 +36,11 @@ npm run dev # http://localhost:5173, proxies /api to
|
||||
Not wired up yet — planned: Nginx serving the SPA build, `/media` and `/library` media
|
||||
directly, and proxying `/api` to Waitress/Django. See `frontend/` design docs for the UI
|
||||
specification.
|
||||
|
||||
## Licence
|
||||
|
||||
Source-available, **non-commercial**: personal and other non-commercial use is
|
||||
welcome under the [Jake Labs Non-Commercial Software Licence](LICENSE), which
|
||||
requires attribution and keeps derivative works under the same licence.
|
||||
Commercial use is not permitted. Third-party dependencies keep their own
|
||||
licences (all permissive: MIT, BSD, Apache-2.0, ISC).
|
||||
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
# J621 Roadmap / To-Dos
|
||||
|
||||
Working list of what's still missing, roughly in priority order. Check items off as
|
||||
they land.
|
||||
|
||||
## 1. Library management
|
||||
|
||||
- [x] **Duplicates engine**
|
||||
- [x] Exact MD5 duplicate detection with grouped results
|
||||
- [x] Perceptual similarity (aHash, dHash, pHash, wHash) with threshold slider
|
||||
and algorithm toggles (`imagehash` server-side)
|
||||
- [x] Visual similarity groups: pagination, selection, delete and dismiss
|
||||
- [x] **Delete & storage page**
|
||||
- [x] Storage overview (watched folder, media folder, temp)
|
||||
- [x] Delete by J-ID with preview grid and bulk selection (plus per-copy
|
||||
deletion of duplicate locations)
|
||||
- [x] Temp folder cleanup
|
||||
- [x] **Library search upgrades**
|
||||
- [x] Search by tags (custom + e621 tags) with a Filename / Tags / Both selector
|
||||
- [x] Tag cloud in the sidebar (click to search, hidden from guests for
|
||||
blacklisted items)
|
||||
- [x] Status filter (matched / custom / unknown — not_found/deleted arrive with
|
||||
the e621 match cache)
|
||||
|
||||
## 2. e621 integration
|
||||
|
||||
- [x] **Download progress bar on the detail view**
|
||||
- [x] Backend download task (status, progress %, downloaded/total bytes, speed)
|
||||
- [x] Progress endpoint the SPA polls; cancel support
|
||||
- [x] Frontend progress bar with speed and cancel on `/detail/<post>`
|
||||
- [x] The status footer's "Active Workers" now counts running download tasks
|
||||
- [x] **Download to client** — streams the e621 original straight to the
|
||||
browser (works for guests; host-restricted proxy, no library write)
|
||||
- [x] **Match local files to e621**
|
||||
- [x] Match by MD5 from the library detail, plus manual post ID linking
|
||||
(with an MD5-mismatch warning) and unlink
|
||||
- [x] Batch cache status for the whole library (`matched` / `not_found` /
|
||||
`deleted`) via background scan tasks (`/api/matches/`) and
|
||||
`manage.py match_e621`
|
||||
- [x] Show match status + e621 metadata in the library: status badge and
|
||||
match card on the detail, and `not_found` / `deleted` status filters
|
||||
- [x] **IQDB reverse search**
|
||||
- [x] Search from a local file (`POST /iqdb_queries.json` from the library detail,
|
||||
using the signed raw file)
|
||||
- [x] Show candidate posts to link (thumbnail, rating, score, favs, tags;
|
||||
select then confirm — same flow as uploads, exact MD5 marked)
|
||||
- [x] **Follows**
|
||||
- [x] Follow tags/pools from the `/followed` page (validated against e621,
|
||||
first feed fetched immediately)
|
||||
- [x] Followed screens: cover cards (latest image), unseen badges and
|
||||
Mark seen per follow
|
||||
- [x] Merged newest-first feed with per-follow filter and unseen-only toggle
|
||||
- [x] Blacklisted-tag cloud built in a daemon thread, then polled
|
||||
(10 min TTL; the user's e621 blacklist, guest default as fallback)
|
||||
- [x] "Followed" nav entry
|
||||
- [x] Periodic sync commands: `sync_followed_tags` and
|
||||
`sync_followed_pools` (one e621 search per unique tag/pool)
|
||||
- [x] **Pools browser** (`/pools` + `/pools/<id>`)
|
||||
- [x] Index search by name; category, active/deleted and sort filters;
|
||||
pagination according to the e621 OpenAPI spec
|
||||
- [x] Cover thumbnails from each pool's first post (one batched post call,
|
||||
blacklist-aware) and deleted-pool markers
|
||||
- [x] Pool detail: DText description, posts kept in pool order with chunked
|
||||
loading and in-library badges, blacklist reveal toggle, Follow button
|
||||
- [ ] `F` shortcut to favorite a post, tag finder in the Ctrl+K palette
|
||||
|
||||
## 3. Uploads & ingestion pipeline
|
||||
|
||||
Files now stage first and are resolved before entering the library.
|
||||
|
||||
- [x] **Backend staging storage**
|
||||
- [x] `TempUpload` model: user, md5, filename, temp path, status
|
||||
(`pending` / `visual_match` / `completed` / `error`), resolution
|
||||
- [x] Files land in a temp folder first; only completed uploads move into
|
||||
the watched library folder
|
||||
- [x] Pending/upload list endpoint, file serving, discard endpoint
|
||||
- [x] `cleanup_temp_uploads` command for old staged files
|
||||
- [x] **Auto-upload / auto-match**
|
||||
- [x] MD5 computed on staging; exact duplicates resolve immediately
|
||||
- [x] MD5 batch-checked against e621; matches auto-complete with post
|
||||
metadata stored and rating seeded
|
||||
- [x] **IQDB similarity on upload** (SPA-driven)
|
||||
- [x] Automatic + manual IQDB checks with candidate posts
|
||||
- [x] "Visual Similarity Detected" state with candidate picker
|
||||
- [x] Perceptual-hash comparison against the library (staged uploads are
|
||||
flagged with their library matches as soon as they land)
|
||||
- [x] **Upload UI**
|
||||
- [x] Three-column board: Pending & Unmatched / Visual Similarity Detected /
|
||||
Auto-uploaded & Indexed
|
||||
- [x] Metadata modal (link to e621 post, IQDB candidates, custom metadata)
|
||||
- [x] Per-file progress plus batch processing indicator
|
||||
|
||||
## 4. Staff tools
|
||||
|
||||
- [ ] **Optimization modal** (image quality, GIF/APNG resolution+FPS+compression,
|
||||
video bitrate/presets/two-pass/hw-accel) with original vs processed preview
|
||||
and override
|
||||
- [ ] **Stats dashboard**
|
||||
- [ ] CPU / RAM / GPU / disk usage
|
||||
- [ ] Active jobs list (this is what makes the footer "Active Workers" real)
|
||||
- [ ] Live log tail
|
||||
|
||||
## 5. Shell & polish
|
||||
|
||||
- [x] 18+ entry screen: an age check gates the app before anything renders
|
||||
(remembered per browser in `localStorage`)
|
||||
- [ ] Toasts instead of inline messages / confirm dialogs
|
||||
- [ ] Mobile drawer polish for metadata panels (design spec §layout)
|
||||
- [x] Profile pictures: staff Users page sets avatars from library J-IDs
|
||||
(self-service picker in Account still pending)
|
||||
- [ ] Profile extras (per-user browse preferences)
|
||||
- [ ] Command palette: tag finder, recent searches
|
||||
|
||||
## 6. Infrastructure
|
||||
|
||||
- [ ] Guest blacklist refresh on a timer (`refresh_guest_blacklist` via cron/systemd)
|
||||
- [ ] Follow sync on a timer (`sync_followed_tags` + `sync_followed_pools` via
|
||||
cron/systemd, e.g. every 30 minutes)
|
||||
- [ ] Production setup: build the SPA, serve via Nginx (static + `/media` +
|
||||
`/library`), systemd unit for Waitress
|
||||
- [ ] Automated tests (backend API + frontend components)
|
||||
- [ ] Backfill e621 metadata for items downloaded before metadata was stored
|
||||
(re-download or a "match" action)
|
||||
|
||||
## Dependencies / notes
|
||||
|
||||
- Duplicates and upload visual-similarity share the perceptual hashing layer
|
||||
(`imagehash`/`imgdd` server-side + a hash cache table).
|
||||
- Download progress, optimization jobs and the stats "Active Workers" count all
|
||||
want the same background-task/progress primitive — design it once.
|
||||
- IQDB and e621 matching depend on e621 credentials being configured; the SPA
|
||||
talks to e621 directly for browsing, while the backend e621 client
|
||||
(`apps/library/e621.py`) handles metadata matching and batch scans.
|
||||
@@ -0,0 +1,8 @@
|
||||
from rest_framework.routers import DefaultRouter
|
||||
|
||||
from .views import UserViewSet
|
||||
|
||||
router = DefaultRouter()
|
||||
router.register("users", UserViewSet, basename="user")
|
||||
|
||||
urlpatterns = router.urls
|
||||
@@ -0,0 +1,20 @@
|
||||
# Generated by Django 6.1.1 on 2026-09-17 17:34
|
||||
|
||||
import django.db.models.deletion
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('accounts', '0003_user_role'),
|
||||
('library', '0006_mediaitem_ahash_mediaitem_dhash_mediaitem_phash_and_more'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name='user',
|
||||
name='avatar',
|
||||
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to='library.mediaitem'),
|
||||
),
|
||||
]
|
||||
@@ -15,6 +15,13 @@ class User(AbstractUser):
|
||||
]
|
||||
|
||||
role = models.CharField(max_length=20, choices=ROLE_CHOICES, default=ROLE_USER)
|
||||
avatar = models.ForeignKey(
|
||||
"library.MediaItem",
|
||||
null=True,
|
||||
blank=True,
|
||||
on_delete=models.SET_NULL,
|
||||
related_name="+",
|
||||
)
|
||||
e621_username = models.CharField(max_length=100, blank=True, default="")
|
||||
e621_api_key = models.CharField(max_length=100, blank=True, default="")
|
||||
e621_base_url = models.CharField(max_length=200, default="https://e621.net")
|
||||
|
||||
@@ -1,12 +1,39 @@
|
||||
from pathlib import Path
|
||||
|
||||
from django.contrib.auth.password_validation import validate_password
|
||||
from django.core import signing
|
||||
from rest_framework import serializers
|
||||
|
||||
from apps.library.services import MEDIA_FILE_SALT, VIDEO_EXTENSIONS
|
||||
|
||||
from .models import User
|
||||
|
||||
|
||||
def signed_media_url(request, item):
|
||||
"""A signed media URL an <img> tag can load for an authenticated user."""
|
||||
if item is None:
|
||||
return None
|
||||
user = getattr(request, "user", None)
|
||||
if user is None or not getattr(user, "is_authenticated", False):
|
||||
return None
|
||||
location = item.locations.first()
|
||||
is_video = (
|
||||
location is not None
|
||||
and Path(location.rel_path).suffix.lower() in VIDEO_EXTENSIONS
|
||||
)
|
||||
action = "thumbnail" if is_video else "raw"
|
||||
signature = signing.dumps(
|
||||
{"item": item.id, "user": user.id, "action": action},
|
||||
salt=MEDIA_FILE_SALT,
|
||||
)
|
||||
return f"/api/files/J-{item.id}/{action}/?sig={signature}"
|
||||
|
||||
|
||||
class UserSerializer(serializers.ModelSerializer):
|
||||
e621_configured = serializers.BooleanField(read_only=True)
|
||||
can_upload = serializers.BooleanField(read_only=True)
|
||||
avatar_j_id = serializers.SerializerMethodField()
|
||||
avatar_url = serializers.SerializerMethodField()
|
||||
|
||||
class Meta:
|
||||
model = User
|
||||
@@ -19,11 +46,54 @@ class UserSerializer(serializers.ModelSerializer):
|
||||
"date_joined",
|
||||
"role",
|
||||
"can_upload",
|
||||
"avatar_j_id",
|
||||
"avatar_url",
|
||||
"e621_username",
|
||||
"e621_configured",
|
||||
]
|
||||
read_only_fields = fields
|
||||
|
||||
def get_avatar_j_id(self, obj):
|
||||
return f"J-{obj.avatar_id}" if obj.avatar_id else None
|
||||
|
||||
def get_avatar_url(self, obj):
|
||||
return signed_media_url(self.context.get("request"), obj.avatar)
|
||||
|
||||
|
||||
class UserListSerializer(serializers.ModelSerializer):
|
||||
avatar_j_id = serializers.SerializerMethodField()
|
||||
avatar_url = serializers.SerializerMethodField()
|
||||
uploads_count = serializers.IntegerField(read_only=True)
|
||||
|
||||
class Meta:
|
||||
model = User
|
||||
fields = [
|
||||
"id",
|
||||
"username",
|
||||
"email",
|
||||
"role",
|
||||
"is_staff",
|
||||
"is_superuser",
|
||||
"date_joined",
|
||||
"avatar_j_id",
|
||||
"avatar_url",
|
||||
"uploads_count",
|
||||
]
|
||||
read_only_fields = fields
|
||||
|
||||
def get_avatar_j_id(self, obj):
|
||||
return f"J-{obj.avatar_id}" if obj.avatar_id else None
|
||||
|
||||
def get_avatar_url(self, obj):
|
||||
return signed_media_url(self.context.get("request"), obj.avatar)
|
||||
|
||||
|
||||
class UserUpdateSerializer(serializers.Serializer):
|
||||
avatar_j_id = serializers.CharField(
|
||||
required=False, allow_blank=True, allow_null=True
|
||||
)
|
||||
role = serializers.ChoiceField(choices=User.ROLE_CHOICES, required=False)
|
||||
|
||||
|
||||
class RegisterSerializer(serializers.ModelSerializer):
|
||||
password = serializers.CharField(write_only=True, validators=[validate_password])
|
||||
|
||||
@@ -1,13 +1,19 @@
|
||||
from rest_framework import status
|
||||
from rest_framework import mixins, permissions, status, viewsets
|
||||
from rest_framework.authtoken.models import Token
|
||||
from rest_framework.permissions import AllowAny, IsAuthenticated
|
||||
from rest_framework.response import Response
|
||||
from rest_framework.views import APIView
|
||||
from django.db.models import Count, Q
|
||||
|
||||
from apps.library.models import MediaItem
|
||||
|
||||
from .models import User
|
||||
from .serializers import (
|
||||
E621CredentialsSerializer,
|
||||
RegisterSerializer,
|
||||
UserListSerializer,
|
||||
UserSerializer,
|
||||
UserUpdateSerializer,
|
||||
)
|
||||
|
||||
|
||||
@@ -20,7 +26,10 @@ class RegisterView(APIView):
|
||||
user = serializer.save()
|
||||
token, _ = Token.objects.get_or_create(user=user)
|
||||
return Response(
|
||||
{"token": token.key, "user": UserSerializer(user).data},
|
||||
{
|
||||
"token": token.key,
|
||||
"user": UserSerializer(user, context={"request": request}).data,
|
||||
},
|
||||
status=status.HTTP_201_CREATED,
|
||||
)
|
||||
|
||||
@@ -29,7 +38,7 @@ class MeView(APIView):
|
||||
permission_classes = [IsAuthenticated]
|
||||
|
||||
def get(self, request):
|
||||
return Response(UserSerializer(request.user).data)
|
||||
return Response(UserSerializer(request.user, context={"request": request}).data)
|
||||
|
||||
|
||||
class LogoutView(APIView):
|
||||
@@ -71,3 +80,81 @@ class E621CredentialsView(APIView):
|
||||
update_fields=["e621_username", "e621_api_key", "e621_base_url"]
|
||||
)
|
||||
return Response(self._payload(user))
|
||||
|
||||
|
||||
class IsStaffUser(permissions.BasePermission):
|
||||
message = "Staff only."
|
||||
|
||||
def has_permission(self, request, view):
|
||||
user = request.user
|
||||
return bool(
|
||||
user
|
||||
and user.is_authenticated
|
||||
and (
|
||||
user.is_superuser
|
||||
or user.is_staff
|
||||
or user.role == user.ROLE_STAFF
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class UserViewSet(
|
||||
mixins.ListModelMixin,
|
||||
mixins.RetrieveModelMixin,
|
||||
viewsets.GenericViewSet,
|
||||
):
|
||||
"""Staff user directory: roles and J-ID avatars."""
|
||||
|
||||
permission_classes = [IsStaffUser]
|
||||
http_method_names = ["get", "patch", "head", "options"]
|
||||
|
||||
def get_queryset(self):
|
||||
queryset = User.objects.annotate(uploads_count=Count("uploads")).order_by("id")
|
||||
search = self.request.query_params.get("search", "").strip()
|
||||
if search:
|
||||
queryset = queryset.filter(
|
||||
Q(username__icontains=search) | Q(email__icontains=search)
|
||||
)
|
||||
return queryset
|
||||
|
||||
def get_serializer_class(self):
|
||||
if self.request.method in {"PATCH", "PUT"}:
|
||||
return UserUpdateSerializer
|
||||
return UserListSerializer
|
||||
|
||||
def update(self, request, *args, **kwargs):
|
||||
user = self.get_object()
|
||||
serializer = UserUpdateSerializer(data=request.data)
|
||||
serializer.is_valid(raise_exception=True)
|
||||
data = serializer.validated_data
|
||||
|
||||
update_fields = []
|
||||
if "role" in data:
|
||||
user.role = data["role"]
|
||||
update_fields.append("role")
|
||||
if "avatar_j_id" in data:
|
||||
value = str(data.get("avatar_j_id") or "").strip()
|
||||
if not value:
|
||||
user.avatar = None
|
||||
else:
|
||||
numeric = value[2:] if value.upper().startswith("J-") else value
|
||||
item = (
|
||||
MediaItem.objects.filter(pk=int(numeric)).first()
|
||||
if numeric.isdigit()
|
||||
else None
|
||||
)
|
||||
if item is None:
|
||||
return Response(
|
||||
{"detail": f"No library item {value}."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
user.avatar = item
|
||||
update_fields.append("avatar")
|
||||
if update_fields:
|
||||
user.save(update_fields=update_fields)
|
||||
|
||||
return Response(
|
||||
UserListSerializer(user, context=self.get_serializer_context()).data
|
||||
)
|
||||
|
||||
partial_update = update
|
||||
|
||||
@@ -9,11 +9,16 @@ from .system_info import get_os_info
|
||||
|
||||
|
||||
def _worker_counts():
|
||||
"""Active/queued background jobs.
|
||||
"""Active/queued background jobs (downloads and e621 match scans)."""
|
||||
from apps.library.models import DownloadTask, MatchTask
|
||||
|
||||
No task pipeline exists yet, so this reports honest zeros for now.
|
||||
"""
|
||||
return {"active": 0, "queued": 0}
|
||||
active = DownloadTask.objects.filter(
|
||||
status=DownloadTask.STATUS_DOWNLOADING
|
||||
).count() + MatchTask.objects.filter(status=MatchTask.STATUS_RUNNING).count()
|
||||
queued = DownloadTask.objects.filter(
|
||||
status=DownloadTask.STATUS_PENDING
|
||||
).count() + MatchTask.objects.filter(status=MatchTask.STATUS_PENDING).count()
|
||||
return {"active": active, "queued": queued}
|
||||
|
||||
|
||||
class StatusView(APIView):
|
||||
@@ -30,7 +35,7 @@ class StatusView(APIView):
|
||||
"version": settings.APP_VERSION,
|
||||
"os": get_os_info(),
|
||||
"watched_folder": settings.WATCHED_FOLDER,
|
||||
"e621_time_ms": None, # no e621 calls in the backend yet
|
||||
"e621_time_ms": None, # e621 latency is measured client-side
|
||||
"workers": _worker_counts(),
|
||||
}
|
||||
payload["server_time_ms"] = round((time.perf_counter() - started) * 1000, 1)
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
from django.apps import AppConfig
|
||||
|
||||
|
||||
class FollowsConfig(AppConfig):
|
||||
name = "apps.follows"
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Blacklisted-tag cloud for the Followed page.
|
||||
|
||||
The cloud counts tags from the user's e621 blacklist that appear in their
|
||||
followed feeds. Building it means one e621 request (the blacklist itself)
|
||||
plus a local scan of stored feed posts, so it runs in a daemon thread and
|
||||
the page polls until the fresh result lands.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from datetime import timedelta
|
||||
from fnmatch import fnmatch
|
||||
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.db import connection
|
||||
from django.utils import timezone
|
||||
|
||||
from apps.library import e621, guest_filter
|
||||
|
||||
from .models import FollowCloud, FollowedPost
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CLOUD_TTL = timedelta(minutes=10)
|
||||
BUILD_STALE = timedelta(minutes=5)
|
||||
MAX_CLOUD_TAGS = 60
|
||||
|
||||
|
||||
def blacklist_patterns_for(user):
|
||||
"""(patterns, source): the user's e621 blacklist, else the guest default."""
|
||||
if e621.credentials_configured(user):
|
||||
try:
|
||||
me = e621.get(user, "/users/me.json")
|
||||
lines = str(me.get("blacklisted_tags") or "").splitlines()
|
||||
patterns = guest_filter.parse_blacklist_patterns(lines)
|
||||
if patterns:
|
||||
return patterns, "user"
|
||||
except e621.E621Error as exc:
|
||||
logger.warning("Could not fetch the e621 blacklist for %s: %s", user, exc)
|
||||
return guest_filter.guest_blacklist_patterns(), "default"
|
||||
|
||||
|
||||
def compute_cloud(user):
|
||||
"""Count blacklist matches across every stored feed post of one user."""
|
||||
patterns, source = blacklist_patterns_for(user)
|
||||
counts = {}
|
||||
seen_posts = set()
|
||||
rows = FollowedPost.objects.filter(user=user).values_list("post_id", "data")
|
||||
for post_id, data in rows:
|
||||
if post_id in seen_posts:
|
||||
continue
|
||||
seen_posts.add(post_id)
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
categories = data.get("tags")
|
||||
if not isinstance(categories, dict):
|
||||
continue
|
||||
for values in categories.values():
|
||||
if not isinstance(values, list):
|
||||
continue
|
||||
for tag in values:
|
||||
name = str(tag).strip().lower()
|
||||
if name and any(fnmatch(name, pattern) for pattern in patterns):
|
||||
counts[name] = counts.get(name, 0) + 1
|
||||
tags = sorted(counts.items(), key=lambda entry: (-entry[1], entry[0]))
|
||||
return {
|
||||
"tags": [[tag, count] for tag, count in tags[:MAX_CLOUD_TAGS]],
|
||||
"posts": len(seen_posts),
|
||||
"source": source,
|
||||
"blacklist_count": len(patterns),
|
||||
}
|
||||
|
||||
|
||||
def start_cloud_build(user_id):
|
||||
thread = threading.Thread(target=run_cloud_build, args=(user_id,), daemon=True)
|
||||
thread.start()
|
||||
|
||||
|
||||
def run_cloud_build(user_id):
|
||||
User = get_user_model()
|
||||
try:
|
||||
user = User.objects.filter(id=user_id).first()
|
||||
if user is None:
|
||||
return
|
||||
data = compute_cloud(user)
|
||||
FollowCloud.objects.filter(user_id=user_id).update(
|
||||
data=data, computed_at=timezone.now()
|
||||
)
|
||||
except Exception: # noqa: BLE001 - background builds must not crash workers
|
||||
logger.exception("Blacklist cloud build failed for user %s", user_id)
|
||||
finally:
|
||||
FollowCloud.objects.filter(user_id=user_id).update(
|
||||
building=False, building_since=None
|
||||
)
|
||||
# Background threads hold their own DB connection; release it so
|
||||
# Gunicorn workers do not leak connections when threads finish.
|
||||
connection.close()
|
||||
@@ -0,0 +1,62 @@
|
||||
from django.core.management.base import BaseCommand, CommandError
|
||||
|
||||
from apps.follows import sync
|
||||
from apps.follows.models import FollowedPool
|
||||
from apps.library import e621
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = (
|
||||
"Fetch new posts for every followed pool: fills each feed, refreshes "
|
||||
"the card cover (latest image), pool name/count and unseen badges."
|
||||
)
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--user",
|
||||
help="Only sync follows belonging to this username "
|
||||
"(default: every user's follows).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--limit",
|
||||
type=int,
|
||||
default=sync.FEED_LIMIT,
|
||||
help=f"Posts fetched per pool (default {sync.FEED_LIMIT}).",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
username = options.get("user")
|
||||
if username and sync.preferred_fetch_user(username) is None:
|
||||
raise CommandError(f"No such user: {username}")
|
||||
fetch_user = sync.preferred_fetch_user(username)
|
||||
|
||||
follows = FollowedPool.objects.select_related("user").order_by(
|
||||
"pool_id", "user_id"
|
||||
)
|
||||
if username:
|
||||
follows = follows.filter(user__username=username)
|
||||
follows = list(follows)
|
||||
grouped = {}
|
||||
for follow in follows:
|
||||
grouped.setdefault(follow.pool_id, []).append(follow)
|
||||
|
||||
created = 0
|
||||
for pool_id, group in grouped.items():
|
||||
try:
|
||||
posts = sync.search_posts(
|
||||
fetch_user, f"pool:{pool_id}", options["limit"]
|
||||
)
|
||||
except e621.E621Error as exc:
|
||||
raise CommandError(f"Aborted at pool #{pool_id}: {exc}")
|
||||
for follow in group:
|
||||
created += sync.sync_pool_follow(
|
||||
follow, fetch_user, posts=posts
|
||||
)
|
||||
|
||||
identity = getattr(fetch_user, "username", None) or "anonymous"
|
||||
self.stdout.write(
|
||||
self.style.SUCCESS(
|
||||
f"Synced {len(grouped)} pool(s) across {len(follows)} follow(s) "
|
||||
f"as {identity}: {created} new post(s)."
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,60 @@
|
||||
from django.core.management.base import BaseCommand, CommandError
|
||||
|
||||
from apps.follows import sync
|
||||
from apps.follows.models import FollowedTag
|
||||
from apps.library import e621
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = (
|
||||
"Fetch new posts for every followed tag: fills each feed, refreshes "
|
||||
"the card cover (latest image) and unseen badges."
|
||||
)
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--user",
|
||||
help="Only sync follows belonging to this username "
|
||||
"(default: every user's follows).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--limit",
|
||||
type=int,
|
||||
default=sync.FEED_LIMIT,
|
||||
help=f"Posts fetched per tag (default {sync.FEED_LIMIT}).",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
username = options.get("user")
|
||||
if username and sync.preferred_fetch_user(username) is None:
|
||||
raise CommandError(f"No such user: {username}")
|
||||
fetch_user = sync.preferred_fetch_user(username)
|
||||
|
||||
follows = FollowedTag.objects.select_related("user").order_by(
|
||||
"tag", "user_id"
|
||||
)
|
||||
if username:
|
||||
follows = follows.filter(user__username=username)
|
||||
follows = list(follows)
|
||||
grouped = {}
|
||||
for follow in follows:
|
||||
grouped.setdefault(follow.tag, []).append(follow)
|
||||
|
||||
created = 0
|
||||
for tag, group in grouped.items():
|
||||
try:
|
||||
posts = sync.search_posts(fetch_user, tag, options["limit"])
|
||||
except e621.E621Error as exc:
|
||||
raise CommandError(f"Aborted at tag '{tag}': {exc}")
|
||||
for follow in group:
|
||||
created += sync.sync_tag_follow(
|
||||
follow, fetch_user, posts=posts
|
||||
)
|
||||
|
||||
identity = getattr(fetch_user, "username", None) or "anonymous"
|
||||
self.stdout.write(
|
||||
self.style.SUCCESS(
|
||||
f"Synced {len(grouped)} tag(s) across {len(follows)} follow(s) "
|
||||
f"as {identity}: {created} new post(s)."
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,94 @@
|
||||
# Generated by Django 6.1.1 on 2026-09-17 19:04
|
||||
|
||||
import django.db.models.deletion
|
||||
from django.conf import settings
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
initial = True
|
||||
|
||||
dependencies = [
|
||||
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name='FollowCloud',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('data', models.JSONField(blank=True, null=True)),
|
||||
('computed_at', models.DateTimeField(blank=True, null=True)),
|
||||
('building', models.BooleanField(default=False)),
|
||||
('building_since', models.DateTimeField(blank=True, null=True)),
|
||||
('user', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='follow_cloud', to=settings.AUTH_USER_MODEL)),
|
||||
],
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name='FollowedPool',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('pool_id', models.IntegerField()),
|
||||
('name', models.CharField(blank=True, default='', max_length=200)),
|
||||
('post_count', models.IntegerField(default=0)),
|
||||
('cover_data', models.JSONField(blank=True, null=True)),
|
||||
('last_synced_at', models.DateTimeField(blank=True, null=True)),
|
||||
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='followed_pools', to=settings.AUTH_USER_MODEL)),
|
||||
],
|
||||
options={
|
||||
'ordering': ['pool_id'],
|
||||
},
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name='FollowedTag',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('tag', models.CharField(max_length=200)),
|
||||
('cover_data', models.JSONField(blank=True, null=True)),
|
||||
('last_synced_at', models.DateTimeField(blank=True, null=True)),
|
||||
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='followed_tags', to=settings.AUTH_USER_MODEL)),
|
||||
],
|
||||
options={
|
||||
'ordering': ['tag'],
|
||||
},
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name='FollowedPost',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('post_id', models.IntegerField(db_index=True)),
|
||||
('data', models.JSONField(default=dict)),
|
||||
('seen', models.BooleanField(db_index=True, default=False)),
|
||||
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||
('pool', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='posts', to='follows.followedpool')),
|
||||
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='followed_posts', to=settings.AUTH_USER_MODEL)),
|
||||
('tag', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='posts', to='follows.followedtag')),
|
||||
],
|
||||
options={
|
||||
'ordering': ['-post_id'],
|
||||
},
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name='followedpool',
|
||||
constraint=models.UniqueConstraint(fields=('user', 'pool_id'), name='uniq_followed_pool_per_user'),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name='followedtag',
|
||||
constraint=models.UniqueConstraint(fields=('user', 'tag'), name='uniq_followed_tag_per_user'),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name='followedpost',
|
||||
index=models.Index(fields=['user', 'post_id'], name='follows_fol_user_id_47fdab_idx'),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name='followedpost',
|
||||
constraint=models.UniqueConstraint(fields=('tag', 'post_id'), name='uniq_followed_tag_post'),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name='followedpost',
|
||||
constraint=models.UniqueConstraint(fields=('pool', 'post_id'), name='uniq_followed_pool_post'),
|
||||
),
|
||||
]
|
||||
@@ -0,0 +1,114 @@
|
||||
from django.conf import settings
|
||||
from django.db import models
|
||||
|
||||
|
||||
class FollowedTag(models.Model):
|
||||
"""A tag one user subscribes to; feeds are filled by the sync command."""
|
||||
|
||||
user = models.ForeignKey(
|
||||
settings.AUTH_USER_MODEL,
|
||||
on_delete=models.CASCADE,
|
||||
related_name="followed_tags",
|
||||
)
|
||||
tag = models.CharField(max_length=200)
|
||||
cover_data = models.JSONField(null=True, blank=True)
|
||||
last_synced_at = models.DateTimeField(null=True, blank=True)
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
|
||||
class Meta:
|
||||
ordering = ["tag"]
|
||||
constraints = [
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "tag"], name="uniq_followed_tag_per_user"
|
||||
),
|
||||
]
|
||||
|
||||
def __str__(self):
|
||||
return f"{self.user} follows tag:{self.tag}"
|
||||
|
||||
|
||||
class FollowedPool(models.Model):
|
||||
"""A pool one user subscribes to; feeds are filled by the sync command."""
|
||||
|
||||
user = models.ForeignKey(
|
||||
settings.AUTH_USER_MODEL,
|
||||
on_delete=models.CASCADE,
|
||||
related_name="followed_pools",
|
||||
)
|
||||
pool_id = models.IntegerField()
|
||||
name = models.CharField(max_length=200, blank=True, default="")
|
||||
post_count = models.IntegerField(default=0)
|
||||
cover_data = models.JSONField(null=True, blank=True)
|
||||
last_synced_at = models.DateTimeField(null=True, blank=True)
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
|
||||
class Meta:
|
||||
ordering = ["pool_id"]
|
||||
constraints = [
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "pool_id"], name="uniq_followed_pool_per_user"
|
||||
),
|
||||
]
|
||||
|
||||
def __str__(self):
|
||||
return f"{self.user} follows pool:{self.pool_id}"
|
||||
|
||||
|
||||
class FollowedPost(models.Model):
|
||||
"""One post seen in a followed feed (stored once per follow source)."""
|
||||
|
||||
user = models.ForeignKey(
|
||||
settings.AUTH_USER_MODEL,
|
||||
on_delete=models.CASCADE,
|
||||
related_name="followed_posts",
|
||||
)
|
||||
tag = models.ForeignKey(
|
||||
FollowedTag,
|
||||
null=True,
|
||||
blank=True,
|
||||
on_delete=models.CASCADE,
|
||||
related_name="posts",
|
||||
)
|
||||
pool = models.ForeignKey(
|
||||
FollowedPool,
|
||||
null=True,
|
||||
blank=True,
|
||||
on_delete=models.CASCADE,
|
||||
related_name="posts",
|
||||
)
|
||||
post_id = models.IntegerField(db_index=True)
|
||||
data = models.JSONField(default=dict)
|
||||
seen = models.BooleanField(default=False, db_index=True)
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
|
||||
class Meta:
|
||||
ordering = ["-post_id"]
|
||||
indexes = [models.Index(fields=["user", "post_id"])]
|
||||
constraints = [
|
||||
models.UniqueConstraint(
|
||||
fields=["tag", "post_id"], name="uniq_followed_tag_post"
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["pool", "post_id"], name="uniq_followed_pool_post"
|
||||
),
|
||||
]
|
||||
|
||||
def __str__(self):
|
||||
return f"#{self.post_id} for {self.user}"
|
||||
|
||||
|
||||
class FollowCloud(models.Model):
|
||||
"""Cached blacklisted-tag cloud for the Followed page, built in a thread."""
|
||||
|
||||
user = models.OneToOneField(
|
||||
settings.AUTH_USER_MODEL,
|
||||
on_delete=models.CASCADE,
|
||||
related_name="follow_cloud",
|
||||
)
|
||||
data = models.JSONField(null=True, blank=True)
|
||||
computed_at = models.DateTimeField(null=True, blank=True)
|
||||
building = models.BooleanField(default=False)
|
||||
building_since = models.DateTimeField(null=True, blank=True)
|
||||
|
||||
def __str__(self):
|
||||
return f"Follow cloud for {self.user}"
|
||||
@@ -0,0 +1,53 @@
|
||||
from rest_framework import serializers
|
||||
|
||||
from .models import FollowedPool, FollowedTag
|
||||
|
||||
|
||||
class FollowedTagSerializer(serializers.ModelSerializer):
|
||||
unseen_count = serializers.IntegerField(read_only=True)
|
||||
|
||||
class Meta:
|
||||
model = FollowedTag
|
||||
fields = [
|
||||
"id",
|
||||
"tag",
|
||||
"unseen_count",
|
||||
"cover_data",
|
||||
"last_synced_at",
|
||||
"created_at",
|
||||
]
|
||||
read_only_fields = [
|
||||
"id",
|
||||
"tag",
|
||||
"unseen_count",
|
||||
"cover_data",
|
||||
"last_synced_at",
|
||||
"created_at",
|
||||
]
|
||||
|
||||
|
||||
class FollowedPoolSerializer(serializers.ModelSerializer):
|
||||
unseen_count = serializers.IntegerField(read_only=True)
|
||||
|
||||
class Meta:
|
||||
model = FollowedPool
|
||||
fields = [
|
||||
"id",
|
||||
"pool_id",
|
||||
"name",
|
||||
"post_count",
|
||||
"unseen_count",
|
||||
"cover_data",
|
||||
"last_synced_at",
|
||||
"created_at",
|
||||
]
|
||||
read_only_fields = [
|
||||
"id",
|
||||
"pool_id",
|
||||
"name",
|
||||
"post_count",
|
||||
"unseen_count",
|
||||
"cover_data",
|
||||
"last_synced_at",
|
||||
"created_at",
|
||||
]
|
||||
@@ -0,0 +1,171 @@
|
||||
"""Fetch and store followed tag/pool feeds from e621.
|
||||
|
||||
Used by the two periodic management commands (`sync_followed_tags`,
|
||||
`sync_followed_pools`) and by the follow API for the first fill of a new
|
||||
subscription. One e621 search per unique tag/pool is shared across every
|
||||
user following it.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.utils import timezone
|
||||
|
||||
from apps.library import e621
|
||||
from apps.library.services import trim_e621_post
|
||||
|
||||
from .models import FollowedPool, FollowedPost, FollowedTag
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
FEED_LIMIT = 24
|
||||
PRUNE_KEEP = 200
|
||||
|
||||
|
||||
def preferred_fetch_user(username=None):
|
||||
"""A configured account for sync calls; None means run anonymously.
|
||||
|
||||
e621 serves post searches and pool reads anonymously, so a missing
|
||||
account is not fatal — credentials mainly raise rate limits.
|
||||
"""
|
||||
User = get_user_model()
|
||||
if username:
|
||||
return User.objects.filter(username=username).first()
|
||||
return (
|
||||
User.objects.filter(is_staff=True)
|
||||
.exclude(e621_username="")
|
||||
.exclude(e621_api_key="")
|
||||
.order_by("id")
|
||||
.first()
|
||||
)
|
||||
|
||||
|
||||
def search_posts(user, term, limit=FEED_LIMIT):
|
||||
"""Newest posts for a search term (``tag`` or ``pool:<id>``)."""
|
||||
payload = e621.get(
|
||||
user,
|
||||
"/posts.json",
|
||||
params={"tags": term, "limit": limit},
|
||||
require_auth=False,
|
||||
)
|
||||
posts = payload.get("posts") if isinstance(payload, dict) else None
|
||||
return posts or []
|
||||
|
||||
|
||||
def cover_payload(post):
|
||||
"""Lightweight card cover: enough to render a thumbnail and rating."""
|
||||
if not isinstance(post, dict):
|
||||
return None
|
||||
preview = post.get("preview") or {}
|
||||
score = post.get("score") or {}
|
||||
return {
|
||||
"post_id": post.get("id"),
|
||||
"rating": post.get("rating"),
|
||||
"preview_url": preview.get("url"),
|
||||
"score": score.get("total"),
|
||||
"fav_count": post.get("fav_count"),
|
||||
}
|
||||
|
||||
|
||||
def feed_payload(post):
|
||||
"""Compact stored post for the feed (tags included for the tag cloud)."""
|
||||
trimmed = trim_e621_post(post)
|
||||
if trimmed is None:
|
||||
return None
|
||||
return {
|
||||
"id": trimmed.get("id"),
|
||||
"created_at": trimmed.get("created_at"),
|
||||
"rating": trimmed.get("rating"),
|
||||
"tags": trimmed.get("tags") or {},
|
||||
"score": trimmed.get("score") or {},
|
||||
"fav_count": trimmed.get("fav_count"),
|
||||
"file": trimmed.get("file") or {},
|
||||
"preview": trimmed.get("preview") or {},
|
||||
"uploader_name": trimmed.get("uploader_name"),
|
||||
}
|
||||
|
||||
|
||||
def _store_posts(follow, posts, tag=None, pool=None):
|
||||
"""Insert unseen rows for new posts. Returns how many were added."""
|
||||
existing = set(follow.posts.values_list("post_id", flat=True))
|
||||
rows = []
|
||||
for post in posts:
|
||||
post_id = post.get("id") if isinstance(post, dict) else None
|
||||
if not isinstance(post_id, int) or post_id in existing:
|
||||
continue
|
||||
data = feed_payload(post)
|
||||
if data is None:
|
||||
continue
|
||||
rows.append(
|
||||
FollowedPost(
|
||||
user=follow.user,
|
||||
post_id=post_id,
|
||||
data=data,
|
||||
tag=tag,
|
||||
pool=pool,
|
||||
)
|
||||
)
|
||||
if rows:
|
||||
FollowedPost.objects.bulk_create(rows, ignore_conflicts=True)
|
||||
_prune(follow)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def _prune(follow):
|
||||
"""Keep only the newest PRUNE_KEEP posts per follow."""
|
||||
keep = list(
|
||||
follow.posts.order_by("-post_id").values_list("id", flat=True)[
|
||||
:PRUNE_KEEP
|
||||
]
|
||||
)
|
||||
follow.posts.exclude(id__in=keep).delete()
|
||||
|
||||
|
||||
def sync_tag_follow(follow: FollowedTag, fetch_user=None, limit=FEED_LIMIT, posts=None):
|
||||
"""Fetch new posts for one followed tag; updates cover and sync time."""
|
||||
if posts is None:
|
||||
posts = search_posts(fetch_user, follow.tag, limit)
|
||||
added = _store_posts(follow, posts, tag=follow)
|
||||
updates = ["last_synced_at"]
|
||||
cover = cover_payload(posts[0]) if posts else None
|
||||
if cover is not None:
|
||||
follow.cover_data = cover
|
||||
updates.append("cover_data")
|
||||
follow.last_synced_at = timezone.now()
|
||||
follow.save(update_fields=updates)
|
||||
return added
|
||||
|
||||
|
||||
def sync_pool_follow(
|
||||
follow: FollowedPool,
|
||||
fetch_user=None,
|
||||
limit=FEED_LIMIT,
|
||||
posts=None,
|
||||
refresh_meta=True,
|
||||
):
|
||||
"""Fetch new posts for one followed pool; updates cover and metadata."""
|
||||
if posts is None:
|
||||
posts = search_posts(fetch_user, f"pool:{follow.pool_id}", limit)
|
||||
added = _store_posts(follow, posts, pool=follow)
|
||||
updates = ["last_synced_at"]
|
||||
cover = cover_payload(posts[0]) if posts else None
|
||||
if cover is not None:
|
||||
follow.cover_data = cover
|
||||
updates.append("cover_data")
|
||||
if refresh_meta or not follow.name:
|
||||
try:
|
||||
pool = e621.get(
|
||||
fetch_user,
|
||||
f"/pools/{follow.pool_id}.json",
|
||||
require_auth=False,
|
||||
)
|
||||
except e621.E621Error as exc:
|
||||
logger.warning("Could not refresh pool %s: %s", follow.pool_id, exc)
|
||||
else:
|
||||
if isinstance(pool, dict) and pool.get("id"):
|
||||
follow.name = str(pool.get("name") or "")[:200]
|
||||
follow.post_count = int(pool.get("post_count") or 0)
|
||||
updates += ["name", "post_count"]
|
||||
follow.last_synced_at = timezone.now()
|
||||
follow.save(update_fields=updates)
|
||||
return added
|
||||
@@ -0,0 +1,25 @@
|
||||
from django.urls import include, path
|
||||
from rest_framework.routers import DefaultRouter
|
||||
|
||||
from .views import (
|
||||
FollowCloudView,
|
||||
FollowedFeedView,
|
||||
FollowedPoolViewSet,
|
||||
FollowedPostSeenView,
|
||||
FollowedTagViewSet,
|
||||
)
|
||||
|
||||
router = DefaultRouter()
|
||||
router.register("follows/tags", FollowedTagViewSet, basename="followed-tag")
|
||||
router.register("follows/pools", FollowedPoolViewSet, basename="followed-pool")
|
||||
|
||||
urlpatterns = [
|
||||
path("", include(router.urls)),
|
||||
path("follows/feed/", FollowedFeedView.as_view(), name="followed_feed"),
|
||||
path("follows/cloud/", FollowCloudView.as_view(), name="followed_cloud"),
|
||||
path(
|
||||
"follows/posts/<int:post_id>/seen/",
|
||||
FollowedPostSeenView.as_view(),
|
||||
name="followed_post_seen",
|
||||
),
|
||||
]
|
||||
@@ -0,0 +1,317 @@
|
||||
import re
|
||||
|
||||
from django.db.models import Count, Q
|
||||
from django.utils import timezone
|
||||
from rest_framework import mixins, status, viewsets
|
||||
from rest_framework.decorators import action
|
||||
from rest_framework.permissions import IsAuthenticated
|
||||
from rest_framework.response import Response
|
||||
from rest_framework.views import APIView
|
||||
|
||||
from apps.library import e621
|
||||
|
||||
from . import cloud as cloud_service
|
||||
from . import sync
|
||||
from .models import FollowCloud, FollowedPool, FollowedPost, FollowedTag
|
||||
from .serializers import FollowedPoolSerializer, FollowedTagSerializer
|
||||
|
||||
POOL_URL_RE = re.compile(r"/pools/(\d+)")
|
||||
|
||||
|
||||
def _parse_page(request, default_per_page=48):
|
||||
try:
|
||||
per_page = int(request.query_params.get("per_page", default_per_page))
|
||||
except ValueError:
|
||||
per_page = default_per_page
|
||||
try:
|
||||
page = int(request.query_params.get("page", 1))
|
||||
except ValueError:
|
||||
page = 1
|
||||
return max(1, page), min(200, max(1, per_page))
|
||||
|
||||
|
||||
class FollowedTagViewSet(
|
||||
mixins.ListModelMixin,
|
||||
mixins.CreateModelMixin,
|
||||
mixins.DestroyModelMixin,
|
||||
viewsets.GenericViewSet,
|
||||
):
|
||||
"""Tags the current user follows."""
|
||||
|
||||
serializer_class = FollowedTagSerializer
|
||||
permission_classes = [IsAuthenticated]
|
||||
http_method_names = ["get", "post", "delete", "head", "options"]
|
||||
|
||||
def get_queryset(self):
|
||||
return (
|
||||
FollowedTag.objects.filter(user=self.request.user)
|
||||
.annotate(
|
||||
unseen_count=Count("posts", filter=Q(posts__seen=False), distinct=True)
|
||||
)
|
||||
.order_by("tag")
|
||||
)
|
||||
|
||||
def create(self, request):
|
||||
tag = str(request.data.get("tag") or "").strip().lower()[:200]
|
||||
if not tag:
|
||||
return Response(
|
||||
{"detail": "A tag is required."}, status=status.HTTP_400_BAD_REQUEST
|
||||
)
|
||||
if FollowedTag.objects.filter(user=request.user, tag=tag).exists():
|
||||
return Response(
|
||||
{"detail": f"Already following '{tag}'."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
try:
|
||||
results = e621.get(
|
||||
request.user,
|
||||
"/tags.json",
|
||||
params={"search[name_matches]": tag, "limit": 5},
|
||||
require_auth=False,
|
||||
)
|
||||
except e621.E621Error as exc:
|
||||
return Response(
|
||||
{"detail": str(exc)}, status=status.HTTP_502_BAD_GATEWAY
|
||||
)
|
||||
names = {
|
||||
str(entry.get("name", "")).lower()
|
||||
for entry in results
|
||||
if isinstance(entry, dict)
|
||||
}
|
||||
if tag not in names:
|
||||
return Response(
|
||||
{"detail": f"e621 has no tag named '{tag}'."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
follow = FollowedTag.objects.create(user=request.user, tag=tag)
|
||||
FollowCloud.objects.filter(user=request.user).update(computed_at=None)
|
||||
try:
|
||||
sync.sync_tag_follow(follow, request.user)
|
||||
except e621.E621Error:
|
||||
# The periodic command will catch up; the follow still stands.
|
||||
pass
|
||||
follow = self.get_queryset().get(pk=follow.pk)
|
||||
return Response(
|
||||
self.get_serializer(follow).data, status=status.HTTP_201_CREATED
|
||||
)
|
||||
|
||||
def perform_destroy(self, instance):
|
||||
instance.delete()
|
||||
FollowCloud.objects.filter(user=self.request.user).update(computed_at=None)
|
||||
|
||||
@action(detail=True, methods=["post"])
|
||||
def seen(self, request, pk=None):
|
||||
follow = self.get_object()
|
||||
follow.posts.update(seen=True)
|
||||
return Response({"success": True})
|
||||
|
||||
|
||||
class FollowedPoolViewSet(
|
||||
mixins.ListModelMixin,
|
||||
mixins.CreateModelMixin,
|
||||
mixins.DestroyModelMixin,
|
||||
viewsets.GenericViewSet,
|
||||
):
|
||||
"""Pools the current user follows."""
|
||||
|
||||
serializer_class = FollowedPoolSerializer
|
||||
permission_classes = [IsAuthenticated]
|
||||
http_method_names = ["get", "post", "delete", "head", "options"]
|
||||
|
||||
def get_queryset(self):
|
||||
return (
|
||||
FollowedPool.objects.filter(user=self.request.user)
|
||||
.annotate(
|
||||
unseen_count=Count("posts", filter=Q(posts__seen=False), distinct=True)
|
||||
)
|
||||
.order_by("pool_id")
|
||||
)
|
||||
|
||||
def create(self, request):
|
||||
raw = str(
|
||||
request.data.get("pool") or request.data.get("pool_id") or ""
|
||||
).strip()
|
||||
match = POOL_URL_RE.search(raw)
|
||||
if match:
|
||||
pool_id = int(match.group(1))
|
||||
elif raw.isdigit():
|
||||
pool_id = int(raw)
|
||||
else:
|
||||
return Response(
|
||||
{"detail": "Enter an e621 pool ID or pool URL."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
if FollowedPool.objects.filter(user=request.user, pool_id=pool_id).exists():
|
||||
return Response(
|
||||
{"detail": f"Already following pool #{pool_id}."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
try:
|
||||
pool = e621.get(
|
||||
request.user, f"/pools/{pool_id}.json", require_auth=False
|
||||
)
|
||||
except e621.E621NotFound:
|
||||
return Response(
|
||||
{"detail": f"e621 has no pool #{pool_id}."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
except e621.E621Error as exc:
|
||||
return Response(
|
||||
{"detail": str(exc)}, status=status.HTTP_502_BAD_GATEWAY
|
||||
)
|
||||
follow = FollowedPool.objects.create(
|
||||
user=request.user,
|
||||
pool_id=pool_id,
|
||||
name=str(pool.get("name") or "")[:200],
|
||||
post_count=int(pool.get("post_count") or 0),
|
||||
)
|
||||
FollowCloud.objects.filter(user=request.user).update(computed_at=None)
|
||||
try:
|
||||
sync.sync_pool_follow(follow, request.user, refresh_meta=False)
|
||||
except e621.E621Error:
|
||||
pass
|
||||
follow = self.get_queryset().get(pk=follow.pk)
|
||||
return Response(
|
||||
self.get_serializer(follow).data, status=status.HTTP_201_CREATED
|
||||
)
|
||||
|
||||
def perform_destroy(self, instance):
|
||||
instance.delete()
|
||||
FollowCloud.objects.filter(user=self.request.user).update(computed_at=None)
|
||||
|
||||
@action(detail=True, methods=["post"])
|
||||
def seen(self, request, pk=None):
|
||||
follow = self.get_object()
|
||||
follow.posts.update(seen=True)
|
||||
return Response({"success": True})
|
||||
|
||||
|
||||
class FollowedFeedView(APIView):
|
||||
"""Merged newest-first feed across followed tags and pools."""
|
||||
|
||||
permission_classes = [IsAuthenticated]
|
||||
|
||||
def get(self, request):
|
||||
page, per_page = _parse_page(request)
|
||||
queryset = FollowedPost.objects.filter(user=request.user).order_by(
|
||||
"-post_id"
|
||||
)
|
||||
tag_id = request.query_params.get("tag", "")
|
||||
pool_id = request.query_params.get("pool", "")
|
||||
if tag_id.isdigit():
|
||||
queryset = queryset.filter(tag_id=int(tag_id))
|
||||
if pool_id.isdigit():
|
||||
queryset = queryset.filter(pool_id=int(pool_id))
|
||||
if request.query_params.get("unseen") == "1":
|
||||
queryset = queryset.filter(seen=False)
|
||||
|
||||
tag_names = dict(
|
||||
FollowedTag.objects.filter(user=request.user).values_list("id", "tag")
|
||||
)
|
||||
pool_names = dict(
|
||||
FollowedPool.objects.filter(user=request.user).values_list(
|
||||
"id", "name"
|
||||
)
|
||||
)
|
||||
entries = {}
|
||||
for post_id, seen, data, tag, pool in queryset.values_list(
|
||||
"post_id", "seen", "data", "tag_id", "pool_id"
|
||||
)[:2000]:
|
||||
entry = entries.get(post_id)
|
||||
if entry is None:
|
||||
entry = entries[post_id] = {
|
||||
"post_id": post_id,
|
||||
"seen": seen,
|
||||
"data": data,
|
||||
"sources": [],
|
||||
}
|
||||
else:
|
||||
entry["seen"] = entry["seen"] and seen
|
||||
if tag and not any(
|
||||
source["kind"] == "tag" and source["id"] == tag
|
||||
for source in entry["sources"]
|
||||
):
|
||||
entry["sources"].append(
|
||||
{
|
||||
"kind": "tag",
|
||||
"id": tag,
|
||||
"label": tag_names.get(tag, f"tag:{tag}"),
|
||||
}
|
||||
)
|
||||
if pool and not any(
|
||||
source["kind"] == "pool" and source["id"] == pool
|
||||
for source in entry["sources"]
|
||||
):
|
||||
entry["sources"].append(
|
||||
{
|
||||
"kind": "pool",
|
||||
"id": pool,
|
||||
"label": pool_names.get(pool) or f"Pool #{pool}",
|
||||
}
|
||||
)
|
||||
|
||||
items = list(entries.values())
|
||||
count = len(items)
|
||||
start = (page - 1) * per_page
|
||||
return Response(
|
||||
{
|
||||
"count": count,
|
||||
"page": page,
|
||||
"per_page": per_page,
|
||||
"has_next": start + per_page < count,
|
||||
"results": items[start : start + per_page],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class FollowedPostSeenView(APIView):
|
||||
"""Mark one post seen across every follow it appears in."""
|
||||
|
||||
permission_classes = [IsAuthenticated]
|
||||
|
||||
def post(self, request, post_id):
|
||||
FollowedPost.objects.filter(user=request.user, post_id=post_id).update(
|
||||
seen=True
|
||||
)
|
||||
return Response({"success": True})
|
||||
|
||||
|
||||
class FollowCloudView(APIView):
|
||||
"""The blacklisted-tag cloud, rebuilt in a thread when stale."""
|
||||
|
||||
permission_classes = [IsAuthenticated]
|
||||
|
||||
def get(self, request):
|
||||
cloud, _ = FollowCloud.objects.get_or_create(user=request.user)
|
||||
now = timezone.now()
|
||||
stale = cloud.computed_at is None or now - cloud.computed_at > (
|
||||
cloud_service.CLOUD_TTL
|
||||
)
|
||||
if stale:
|
||||
reclaim = Q(building=False) | Q(
|
||||
building_since__lt=now - cloud_service.BUILD_STALE
|
||||
)
|
||||
started = (
|
||||
FollowCloud.objects.filter(pk=cloud.pk)
|
||||
.filter(reclaim)
|
||||
.update(building=True, building_since=now)
|
||||
)
|
||||
if started:
|
||||
cloud_service.start_cloud_build(request.user.id)
|
||||
cloud.refresh_from_db()
|
||||
building = cloud.building and not (
|
||||
cloud.building_since
|
||||
and now - cloud.building_since > cloud_service.BUILD_STALE
|
||||
)
|
||||
data = cloud.data or {}
|
||||
return Response(
|
||||
{
|
||||
"status": "building" if building else "ready",
|
||||
"stale": stale,
|
||||
"tags": data.get("tags", []),
|
||||
"posts": data.get("posts", 0),
|
||||
"source": data.get("source"),
|
||||
"blacklist_count": data.get("blacklist_count", 0),
|
||||
"computed_at": cloud.computed_at,
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,157 @@
|
||||
"""Background 'Download to Library' jobs with progress tracking."""
|
||||
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from django.conf import settings
|
||||
from django.db import connection
|
||||
from django.utils import timezone
|
||||
|
||||
from . import services
|
||||
from .models import DownloadTask
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
STALE_AFTER = timedelta(minutes=30)
|
||||
|
||||
|
||||
def reap_stale_downloads():
|
||||
"""Mark tasks left hanging by a recycled worker as failed.
|
||||
|
||||
Gunicorn recycles workers (--max-requests, timeouts); a download thread
|
||||
dies with its worker, so long-stuck tasks are surfaced as errors instead
|
||||
of pretending to run forever.
|
||||
"""
|
||||
cutoff = timezone.now() - STALE_AFTER
|
||||
return DownloadTask.objects.filter(
|
||||
status__in=[DownloadTask.STATUS_PENDING, DownloadTask.STATUS_DOWNLOADING],
|
||||
updated_at__lt=cutoff,
|
||||
).update(
|
||||
status=DownloadTask.STATUS_ERROR,
|
||||
error="The worker restarted before this download finished.",
|
||||
speed=None,
|
||||
updated_at=timezone.now(),
|
||||
)
|
||||
|
||||
|
||||
def start_download_task(task_id):
|
||||
thread = threading.Thread(target=run_download_task, args=(task_id,), daemon=True)
|
||||
thread.start()
|
||||
|
||||
|
||||
def run_download_task(task_id):
|
||||
task = DownloadTask.objects.filter(id=task_id).first()
|
||||
if task is None:
|
||||
return
|
||||
|
||||
folder = Path(settings.WATCHED_FOLDER)
|
||||
name = (
|
||||
task.filename
|
||||
or Path(urlparse(task.url).path).name
|
||||
or f"post-{task.post_id or 'download'}"
|
||||
)
|
||||
destination = services.unique_destination(folder, name)
|
||||
|
||||
state = {
|
||||
"progress_at": 0.0,
|
||||
"bytes_at": 0,
|
||||
"cancel_at": 0.0,
|
||||
"cancelled": False,
|
||||
}
|
||||
|
||||
def should_cancel():
|
||||
now = time.monotonic()
|
||||
if now - state["cancel_at"] >= 1.0:
|
||||
state["cancelled"] = DownloadTask.objects.filter(
|
||||
id=task_id, cancelled=True
|
||||
).exists()
|
||||
state["cancel_at"] = now
|
||||
return state["cancelled"]
|
||||
|
||||
def on_progress(downloaded, total):
|
||||
now = time.monotonic()
|
||||
if now - state["progress_at"] < 0.5 and (total == 0 or downloaded < total):
|
||||
return
|
||||
elapsed = max(now - state["progress_at"], 0.001)
|
||||
speed = None
|
||||
if state["bytes_at"] > 0 and downloaded >= state["bytes_at"]:
|
||||
speed = (downloaded - state["bytes_at"]) / elapsed
|
||||
state["progress_at"] = now
|
||||
state["bytes_at"] = downloaded
|
||||
DownloadTask.objects.filter(id=task_id).update(
|
||||
downloaded=downloaded,
|
||||
total=total,
|
||||
progress=int(downloaded * 100 / total) if total else 0,
|
||||
speed=speed,
|
||||
updated_at=timezone.now(),
|
||||
)
|
||||
|
||||
DownloadTask.objects.filter(id=task_id).update(
|
||||
status=DownloadTask.STATUS_DOWNLOADING, updated_at=timezone.now()
|
||||
)
|
||||
try:
|
||||
services.download_file(
|
||||
task.url,
|
||||
destination,
|
||||
progress_callback=on_progress,
|
||||
should_cancel=should_cancel,
|
||||
)
|
||||
item, _, location, _ = services.index_file(destination, folder)
|
||||
services.rename_location_to_j_id(item, location)
|
||||
services.ensure_visual_hashes(item)
|
||||
|
||||
update_fields = []
|
||||
if item.uploaded_by_id is None and task.user_id is not None:
|
||||
item.uploaded_by = task.user
|
||||
update_fields.append("uploaded_by")
|
||||
if task.e621_data:
|
||||
item.e621_post_id = task.post_id
|
||||
item.e621_data = task.e621_data
|
||||
item.e621_match_status = MediaItem.E621_MATCHED
|
||||
item.e621_checked_at = timezone.now()
|
||||
update_fields += [
|
||||
"e621_post_id",
|
||||
"e621_data",
|
||||
"e621_match_status",
|
||||
"e621_checked_at",
|
||||
]
|
||||
rating = (task.e621_data or {}).get("rating")
|
||||
if not item.rating and rating in {"s", "q", "e"}:
|
||||
item.rating = rating
|
||||
update_fields.append("rating")
|
||||
if update_fields:
|
||||
item.save(update_fields=update_fields + ["updated_at"])
|
||||
|
||||
DownloadTask.objects.filter(id=task_id).update(
|
||||
status=DownloadTask.STATUS_COMPLETE,
|
||||
progress=100,
|
||||
speed=None,
|
||||
library_item=item,
|
||||
updated_at=timezone.now(),
|
||||
)
|
||||
except services.DownloadCancelled:
|
||||
destination.unlink(missing_ok=True)
|
||||
DownloadTask.objects.filter(id=task_id).update(
|
||||
status=DownloadTask.STATUS_CANCELLED,
|
||||
progress=0,
|
||||
downloaded=0,
|
||||
speed=None,
|
||||
updated_at=timezone.now(),
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - report background failures
|
||||
destination.unlink(missing_ok=True)
|
||||
logger.exception("Download task %s failed", task_id)
|
||||
DownloadTask.objects.filter(id=task_id).update(
|
||||
status=DownloadTask.STATUS_ERROR,
|
||||
error=str(exc),
|
||||
speed=None,
|
||||
updated_at=timezone.now(),
|
||||
)
|
||||
finally:
|
||||
# Background threads hold their own DB connection; release it so
|
||||
# Gunicorn workers do not leak connections when threads finish.
|
||||
connection.close()
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Minimal e621 API client for server-side matching and metadata refresh.
|
||||
|
||||
The SPA talks to e621 directly for browsing; this client exists for work the
|
||||
browser cannot do reliably: long batch scans, and requests tied to a library
|
||||
item rather than an open page. It uses the requesting user's stored
|
||||
credentials and a global throttle (e621 asks for at most two requests per
|
||||
second).
|
||||
"""
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import requests
|
||||
from django.conf import settings
|
||||
|
||||
REQUEST_INTERVAL = 0.5 # seconds between requests, per process
|
||||
|
||||
|
||||
class E621Error(Exception):
|
||||
"""A non-404 failure while talking to e621."""
|
||||
|
||||
|
||||
class E621NotFound(E621Error):
|
||||
"""The requested post does not exist (HTTP 404)."""
|
||||
|
||||
|
||||
_throttle_lock = threading.Lock()
|
||||
_last_request_at = 0.0
|
||||
|
||||
|
||||
def credentials_configured(user):
|
||||
return bool(user is not None and getattr(user, "e621_configured", False))
|
||||
|
||||
|
||||
def _wait_for_slot():
|
||||
global _last_request_at
|
||||
with _throttle_lock:
|
||||
delay = _last_request_at + REQUEST_INTERVAL - time.monotonic()
|
||||
if delay > 0:
|
||||
time.sleep(delay)
|
||||
_last_request_at = time.monotonic()
|
||||
|
||||
|
||||
def get(user, path, params=None, timeout=30, require_auth=True):
|
||||
"""GET an e621 API path using the user's credentials.
|
||||
|
||||
Reads that e621 serves anonymously (searches, pools, tags) can pass
|
||||
require_auth=False; matching endpoints keep requiring credentials.
|
||||
|
||||
Raises E621NotFound for 404s and E621Error for everything else that isn't
|
||||
a 2xx, so callers never see requests exceptions.
|
||||
"""
|
||||
configured = credentials_configured(user)
|
||||
if require_auth and not configured:
|
||||
raise E621Error("Configure your e621 credentials in Account first.")
|
||||
base = (getattr(user, "e621_base_url", "") or "https://e621.net").rstrip("/")
|
||||
_wait_for_slot()
|
||||
try:
|
||||
response = requests.get(
|
||||
f"{base}{path}",
|
||||
params=params,
|
||||
auth=(user.e621_username, user.e621_api_key) if configured else None,
|
||||
headers={"User-Agent": settings.USER_AGENT},
|
||||
timeout=timeout,
|
||||
)
|
||||
except requests.RequestException as exc:
|
||||
raise E621Error(f"Could not reach e621: {exc}") from exc
|
||||
if response.status_code == 404:
|
||||
raise E621NotFound(f"e621 returned 404 for {path}")
|
||||
if response.status_code >= 400:
|
||||
raise E621Error(f"e621 replied {response.status_code} for {path}")
|
||||
try:
|
||||
return response.json()
|
||||
except ValueError as exc:
|
||||
raise E621Error("e621 returned an unexpected response.") from exc
|
||||
|
||||
|
||||
def find_post_by_md5(user, md5):
|
||||
"""The e621 post with this exact MD5, or None."""
|
||||
payload = get(user, "/posts.json", params={"tags": f"md5:{md5}", "limit": 1})
|
||||
posts = payload.get("posts") if isinstance(payload, dict) else None
|
||||
if not posts:
|
||||
return None
|
||||
return posts[0]
|
||||
|
||||
|
||||
def fetch_post(user, post_id):
|
||||
"""One post by id. Raises E621NotFound when the post is gone."""
|
||||
payload = get(user, f"/posts/{int(post_id)}.json")
|
||||
post = payload.get("post") if isinstance(payload, dict) else None
|
||||
if not isinstance(post, dict):
|
||||
raise E621Error("e621 returned an unexpected post payload.")
|
||||
return post
|
||||
@@ -20,6 +20,7 @@ from django.core.cache import cache
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CACHE_KEY = "j621.guest_blacklist"
|
||||
MIRROR_KEY = "j621.guest_blacklist.mirror"
|
||||
|
||||
|
||||
def parse_blacklist_patterns(lines):
|
||||
@@ -37,10 +38,18 @@ def parse_blacklist_patterns(lines):
|
||||
|
||||
|
||||
def cached_guest_blacklist():
|
||||
"""Raw mirrored blacklist lines. Never hits the network."""
|
||||
"""Raw mirrored blacklist lines. Never hits the network.
|
||||
|
||||
Prefers the TTL'd entry, falls back to the persistent mirror (last
|
||||
successful e621 fetch) and only then to the local fallback list, so an
|
||||
expired TTL never degrades filtering.
|
||||
"""
|
||||
stored = cache.get(CACHE_KEY)
|
||||
if stored is not None:
|
||||
return list(stored)
|
||||
mirror = cache.get(MIRROR_KEY)
|
||||
if mirror is not None:
|
||||
return list(mirror)
|
||||
return list(settings.GUEST_BLACKLIST_FALLBACK)
|
||||
|
||||
|
||||
@@ -101,7 +110,11 @@ def refresh_guest_blacklist():
|
||||
lines = fetch_e621_default_blacklist()
|
||||
used_fallback = lines is None
|
||||
if used_fallback:
|
||||
lines = list(settings.GUEST_BLACKLIST_FALLBACK)
|
||||
# Keep the last successful mirror (or the local list) rather than
|
||||
# degrading guest filtering when e621 is unreachable.
|
||||
lines = cache.get(MIRROR_KEY) or list(settings.GUEST_BLACKLIST_FALLBACK)
|
||||
else:
|
||||
cache.set(MIRROR_KEY, lines, timeout=None)
|
||||
cache.set(CACHE_KEY, lines, settings.GUEST_BLACKLIST_TTL)
|
||||
|
||||
patterns = parse_blacklist_patterns(lines)
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
from datetime import timedelta
|
||||
|
||||
from django.core.management.base import BaseCommand
|
||||
from django.utils import timezone
|
||||
|
||||
from apps.library.models import TempUpload
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Delete staged uploads older than N hours (default 24)."
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument("--hours", type=int, default=24)
|
||||
parser.add_argument(
|
||||
"--include-completed",
|
||||
action="store_true",
|
||||
help="Also remove completed staging records.",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
cutoff = timezone.now() - timedelta(hours=options["hours"])
|
||||
queryset = TempUpload.objects.filter(created_at__lt=cutoff)
|
||||
if not options["include_completed"]:
|
||||
queryset = queryset.exclude(status=TempUpload.STATUS_COMPLETED)
|
||||
|
||||
removed = 0
|
||||
for temp in queryset:
|
||||
if temp.file:
|
||||
temp.file.delete(save=False)
|
||||
temp.delete()
|
||||
removed += 1
|
||||
|
||||
self.stdout.write(
|
||||
self.style.SUCCESS(f"Removed {removed} staged upload(s).")
|
||||
)
|
||||
@@ -0,0 +1,43 @@
|
||||
from django.core.management.base import BaseCommand
|
||||
from django.db.models import Q
|
||||
|
||||
from apps.library.models import MediaItem
|
||||
from apps.library.services import HASH_FIELDS, ensure_visual_hashes
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Compute perceptual hashes for library items missing them."
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="Recompute hashes even when they already exist.",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
queryset = MediaItem.objects.prefetch_related("locations").order_by("id")
|
||||
if not options["force"]:
|
||||
missing = Q()
|
||||
for field in HASH_FIELDS:
|
||||
missing |= Q(**{field: ""})
|
||||
queryset = queryset.filter(missing)
|
||||
|
||||
total = queryset.count()
|
||||
if total == 0:
|
||||
self.stdout.write(self.style.SUCCESS("All items already have hashes."))
|
||||
return
|
||||
|
||||
self.stdout.write(f"Computing hashes for {total} item(s) ...")
|
||||
done = 0
|
||||
for item in queryset.iterator(chunk_size=100):
|
||||
if options["force"]:
|
||||
item.ahash = item.dhash = item.phash = item.whash = ""
|
||||
ensure_visual_hashes(item)
|
||||
done += 1
|
||||
if done % 100 == 0:
|
||||
self.stdout.write(f"Processed {done} ...")
|
||||
|
||||
self.stdout.write(
|
||||
self.style.SUCCESS(f"Done. Processed {done} item(s).")
|
||||
)
|
||||
@@ -0,0 +1,70 @@
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.core.management.base import BaseCommand, CommandError
|
||||
|
||||
from apps.library import e621, matching
|
||||
from apps.library.models import MatchTask, MediaItem
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = (
|
||||
"Match library items against e621 by MD5 and refresh matched metadata "
|
||||
"(also detects deleted posts). Uses a user's stored e621 credentials."
|
||||
)
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--user",
|
||||
help="Username whose e621 credentials to use "
|
||||
"(defaults to the first staff account with credentials).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--scope",
|
||||
choices=[MatchTask.SCOPE_MISSING, MatchTask.SCOPE_ALL],
|
||||
default=MatchTask.SCOPE_MISSING,
|
||||
help="'missing' checks unchecked items only; 'all' also refreshes "
|
||||
"already matched items (default: missing).",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
User = get_user_model()
|
||||
username = options.get("user")
|
||||
if username:
|
||||
user = User.objects.filter(username=username).first()
|
||||
if user is None:
|
||||
raise CommandError(f"No such user: {username}")
|
||||
else:
|
||||
user = (
|
||||
User.objects.filter(
|
||||
is_staff=True, e621_api_key__gt="", e621_username__gt=""
|
||||
)
|
||||
.order_by("id")
|
||||
.first()
|
||||
)
|
||||
if not e621.credentials_configured(user):
|
||||
raise CommandError(
|
||||
"No e621 credentials found. Pass --user or configure them in Account."
|
||||
)
|
||||
|
||||
queryset = matching.scope_queryset(options["scope"])
|
||||
total = queryset.count()
|
||||
counters = {"matched": 0, "not_found": 0, "deleted": 0}
|
||||
for index, item_id in enumerate(queryset.values_list("id", flat=True), start=1):
|
||||
item = MediaItem.objects.filter(id=item_id).first()
|
||||
if item is None:
|
||||
continue
|
||||
try:
|
||||
status = matching.check_item(item, user)
|
||||
except e621.E621Error as exc:
|
||||
raise CommandError(f"Aborted after {index - 1}/{total}: {exc}")
|
||||
counters[status] = counters.get(status, 0) + 1
|
||||
self.stdout.write(
|
||||
f"[{index}/{total}] J-{item_id}: {status}", ending="\r"
|
||||
)
|
||||
|
||||
self.stdout.write(
|
||||
self.style.SUCCESS(
|
||||
"\nDone: "
|
||||
f"{counters['matched']} matched, {counters['not_found']} not found, "
|
||||
f"{counters['deleted']} deleted (scope: {options['scope']})."
|
||||
)
|
||||
)
|
||||
@@ -67,7 +67,9 @@ class Command(BaseCommand):
|
||||
continue
|
||||
path = Path(root) / name
|
||||
try:
|
||||
item, created_item, created_location = index_file(path, folder)
|
||||
item, created_item, _location, created_location = index_file(
|
||||
path, folder
|
||||
)
|
||||
except OSError as exc:
|
||||
skipped += 1
|
||||
self.stderr.write(f"Skipped {path}: {exc}")
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
"""Server-side matching of library items against e621.
|
||||
|
||||
MD5 lookups, manual post links, metadata refresh (which is also how deleted
|
||||
posts are detected) and the background library-wide scan.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from datetime import timedelta
|
||||
|
||||
from django.db import connection
|
||||
from django.utils import timezone
|
||||
|
||||
from . import e621
|
||||
from .models import MatchTask, MediaItem
|
||||
from .services import trim_e621_post
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
STALE_AFTER = timedelta(minutes=30)
|
||||
RATING_VALUES = {"s", "q", "e"}
|
||||
|
||||
|
||||
# --- single-item operations -------------------------------------------------
|
||||
|
||||
|
||||
def apply_post_to_item(item, post):
|
||||
"""Store trimmed e621 metadata on an item and mark it matched."""
|
||||
trimmed = trim_e621_post(post)
|
||||
if trimmed is None or not trimmed.get("id"):
|
||||
raise e621.E621Error("e621 returned an unexpected post payload.")
|
||||
item.e621_post_id = int(trimmed["id"])
|
||||
item.e621_data = trimmed
|
||||
item.e621_match_status = MediaItem.E621_MATCHED
|
||||
item.e621_checked_at = timezone.now()
|
||||
update_fields = [
|
||||
"e621_post_id",
|
||||
"e621_data",
|
||||
"e621_match_status",
|
||||
"e621_checked_at",
|
||||
]
|
||||
rating = trimmed.get("rating")
|
||||
if not item.rating and rating in RATING_VALUES:
|
||||
item.rating = rating
|
||||
update_fields.append("rating")
|
||||
item.save(update_fields=update_fields + ["updated_at"])
|
||||
return trimmed
|
||||
|
||||
|
||||
def mark_not_found(item):
|
||||
item.e621_match_status = MediaItem.E621_NOT_FOUND
|
||||
item.e621_checked_at = timezone.now()
|
||||
item.save(update_fields=["e621_match_status", "e621_checked_at", "updated_at"])
|
||||
|
||||
|
||||
def mark_deleted(item):
|
||||
item.e621_match_status = MediaItem.E621_DELETED
|
||||
item.e621_checked_at = timezone.now()
|
||||
item.save(update_fields=["e621_match_status", "e621_checked_at", "updated_at"])
|
||||
|
||||
|
||||
def match_item_by_md5(item, user):
|
||||
"""Look the item's MD5 up on e621. Returns the resulting match status."""
|
||||
post = e621.find_post_by_md5(user, item.md5)
|
||||
if post is None:
|
||||
mark_not_found(item)
|
||||
return item.e621_match_status
|
||||
apply_post_to_item(item, post)
|
||||
return item.e621_match_status
|
||||
|
||||
|
||||
def refresh_item(item, user):
|
||||
"""Re-fetch a matched post: refresh metadata or flag it as deleted."""
|
||||
if not item.e621_post_id:
|
||||
return match_item_by_md5(item, user)
|
||||
try:
|
||||
post = e621.fetch_post(user, item.e621_post_id)
|
||||
except e621.E621NotFound:
|
||||
mark_deleted(item)
|
||||
return item.e621_match_status
|
||||
apply_post_to_item(item, post)
|
||||
return item.e621_match_status
|
||||
|
||||
|
||||
def link_item_to_post(item, post_id, user):
|
||||
"""Manually link an item to a post id. Returns True if the MD5s match."""
|
||||
post = e621.fetch_post(user, post_id)
|
||||
trimmed = apply_post_to_item(item, post)
|
||||
post_md5 = str((trimmed.get("file") or {}).get("md5") or "").strip().lower()
|
||||
return bool(post_md5) and post_md5 == item.md5.lower()
|
||||
|
||||
|
||||
def check_item(item, user):
|
||||
"""Best check for this item: refresh a known link, else MD5 lookup."""
|
||||
if item.e621_post_id and item.e621_match_status == MediaItem.E621_MATCHED:
|
||||
return refresh_item(item, user)
|
||||
return match_item_by_md5(item, user)
|
||||
|
||||
|
||||
def unlink_item(item):
|
||||
"""Forget the e621 link so the item can be checked again."""
|
||||
item.e621_post_id = None
|
||||
item.e621_data = None
|
||||
item.e621_match_status = MediaItem.E621_UNKNOWN
|
||||
item.e621_checked_at = None
|
||||
item.save(
|
||||
update_fields=[
|
||||
"e621_post_id",
|
||||
"e621_data",
|
||||
"e621_match_status",
|
||||
"e621_checked_at",
|
||||
"updated_at",
|
||||
]
|
||||
)
|
||||
return item
|
||||
|
||||
|
||||
# --- batch scan -------------------------------------------------------------
|
||||
|
||||
|
||||
def reap_stale_match_tasks():
|
||||
"""Surface scans left hanging by a recycled worker as errors."""
|
||||
cutoff = timezone.now() - STALE_AFTER
|
||||
return MatchTask.objects.filter(
|
||||
status__in=[MatchTask.STATUS_PENDING, MatchTask.STATUS_RUNNING],
|
||||
updated_at__lt=cutoff,
|
||||
).update(
|
||||
status=MatchTask.STATUS_ERROR,
|
||||
error="The worker restarted before this scan finished.",
|
||||
updated_at=timezone.now(),
|
||||
)
|
||||
|
||||
|
||||
def scope_queryset(scope):
|
||||
queryset = MediaItem.objects.order_by("id")
|
||||
if scope == MatchTask.SCOPE_ALL:
|
||||
return queryset
|
||||
return queryset.filter(e621_match_status=MediaItem.E621_UNKNOWN)
|
||||
|
||||
|
||||
def start_match_task(task_id):
|
||||
thread = threading.Thread(target=run_match_task, args=(task_id,), daemon=True)
|
||||
thread.start()
|
||||
|
||||
|
||||
def run_match_task(task_id):
|
||||
task = MatchTask.objects.filter(id=task_id).first()
|
||||
if task is None:
|
||||
return
|
||||
|
||||
item_ids = list(scope_queryset(task.scope).values_list("id", flat=True))
|
||||
counters = {"matched": 0, "not_found": 0, "deleted": 0}
|
||||
failed = 0
|
||||
MatchTask.objects.filter(id=task_id).update(
|
||||
status=MatchTask.STATUS_RUNNING,
|
||||
total=len(item_ids),
|
||||
processed=0,
|
||||
updated_at=timezone.now(),
|
||||
)
|
||||
|
||||
def save_progress(processed):
|
||||
MatchTask.objects.filter(id=task_id).update(
|
||||
processed=processed,
|
||||
failed=failed,
|
||||
updated_at=timezone.now(),
|
||||
**counters,
|
||||
)
|
||||
|
||||
try:
|
||||
processed = 0
|
||||
for item_id in item_ids:
|
||||
if MatchTask.objects.filter(id=task_id, cancelled=True).exists():
|
||||
MatchTask.objects.filter(id=task_id).update(
|
||||
status=MatchTask.STATUS_CANCELLED, updated_at=timezone.now()
|
||||
)
|
||||
return
|
||||
item = MediaItem.objects.filter(id=item_id).first()
|
||||
if item is None:
|
||||
processed += 1
|
||||
continue
|
||||
try:
|
||||
status = check_item(item, task.user)
|
||||
counters[status] = counters.get(status, 0) + 1
|
||||
except e621.E621Error as exc:
|
||||
# A broken credential or an e621 outage would fail every
|
||||
# remaining item; abort instead of hammering the API.
|
||||
logger.warning("Match scan %s aborted: %s", task_id, exc)
|
||||
failed += 1
|
||||
MatchTask.objects.filter(id=task_id).update(
|
||||
status=MatchTask.STATUS_ERROR,
|
||||
processed=processed,
|
||||
failed=failed,
|
||||
error=str(exc),
|
||||
updated_at=timezone.now(),
|
||||
**counters,
|
||||
)
|
||||
return
|
||||
processed += 1
|
||||
save_progress(processed)
|
||||
MatchTask.objects.filter(id=task_id).update(
|
||||
status=MatchTask.STATUS_COMPLETE,
|
||||
processed=len(item_ids),
|
||||
updated_at=timezone.now(),
|
||||
**counters,
|
||||
)
|
||||
finally:
|
||||
# Background threads hold their own DB connection; release it so
|
||||
# Gunicorn workers do not leak connections when threads finish.
|
||||
connection.close()
|
||||
@@ -0,0 +1,42 @@
|
||||
# Generated by Django 6.1.1 on 2026-09-17 16:06
|
||||
|
||||
import django.db.models.deletion
|
||||
import uuid
|
||||
from django.conf import settings
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('library', '0003_mediaitem_e621_data_mediaitem_e621_post_id'),
|
||||
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name='TempUpload',
|
||||
fields=[
|
||||
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
|
||||
('file', models.FileField(blank=True, upload_to='uploads/temp/')),
|
||||
('original_filename', models.CharField(max_length=255)),
|
||||
('md5', models.CharField(db_index=True, max_length=32)),
|
||||
('size', models.BigIntegerField(default=0)),
|
||||
('status', models.CharField(choices=[('pending', 'Pending'), ('visual_match', 'Visual Similarity Detected'), ('completed', 'Completed'), ('error', 'Error')], default='pending', max_length=20)),
|
||||
('resolution', models.CharField(blank=True, default='', max_length=20)),
|
||||
('e621_post_id', models.IntegerField(blank=True, null=True)),
|
||||
('e621_data', models.JSONField(blank=True, null=True)),
|
||||
('custom_rating', models.CharField(blank=True, default='', max_length=1)),
|
||||
('custom_tags', models.JSONField(blank=True, default=list)),
|
||||
('custom_notes', models.TextField(blank=True, default='')),
|
||||
('iqdb_data', models.JSONField(blank=True, null=True)),
|
||||
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||
('updated_at', models.DateTimeField(auto_now=True)),
|
||||
('library_item', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='temp_uploads', to='library.mediaitem')),
|
||||
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='temp_uploads', to=settings.AUTH_USER_MODEL)),
|
||||
],
|
||||
options={
|
||||
'ordering': ['-created_at'],
|
||||
},
|
||||
),
|
||||
]
|
||||
@@ -0,0 +1,41 @@
|
||||
# Generated by Django 6.1.1 on 2026-09-17 17:03
|
||||
|
||||
import django.db.models.deletion
|
||||
import uuid
|
||||
from django.conf import settings
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('library', '0004_tempupload'),
|
||||
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name='DownloadTask',
|
||||
fields=[
|
||||
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
|
||||
('post_id', models.IntegerField(blank=True, null=True)),
|
||||
('url', models.CharField(max_length=1000)),
|
||||
('filename', models.CharField(blank=True, default='', max_length=255)),
|
||||
('e621_data', models.JSONField(blank=True, null=True)),
|
||||
('status', models.CharField(choices=[('pending', 'Pending'), ('downloading', 'Downloading'), ('complete', 'Complete'), ('error', 'Error'), ('cancelled', 'Cancelled')], default='pending', max_length=20)),
|
||||
('progress', models.IntegerField(default=0)),
|
||||
('downloaded', models.BigIntegerField(default=0)),
|
||||
('total', models.BigIntegerField(default=0)),
|
||||
('speed', models.FloatField(blank=True, null=True)),
|
||||
('error', models.TextField(blank=True, default='')),
|
||||
('cancelled', models.BooleanField(default=False)),
|
||||
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||
('updated_at', models.DateTimeField(auto_now=True)),
|
||||
('library_item', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='download_tasks', to='library.mediaitem')),
|
||||
('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='download_tasks', to=settings.AUTH_USER_MODEL)),
|
||||
],
|
||||
options={
|
||||
'ordering': ['-created_at'],
|
||||
},
|
||||
),
|
||||
]
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
# Generated by Django 6.1.1 on 2026-09-17 17:34
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('library', '0005_downloadtask'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name='mediaitem',
|
||||
name='ahash',
|
||||
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='mediaitem',
|
||||
name='dhash',
|
||||
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='mediaitem',
|
||||
name='phash',
|
||||
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='mediaitem',
|
||||
name='whash',
|
||||
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
|
||||
),
|
||||
]
|
||||
@@ -0,0 +1,18 @@
|
||||
# Generated by Django 6.1.1 on 2026-09-17 17:42
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('library', '0006_mediaitem_ahash_mediaitem_dhash_mediaitem_phash_and_more'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name='tempupload',
|
||||
name='visual_matches',
|
||||
field=models.JSONField(blank=True, null=True),
|
||||
),
|
||||
]
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
# Generated by Django 6.1.1 on 2026-09-17 18:04
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
def backfill_derived_fields(apps, schema_editor):
|
||||
MediaItem = apps.get_model("library", "MediaItem")
|
||||
for item in MediaItem.objects.all().iterator(chunk_size=200):
|
||||
names = [str(tag) for tag in (item.tags or [])]
|
||||
data = item.e621_data if isinstance(item.e621_data, dict) else {}
|
||||
categories = data.get("tags")
|
||||
if isinstance(categories, dict):
|
||||
for values in categories.values():
|
||||
if isinstance(values, list):
|
||||
names.extend(str(tag) for tag in values)
|
||||
search_tags = " ".join(
|
||||
sorted({str(name).strip().lower() for name in names if str(name).strip()})
|
||||
)
|
||||
MediaItem.objects.filter(pk=item.pk).update(
|
||||
search_tags=search_tags,
|
||||
has_custom_data=bool(item.tags or item.notes),
|
||||
)
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('library', '0007_tempupload_visual_matches'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name='mediaitem',
|
||||
name='has_custom_data',
|
||||
field=models.BooleanField(db_index=True, default=False),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='mediaitem',
|
||||
name='search_tags',
|
||||
field=models.TextField(blank=True, default=''),
|
||||
),
|
||||
migrations.RunPython(backfill_derived_fields, migrations.RunPython.noop),
|
||||
]
|
||||
@@ -0,0 +1,67 @@
|
||||
# Generated by Django 6.1.1 on 2026-09-17 18:35
|
||||
|
||||
import django.db.models.deletion
|
||||
import uuid
|
||||
from django.conf import settings
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
def mark_existing_matches(apps, schema_editor):
|
||||
"""Items downloaded/linked before the match cache existed."""
|
||||
MediaItem = apps.get_model("library", "MediaItem")
|
||||
MediaItem.objects.filter(e621_post_id__isnull=False).update(
|
||||
e621_match_status="matched",
|
||||
e621_checked_at=models.F("updated_at"),
|
||||
)
|
||||
|
||||
|
||||
def unmark_matches(apps, schema_editor):
|
||||
MediaItem = apps.get_model("library", "MediaItem")
|
||||
MediaItem.objects.filter(e621_post_id__isnull=False).update(
|
||||
e621_match_status="unknown",
|
||||
e621_checked_at=None,
|
||||
)
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('library', '0008_mediaitem_has_custom_data_mediaitem_search_tags'),
|
||||
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name='mediaitem',
|
||||
name='e621_checked_at',
|
||||
field=models.DateTimeField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='mediaitem',
|
||||
name='e621_match_status',
|
||||
field=models.CharField(choices=[('unknown', 'Not checked'), ('matched', 'Matched'), ('not_found', 'Not found on e621'), ('deleted', 'Post deleted on e621')], db_index=True, default='unknown', max_length=20),
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name='MatchTask',
|
||||
fields=[
|
||||
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
|
||||
('scope', models.CharField(choices=[('missing', 'Unchecked items'), ('all', 'Whole library')], default='missing', max_length=20)),
|
||||
('status', models.CharField(choices=[('pending', 'Pending'), ('running', 'Running'), ('complete', 'Complete'), ('error', 'Error'), ('cancelled', 'Cancelled')], default='pending', max_length=20)),
|
||||
('total', models.IntegerField(default=0)),
|
||||
('processed', models.IntegerField(default=0)),
|
||||
('matched', models.IntegerField(default=0)),
|
||||
('not_found', models.IntegerField(default=0)),
|
||||
('deleted', models.IntegerField(default=0)),
|
||||
('failed', models.IntegerField(default=0)),
|
||||
('error', models.TextField(blank=True, default='')),
|
||||
('cancelled', models.BooleanField(default=False)),
|
||||
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||
('updated_at', models.DateTimeField(auto_now=True)),
|
||||
('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='match_tasks', to=settings.AUTH_USER_MODEL)),
|
||||
],
|
||||
options={
|
||||
'ordering': ['-created_at'],
|
||||
},
|
||||
),
|
||||
migrations.RunPython(mark_existing_matches, unmark_matches),
|
||||
]
|
||||
@@ -1,10 +1,43 @@
|
||||
from django.conf import settings
|
||||
from django.db import models
|
||||
|
||||
import uuid
|
||||
|
||||
|
||||
TAG_CLOUD_CACHE_KEYS = [
|
||||
"j621.library.tag_cloud.auth",
|
||||
"j621.library.tag_cloud.guest",
|
||||
]
|
||||
|
||||
|
||||
def build_search_tags(tags, e621_data):
|
||||
"""Lowercase tag text (custom + e621) used for library tag search."""
|
||||
names = [str(tag) for tag in (tags or [])]
|
||||
data = e621_data if isinstance(e621_data, dict) else {}
|
||||
categories = data.get("tags")
|
||||
if isinstance(categories, dict):
|
||||
for values in categories.values():
|
||||
if isinstance(values, list):
|
||||
names.extend(str(tag) for tag in values)
|
||||
return " ".join(
|
||||
sorted({str(name).strip().lower() for name in names if str(name).strip()})
|
||||
)
|
||||
|
||||
|
||||
class MediaItem(models.Model):
|
||||
"""A logical media file, identified by its MD5 fingerprint."""
|
||||
|
||||
E621_UNKNOWN = "unknown"
|
||||
E621_MATCHED = "matched"
|
||||
E621_NOT_FOUND = "not_found"
|
||||
E621_DELETED = "deleted"
|
||||
E621_STATUS_CHOICES = [
|
||||
(E621_UNKNOWN, "Not checked"),
|
||||
(E621_MATCHED, "Matched"),
|
||||
(E621_NOT_FOUND, "Not found on e621"),
|
||||
(E621_DELETED, "Post deleted on e621"),
|
||||
]
|
||||
|
||||
md5 = models.CharField(max_length=32, unique=True)
|
||||
size = models.BigIntegerField(default=0)
|
||||
# Custom rating: "s" (safe), "q" (questionable), "e" (explicit) or "" (unrated).
|
||||
@@ -22,6 +55,22 @@ class MediaItem(models.Model):
|
||||
hidden_from_guests = models.BooleanField(default=False, db_index=True)
|
||||
e621_post_id = models.IntegerField(null=True, blank=True, db_index=True)
|
||||
e621_data = models.JSONField(null=True, blank=True)
|
||||
# Cached result of the last MD5/manual match against e621.
|
||||
e621_match_status = models.CharField(
|
||||
max_length=20,
|
||||
choices=E621_STATUS_CHOICES,
|
||||
default=E621_UNKNOWN,
|
||||
db_index=True,
|
||||
)
|
||||
e621_checked_at = models.DateTimeField(null=True, blank=True)
|
||||
# Perceptual hashes (hex strings) used by the duplicates engine.
|
||||
ahash = models.CharField(max_length=32, blank=True, default="", db_index=True)
|
||||
dhash = models.CharField(max_length=32, blank=True, default="", db_index=True)
|
||||
phash = models.CharField(max_length=32, blank=True, default="", db_index=True)
|
||||
whash = models.CharField(max_length=32, blank=True, default="", db_index=True)
|
||||
# Derived search/filter helpers: flattened tag text and custom-data flag.
|
||||
search_tags = models.TextField(blank=True, default="")
|
||||
has_custom_data = models.BooleanField(default=False, db_index=True)
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
updated_at = models.DateTimeField(auto_now=True)
|
||||
|
||||
@@ -32,13 +81,22 @@ class MediaItem(models.Model):
|
||||
return f"J-{self.pk} ({self.md5})"
|
||||
|
||||
def save(self, *args, **kwargs):
|
||||
from django.core.cache import cache
|
||||
|
||||
from .guest_filter import item_is_hidden_for_guests
|
||||
|
||||
self.hidden_from_guests = item_is_hidden_for_guests(self)
|
||||
self.has_custom_data = bool(self.tags or self.notes)
|
||||
self.search_tags = build_search_tags(self.tags, self.e621_data)
|
||||
update_fields = kwargs.get("update_fields")
|
||||
if update_fields is not None:
|
||||
kwargs["update_fields"] = set(update_fields) | {"hidden_from_guests"}
|
||||
kwargs["update_fields"] = set(update_fields) | {
|
||||
"hidden_from_guests",
|
||||
"has_custom_data",
|
||||
"search_tags",
|
||||
}
|
||||
super().save(*args, **kwargs)
|
||||
cache.delete_many(TAG_CLOUD_CACHE_KEYS)
|
||||
|
||||
|
||||
class MediaLocation(models.Model):
|
||||
@@ -57,3 +115,169 @@ class MediaLocation(models.Model):
|
||||
|
||||
def __str__(self):
|
||||
return self.rel_path
|
||||
|
||||
|
||||
class TempUpload(models.Model):
|
||||
"""A staged upload that has not been accepted into the library yet."""
|
||||
|
||||
STATUS_PENDING = "pending"
|
||||
STATUS_VISUAL_MATCH = "visual_match"
|
||||
STATUS_COMPLETED = "completed"
|
||||
STATUS_ERROR = "error"
|
||||
STATUS_CHOICES = [
|
||||
(STATUS_PENDING, "Pending"),
|
||||
(STATUS_VISUAL_MATCH, "Visual Similarity Detected"),
|
||||
(STATUS_COMPLETED, "Completed"),
|
||||
(STATUS_ERROR, "Error"),
|
||||
]
|
||||
|
||||
RESOLUTION_AUTO_MD5 = "auto_md5"
|
||||
RESOLUTION_DUPLICATE = "duplicate"
|
||||
RESOLUTION_LINKED = "linked"
|
||||
RESOLUTION_CUSTOM = "custom"
|
||||
|
||||
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
|
||||
user = models.ForeignKey(
|
||||
settings.AUTH_USER_MODEL,
|
||||
on_delete=models.CASCADE,
|
||||
related_name="temp_uploads",
|
||||
)
|
||||
file = models.FileField(upload_to="uploads/temp/", blank=True)
|
||||
original_filename = models.CharField(max_length=255)
|
||||
md5 = models.CharField(max_length=32, db_index=True)
|
||||
size = models.BigIntegerField(default=0)
|
||||
status = models.CharField(
|
||||
max_length=20, choices=STATUS_CHOICES, default=STATUS_PENDING
|
||||
)
|
||||
resolution = models.CharField(max_length=20, blank=True, default="")
|
||||
e621_post_id = models.IntegerField(null=True, blank=True)
|
||||
e621_data = models.JSONField(null=True, blank=True)
|
||||
custom_rating = models.CharField(max_length=1, blank=True, default="")
|
||||
custom_tags = models.JSONField(default=list, blank=True)
|
||||
custom_notes = models.TextField(blank=True, default="")
|
||||
iqdb_data = models.JSONField(null=True, blank=True)
|
||||
visual_matches = models.JSONField(null=True, blank=True)
|
||||
library_item = models.ForeignKey(
|
||||
MediaItem,
|
||||
null=True,
|
||||
blank=True,
|
||||
on_delete=models.SET_NULL,
|
||||
related_name="temp_uploads",
|
||||
)
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
updated_at = models.DateTimeField(auto_now=True)
|
||||
|
||||
class Meta:
|
||||
ordering = ["-created_at"]
|
||||
|
||||
def __str__(self):
|
||||
return f"{self.original_filename} ({self.status})"
|
||||
|
||||
|
||||
class DownloadTask(models.Model):
|
||||
"""A background 'Download to Library' job with progress tracking."""
|
||||
|
||||
STATUS_PENDING = "pending"
|
||||
STATUS_DOWNLOADING = "downloading"
|
||||
STATUS_COMPLETE = "complete"
|
||||
STATUS_ERROR = "error"
|
||||
STATUS_CANCELLED = "cancelled"
|
||||
STATUS_CHOICES = [
|
||||
(STATUS_PENDING, "Pending"),
|
||||
(STATUS_DOWNLOADING, "Downloading"),
|
||||
(STATUS_COMPLETE, "Complete"),
|
||||
(STATUS_ERROR, "Error"),
|
||||
(STATUS_CANCELLED, "Cancelled"),
|
||||
]
|
||||
|
||||
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
|
||||
user = models.ForeignKey(
|
||||
settings.AUTH_USER_MODEL,
|
||||
null=True,
|
||||
blank=True,
|
||||
on_delete=models.SET_NULL,
|
||||
related_name="download_tasks",
|
||||
)
|
||||
post_id = models.IntegerField(null=True, blank=True)
|
||||
url = models.CharField(max_length=1000)
|
||||
filename = models.CharField(max_length=255, blank=True, default="")
|
||||
e621_data = models.JSONField(null=True, blank=True)
|
||||
status = models.CharField(
|
||||
max_length=20, choices=STATUS_CHOICES, default=STATUS_PENDING
|
||||
)
|
||||
progress = models.IntegerField(default=0)
|
||||
downloaded = models.BigIntegerField(default=0)
|
||||
total = models.BigIntegerField(default=0)
|
||||
speed = models.FloatField(null=True, blank=True)
|
||||
error = models.TextField(blank=True, default="")
|
||||
cancelled = models.BooleanField(default=False)
|
||||
library_item = models.ForeignKey(
|
||||
MediaItem,
|
||||
null=True,
|
||||
blank=True,
|
||||
on_delete=models.SET_NULL,
|
||||
related_name="download_tasks",
|
||||
)
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
updated_at = models.DateTimeField(auto_now=True)
|
||||
|
||||
class Meta:
|
||||
ordering = ["-created_at"]
|
||||
|
||||
def __str__(self):
|
||||
return f"Download {self.id} ({self.status})"
|
||||
|
||||
|
||||
class MatchTask(models.Model):
|
||||
"""A background library-wide e621 match scan with progress tracking."""
|
||||
|
||||
STATUS_PENDING = "pending"
|
||||
STATUS_RUNNING = "running"
|
||||
STATUS_COMPLETE = "complete"
|
||||
STATUS_ERROR = "error"
|
||||
STATUS_CANCELLED = "cancelled"
|
||||
STATUS_CHOICES = [
|
||||
(STATUS_PENDING, "Pending"),
|
||||
(STATUS_RUNNING, "Running"),
|
||||
(STATUS_COMPLETE, "Complete"),
|
||||
(STATUS_ERROR, "Error"),
|
||||
(STATUS_CANCELLED, "Cancelled"),
|
||||
]
|
||||
|
||||
SCOPE_MISSING = "missing"
|
||||
SCOPE_ALL = "all"
|
||||
SCOPE_CHOICES = [
|
||||
(SCOPE_MISSING, "Unchecked items"),
|
||||
(SCOPE_ALL, "Whole library"),
|
||||
]
|
||||
|
||||
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
|
||||
user = models.ForeignKey(
|
||||
settings.AUTH_USER_MODEL,
|
||||
null=True,
|
||||
blank=True,
|
||||
on_delete=models.SET_NULL,
|
||||
related_name="match_tasks",
|
||||
)
|
||||
scope = models.CharField(
|
||||
max_length=20, choices=SCOPE_CHOICES, default=SCOPE_MISSING
|
||||
)
|
||||
status = models.CharField(
|
||||
max_length=20, choices=STATUS_CHOICES, default=STATUS_PENDING
|
||||
)
|
||||
total = models.IntegerField(default=0)
|
||||
processed = models.IntegerField(default=0)
|
||||
matched = models.IntegerField(default=0)
|
||||
not_found = models.IntegerField(default=0)
|
||||
deleted = models.IntegerField(default=0)
|
||||
failed = models.IntegerField(default=0)
|
||||
error = models.TextField(blank=True, default="")
|
||||
cancelled = models.BooleanField(default=False)
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
updated_at = models.DateTimeField(auto_now=True)
|
||||
|
||||
class Meta:
|
||||
ordering = ["-created_at"]
|
||||
|
||||
def __str__(self):
|
||||
return f"Match scan {self.id} ({self.status})"
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from django.core import signing
|
||||
from rest_framework import serializers
|
||||
|
||||
from .models import MediaItem, MediaLocation
|
||||
from .services import VIDEO_EXTENSIONS
|
||||
from .models import MediaItem, MediaLocation, TempUpload, DownloadTask, MatchTask
|
||||
from .services import (
|
||||
MEDIA_FILE_SALT,
|
||||
UPLOAD_FILE_SALT,
|
||||
VIDEO_EXTENSIONS,
|
||||
signed_media_url,
|
||||
)
|
||||
|
||||
|
||||
class MediaLocationSerializer(serializers.ModelSerializer):
|
||||
@@ -20,6 +27,8 @@ class MediaItemSerializer(serializers.ModelSerializer):
|
||||
locations = MediaLocationSerializer(many=True, read_only=True)
|
||||
j_id = serializers.SerializerMethodField()
|
||||
display_rating = serializers.SerializerMethodField()
|
||||
raw_url = serializers.SerializerMethodField()
|
||||
thumbnail_url = serializers.SerializerMethodField()
|
||||
filename = serializers.SerializerMethodField()
|
||||
extension = serializers.SerializerMethodField()
|
||||
kind = serializers.SerializerMethodField()
|
||||
@@ -40,10 +49,14 @@ class MediaItemSerializer(serializers.ModelSerializer):
|
||||
"filename",
|
||||
"extension",
|
||||
"kind",
|
||||
"raw_url",
|
||||
"thumbnail_url",
|
||||
"uploaded_by",
|
||||
"uploaded_by_id",
|
||||
"e621_post_id",
|
||||
"e621_data",
|
||||
"e621_match_status",
|
||||
"e621_checked_at",
|
||||
"locations",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
@@ -57,18 +70,35 @@ class MediaItemSerializer(serializers.ModelSerializer):
|
||||
"filename",
|
||||
"extension",
|
||||
"kind",
|
||||
"raw_url",
|
||||
"thumbnail_url",
|
||||
"uploaded_by",
|
||||
"uploaded_by_id",
|
||||
"e621_post_id",
|
||||
"e621_data",
|
||||
"e621_match_status",
|
||||
"e621_checked_at",
|
||||
"locations",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
]
|
||||
|
||||
def _request_user(self):
|
||||
request = self.context.get("request")
|
||||
user = getattr(request, "user", None)
|
||||
if user is None or not getattr(user, "is_authenticated", False):
|
||||
return None
|
||||
return user
|
||||
|
||||
def get_j_id(self, obj):
|
||||
return f"J-{obj.id}"
|
||||
|
||||
def get_raw_url(self, obj):
|
||||
return signed_media_url(obj, self._request_user(), "raw")
|
||||
|
||||
def get_thumbnail_url(self, obj):
|
||||
return signed_media_url(obj, self._request_user(), "thumbnail")
|
||||
|
||||
def get_display_rating(self, obj):
|
||||
if obj.rating:
|
||||
return obj.rating
|
||||
@@ -92,3 +122,127 @@ class MediaItemSerializer(serializers.ModelSerializer):
|
||||
|
||||
def get_kind(self, obj):
|
||||
return "video" if "." + self.get_extension(obj) in VIDEO_EXTENSIONS else "image"
|
||||
|
||||
|
||||
class TempUploadSerializer(serializers.ModelSerializer):
|
||||
temp_id = serializers.UUIDField(source="id", read_only=True)
|
||||
library_j_id = serializers.SerializerMethodField()
|
||||
file_url = serializers.SerializerMethodField()
|
||||
preview_url = serializers.SerializerMethodField()
|
||||
|
||||
class Meta:
|
||||
model = TempUpload
|
||||
fields = [
|
||||
"temp_id",
|
||||
"original_filename",
|
||||
"md5",
|
||||
"size",
|
||||
"status",
|
||||
"resolution",
|
||||
"e621_post_id",
|
||||
"e621_data",
|
||||
"custom_rating",
|
||||
"custom_tags",
|
||||
"custom_notes",
|
||||
"iqdb_data",
|
||||
"visual_matches",
|
||||
"library_j_id",
|
||||
"file_url",
|
||||
"preview_url",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
]
|
||||
read_only_fields = fields
|
||||
|
||||
def _request_user(self):
|
||||
request = self.context.get("request")
|
||||
user = getattr(request, "user", None)
|
||||
if user is None or not getattr(user, "is_authenticated", False):
|
||||
return None
|
||||
return user
|
||||
|
||||
def get_library_j_id(self, obj):
|
||||
return f"J-{obj.library_item_id}" if obj.library_item_id else None
|
||||
|
||||
def get_file_url(self, obj):
|
||||
"""Signed URL so <img>/<video> tags can fetch the staged file."""
|
||||
if not obj.file:
|
||||
return None
|
||||
user = self._request_user()
|
||||
if user is None:
|
||||
return None
|
||||
signature = signing.dumps(
|
||||
{"temp": str(obj.id), "user": user.id},
|
||||
salt=UPLOAD_FILE_SALT,
|
||||
)
|
||||
return f"/api/uploads/{obj.id}/file/?sig={signature}"
|
||||
|
||||
def get_preview_url(self, obj):
|
||||
"""A URL an <img>/<video> tag can load: staged file or library item."""
|
||||
user = self._request_user()
|
||||
if user is None:
|
||||
return None
|
||||
if obj.file:
|
||||
return self.get_file_url(obj)
|
||||
item = obj.library_item
|
||||
if item is None:
|
||||
return None
|
||||
location = item.locations.first()
|
||||
is_video = (
|
||||
location is not None
|
||||
and Path(location.rel_path).suffix.lower() in VIDEO_EXTENSIONS
|
||||
)
|
||||
action = "thumbnail" if is_video else "raw"
|
||||
signature = signing.dumps(
|
||||
{"item": item.id, "user": user.id, "action": action},
|
||||
salt=MEDIA_FILE_SALT,
|
||||
)
|
||||
return f"/api/files/J-{item.id}/{action}/?sig={signature}"
|
||||
|
||||
|
||||
class DownloadTaskSerializer(serializers.ModelSerializer):
|
||||
task_id = serializers.UUIDField(source="id", read_only=True)
|
||||
library_j_id = serializers.SerializerMethodField()
|
||||
|
||||
class Meta:
|
||||
model = DownloadTask
|
||||
fields = [
|
||||
"task_id",
|
||||
"post_id",
|
||||
"filename",
|
||||
"status",
|
||||
"progress",
|
||||
"downloaded",
|
||||
"total",
|
||||
"speed",
|
||||
"error",
|
||||
"library_j_id",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
]
|
||||
read_only_fields = fields
|
||||
|
||||
def get_library_j_id(self, obj):
|
||||
return f"J-{obj.library_item_id}" if obj.library_item_id else None
|
||||
|
||||
|
||||
class MatchTaskSerializer(serializers.ModelSerializer):
|
||||
task_id = serializers.UUIDField(source="id", read_only=True)
|
||||
|
||||
class Meta:
|
||||
model = MatchTask
|
||||
fields = [
|
||||
"task_id",
|
||||
"scope",
|
||||
"status",
|
||||
"total",
|
||||
"processed",
|
||||
"matched",
|
||||
"not_found",
|
||||
"deleted",
|
||||
"failed",
|
||||
"error",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
]
|
||||
read_only_fields = fields
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import mimetypes
|
||||
import os
|
||||
import re
|
||||
@@ -7,14 +8,33 @@ import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import imagehash
|
||||
from django.conf import settings
|
||||
from django.core import signing
|
||||
from django.http import FileResponse, Http404, HttpResponse
|
||||
from django.utils.text import get_valid_filename
|
||||
from PIL import Image
|
||||
|
||||
from .models import MediaItem, MediaLocation
|
||||
|
||||
ALLOWED_EXTENSIONS = {".jpg", ".jpeg", ".png", ".gif", ".apng", ".mp4", ".webm"}
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
HASH_FIELDS = ("ahash", "dhash", "phash", "whash")
|
||||
IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".gif", ".apng", ".webp"}
|
||||
|
||||
ALLOWED_EXTENSIONS = {
|
||||
".jpg",
|
||||
".jpeg",
|
||||
".png",
|
||||
".gif",
|
||||
".apng",
|
||||
".webp",
|
||||
".mp4",
|
||||
".webm",
|
||||
}
|
||||
VIDEO_EXTENSIONS = {".mp4", ".webm"}
|
||||
UPLOAD_FILE_SALT = "j621.upload-file"
|
||||
MEDIA_FILE_SALT = "j621.media-file"
|
||||
CHUNK_SIZE = 1024 * 1024
|
||||
RANGE_RE = re.compile(r"bytes=(\d*)-(\d*)$")
|
||||
|
||||
@@ -30,7 +50,7 @@ def compute_md5(path):
|
||||
def index_file(path, folder):
|
||||
"""Index one file into MediaItem/MediaLocation.
|
||||
|
||||
Returns (item, created_item, created_location).
|
||||
Returns (item, created_item, location, created_location).
|
||||
"""
|
||||
path = Path(path)
|
||||
folder = Path(folder)
|
||||
@@ -50,7 +70,72 @@ def index_file(path, folder):
|
||||
"mtime": stat.st_mtime,
|
||||
},
|
||||
)
|
||||
return item, created_item, created_location
|
||||
return item, created_item, location, created_location
|
||||
|
||||
|
||||
def signed_media_url(item, user, action="raw"):
|
||||
"""Media URL that <img>/<video> tags can load for a signed-in user."""
|
||||
base = f"/api/files/J-{item.id}/{action}/"
|
||||
if user is None or not getattr(user, "is_authenticated", False):
|
||||
return base
|
||||
signature = signing.dumps(
|
||||
{"item": item.id, "user": user.id, "action": action},
|
||||
salt=MEDIA_FILE_SALT,
|
||||
)
|
||||
return f"{base}?sig={signature}"
|
||||
|
||||
|
||||
def rename_location_to_j_id(item, location):
|
||||
"""Name a freshly indexed copy J-<id>.<ext> inside its own folder."""
|
||||
path = Path(location.path)
|
||||
if not path.exists():
|
||||
return location
|
||||
if path.stem == f"J-{item.id}":
|
||||
return location
|
||||
target = unique_destination(path.parent, f"J-{item.id}{path.suffix}")
|
||||
path.rename(target)
|
||||
parent = Path(location.rel_path).parent
|
||||
location.path = str(target)
|
||||
location.rel_path = (
|
||||
str(parent / target.name) if str(parent) != "." else target.name
|
||||
)
|
||||
location.save(update_fields=["path", "rel_path"])
|
||||
return location
|
||||
|
||||
|
||||
def compute_visual_hashes(path):
|
||||
"""Perceptual hashes for an image file (empty dict for other files)."""
|
||||
path = Path(path)
|
||||
if path.suffix.lower() not in IMAGE_EXTENSIONS:
|
||||
return {}
|
||||
try:
|
||||
with Image.open(path) as image:
|
||||
converted = image.convert("RGB")
|
||||
return {
|
||||
"ahash": str(imagehash.average_hash(converted, hash_size=8)),
|
||||
"dhash": str(imagehash.dhash(converted, hash_size=8)),
|
||||
"phash": str(imagehash.phash(converted, hash_size=8)),
|
||||
"whash": str(imagehash.whash(converted, hash_size=8)),
|
||||
}
|
||||
except Exception: # noqa: BLE001 - hashing must never break indexing
|
||||
logger.exception("Could not compute visual hashes for %s", path)
|
||||
return {}
|
||||
|
||||
|
||||
def ensure_visual_hashes(item):
|
||||
"""Fill in missing perceptual hashes for a media item."""
|
||||
if all(getattr(item, field) for field in HASH_FIELDS):
|
||||
return item
|
||||
location = item.locations.first()
|
||||
if location is None:
|
||||
return item
|
||||
hashes = compute_visual_hashes(location.path)
|
||||
if not hashes:
|
||||
return item
|
||||
for field, value in hashes.items():
|
||||
setattr(item, field, value)
|
||||
item.save(update_fields=[*hashes.keys(), "updated_at"])
|
||||
return item
|
||||
|
||||
|
||||
def parse_tags(raw):
|
||||
@@ -160,17 +245,32 @@ def serve_file(request, path, download=False):
|
||||
return response
|
||||
|
||||
|
||||
def download_file(url, destination):
|
||||
class DownloadCancelled(Exception):
|
||||
"""Raised when a streamed download is cancelled by the user."""
|
||||
|
||||
|
||||
def download_file(
|
||||
url, destination, progress_callback=None, should_cancel=None
|
||||
):
|
||||
"""Stream a remote file into destination (used by Download to Library)."""
|
||||
import requests
|
||||
|
||||
headers = {"User-Agent": settings.USER_AGENT}
|
||||
with requests.get(url, headers=headers, stream=True, timeout=120) as response:
|
||||
response.raise_for_status()
|
||||
total = int(response.headers.get("content-length") or 0)
|
||||
downloaded = 0
|
||||
with open(destination, "wb") as handle:
|
||||
for chunk in response.iter_content(chunk_size=CHUNK_SIZE):
|
||||
if should_cancel is not None and should_cancel():
|
||||
raise DownloadCancelled("download cancelled")
|
||||
if chunk:
|
||||
handle.write(chunk)
|
||||
downloaded += len(chunk)
|
||||
if progress_callback is not None:
|
||||
progress_callback(downloaded, total)
|
||||
if progress_callback is not None:
|
||||
progress_callback(downloaded, total or downloaded)
|
||||
|
||||
|
||||
E621_DESCRIPTION_LIMIT = 20000
|
||||
@@ -221,11 +321,61 @@ def trim_e621_post(post):
|
||||
"size": file_data.get("size"),
|
||||
"width": file_data.get("width"),
|
||||
"height": file_data.get("height"),
|
||||
"url": file_data.get("url"),
|
||||
},
|
||||
"preview": {
|
||||
"url": (post.get("preview") or {}).get("url"),
|
||||
},
|
||||
"uploader_name": post.get("uploader_name"),
|
||||
}
|
||||
|
||||
|
||||
def sanitize_iqdb_results(results):
|
||||
"""Keep a compact, size-bounded copy of IQDB candidates."""
|
||||
cleaned = []
|
||||
for result in results[:10]:
|
||||
if not isinstance(result, dict):
|
||||
continue
|
||||
score = result.get("score")
|
||||
post_id = result.get("post_id")
|
||||
tags_preview = result.get("tags_preview")
|
||||
entry = {
|
||||
"post_id": post_id if isinstance(post_id, int) else None,
|
||||
"score": float(score) if isinstance(score, (int, float)) else None,
|
||||
"preview_url": str(result.get("preview_url") or "")[:500] or None,
|
||||
"rating": str(result.get("rating") or "")[:1] or None,
|
||||
"md5": str(result.get("md5") or "")[:32] or None,
|
||||
"score_total": (
|
||||
result.get("score_total")
|
||||
if isinstance(result.get("score_total"), int)
|
||||
else None
|
||||
),
|
||||
"fav_count": (
|
||||
result.get("fav_count")
|
||||
if isinstance(result.get("fav_count"), int)
|
||||
else None
|
||||
),
|
||||
"width": (
|
||||
result.get("width")
|
||||
if isinstance(result.get("width"), int)
|
||||
else None
|
||||
),
|
||||
"height": (
|
||||
result.get("height")
|
||||
if isinstance(result.get("height"), int)
|
||||
else None
|
||||
),
|
||||
"tags_preview": (
|
||||
[str(tag)[:100] for tag in tags_preview][:8]
|
||||
if isinstance(tags_preview, list)
|
||||
else []
|
||||
),
|
||||
}
|
||||
if entry["post_id"] or entry["preview_url"]:
|
||||
cleaned.append(entry)
|
||||
return cleaned
|
||||
|
||||
|
||||
def generate_video_thumbnail(md5, path):
|
||||
"""Extract a JPEG thumbnail from a video, cached under MEDIA_ROOT/thumbs."""
|
||||
if not shutil.which("ffmpeg"):
|
||||
|
||||
@@ -0,0 +1,451 @@
|
||||
"""Library tooling: duplicates, deletion and storage overview."""
|
||||
|
||||
import logging
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from django.conf import settings
|
||||
from django.core.cache import cache
|
||||
from django.db.models import Count, Q
|
||||
from rest_framework import status
|
||||
from rest_framework.permissions import AllowAny, IsAuthenticated
|
||||
from rest_framework.response import Response
|
||||
from rest_framework.views import APIView
|
||||
|
||||
from .models import TAG_CLOUD_CACHE_KEYS, MediaItem, MediaLocation
|
||||
from .permissions import CanUpload
|
||||
from .services import signed_media_url
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
HASH_FIELDS = ("ahash", "dhash", "phash", "whash")
|
||||
STORAGE_CACHE_KEY = "j621.library.storage"
|
||||
STORAGE_CACHE_TTL = 60
|
||||
GROUPS_PER_PAGE = 20
|
||||
|
||||
|
||||
def parse_threshold(value, default=0.8):
|
||||
try:
|
||||
parsed = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
if parsed > 1:
|
||||
parsed = parsed / 100.0
|
||||
return min(max(parsed, 0.0), 1.0)
|
||||
|
||||
|
||||
def parse_algorithms(value):
|
||||
if not isinstance(value, list):
|
||||
return list(HASH_FIELDS)
|
||||
selected = [field for field in value if field in HASH_FIELDS]
|
||||
return selected or list(HASH_FIELDS)
|
||||
|
||||
|
||||
def hashes_similarity(first_hashes, second_hashes, algorithms, threshold):
|
||||
"""Best similarity between two hash mappings, or None below threshold."""
|
||||
best = None
|
||||
for field in algorithms:
|
||||
first = first_hashes.get(field) or ""
|
||||
second = second_hashes.get(field) or ""
|
||||
if not first or not second:
|
||||
continue
|
||||
try:
|
||||
distance = bin(int(first, 16) ^ int(second, 16)).count("1")
|
||||
except ValueError:
|
||||
continue
|
||||
value = 1.0 - distance / 64.0
|
||||
if best is None or value > best:
|
||||
best = value
|
||||
if best is None or best < threshold:
|
||||
return None
|
||||
return best
|
||||
|
||||
|
||||
def similarity_between(item_a, item_b, algorithms, threshold):
|
||||
"""Best similarity across the selected algorithms, or None below threshold."""
|
||||
return hashes_similarity(
|
||||
{field: getattr(item_a, field, "") for field in algorithms},
|
||||
{field: getattr(item_b, field, "") for field in algorithms},
|
||||
algorithms,
|
||||
threshold,
|
||||
)
|
||||
|
||||
|
||||
def hashed_items(algorithms):
|
||||
query = Q()
|
||||
for field in algorithms:
|
||||
query |= ~Q(**{field: ""})
|
||||
return list(MediaItem.objects.filter(query).prefetch_related("locations"))
|
||||
|
||||
|
||||
def display_rating(item):
|
||||
if item.rating:
|
||||
return item.rating
|
||||
data = item.e621_data or {}
|
||||
rating = data.get("rating") if isinstance(data, dict) else None
|
||||
return rating if rating in {"s", "q", "e"} else ""
|
||||
|
||||
|
||||
def item_brief(item, request=None):
|
||||
locations = list(item.locations.all())
|
||||
location = locations[0] if locations else None
|
||||
user = getattr(request, "user", None)
|
||||
return {
|
||||
"j_id": f"J-{item.id}",
|
||||
"md5": item.md5,
|
||||
"filename": Path(location.rel_path).name if location else item.md5,
|
||||
"size": item.size,
|
||||
"rating": display_rating(item),
|
||||
"location_count": len(locations),
|
||||
"uploaded_by": item.uploaded_by.username if item.uploaded_by else None,
|
||||
"e621_post_id": item.e621_post_id,
|
||||
"thumbnail_url": signed_media_url(item, user, "thumbnail"),
|
||||
}
|
||||
|
||||
|
||||
def resolve_item(data):
|
||||
j_id = str(data.get("j_id") or "").strip()
|
||||
md5 = str(data.get("md5") or "").strip().lower()
|
||||
if j_id:
|
||||
numeric = j_id[2:] if j_id.upper().startswith("J-") else j_id
|
||||
if numeric.isdigit():
|
||||
item = MediaItem.objects.filter(pk=int(numeric)).first()
|
||||
if item is not None:
|
||||
return item
|
||||
if md5:
|
||||
return MediaItem.objects.filter(md5=md5).first()
|
||||
return None
|
||||
|
||||
|
||||
class ExactDuplicatesView(APIView):
|
||||
"""Items whose content exists at more than one path."""
|
||||
|
||||
permission_classes = [IsAuthenticated]
|
||||
|
||||
def get(self, request):
|
||||
items = (
|
||||
MediaItem.objects.annotate(location_count=Count("locations"))
|
||||
.filter(location_count__gt=1)
|
||||
.prefetch_related("locations")
|
||||
.order_by("-location_count", "id")
|
||||
)
|
||||
groups = []
|
||||
for item in items:
|
||||
brief = item_brief(item, request)
|
||||
brief["locations"] = [
|
||||
{"id": location.id, "rel_path": location.rel_path}
|
||||
for location in item.locations.all()
|
||||
]
|
||||
groups.append(brief)
|
||||
return Response({"count": len(groups), "groups": groups})
|
||||
|
||||
|
||||
class VisualMatchesView(APIView):
|
||||
"""Items visually similar to one library item."""
|
||||
|
||||
permission_classes = [IsAuthenticated]
|
||||
|
||||
def post(self, request):
|
||||
threshold = parse_threshold(request.data.get("threshold"))
|
||||
algorithms = parse_algorithms(request.data.get("algorithms"))
|
||||
target = resolve_item(request.data)
|
||||
if target is None:
|
||||
return Response(
|
||||
{"detail": "A j_id or md5 is required."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
|
||||
matches = []
|
||||
for item in hashed_items(algorithms):
|
||||
if item.pk == target.pk:
|
||||
continue
|
||||
similarity = similarity_between(target, item, algorithms, threshold)
|
||||
if similarity is None:
|
||||
continue
|
||||
brief = item_brief(item, request)
|
||||
brief["similarity"] = round(similarity * 100, 1)
|
||||
matches.append(brief)
|
||||
matches.sort(key=lambda entry: entry["similarity"], reverse=True)
|
||||
|
||||
return Response(
|
||||
{
|
||||
"target": item_brief(target, request),
|
||||
"threshold": round(threshold * 100, 1),
|
||||
"algorithms": algorithms,
|
||||
"count": len(matches),
|
||||
"matches": matches[:200],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class VisualGroupsView(APIView):
|
||||
"""Groups of visually similar items (union-find over perceptual hashes).
|
||||
|
||||
Pairwise comparison is O(n^2) with fast bit operations, which is fine for
|
||||
a personal library. Revisit with a bucketed index if libraries grow huge.
|
||||
"""
|
||||
|
||||
permission_classes = [IsAuthenticated]
|
||||
|
||||
def post(self, request):
|
||||
threshold = parse_threshold(request.data.get("threshold"))
|
||||
algorithms = parse_algorithms(request.data.get("algorithms"))
|
||||
page = max(1, int(request.data.get("page") or 1))
|
||||
|
||||
items = hashed_items(algorithms)
|
||||
parent = list(range(len(items)))
|
||||
|
||||
def find(index):
|
||||
while parent[index] != index:
|
||||
parent[index] = parent[parent[index]]
|
||||
index = parent[index]
|
||||
return index
|
||||
|
||||
def union(first, second):
|
||||
root_a = find(first)
|
||||
root_b = find(second)
|
||||
if root_a != root_b:
|
||||
parent[root_b] = root_a
|
||||
|
||||
for first in range(len(items)):
|
||||
for second in range(first + 1, len(items)):
|
||||
if find(first) == find(second):
|
||||
continue
|
||||
if (
|
||||
similarity_between(
|
||||
items[first], items[second], algorithms, threshold
|
||||
)
|
||||
is not None
|
||||
):
|
||||
union(first, second)
|
||||
|
||||
grouped = {}
|
||||
for index, item in enumerate(items):
|
||||
grouped.setdefault(find(index), []).append(item)
|
||||
|
||||
groups = [members for members in grouped.values() if len(members) >= 2]
|
||||
groups.sort(key=len, reverse=True)
|
||||
|
||||
total = len(groups)
|
||||
start = (page - 1) * GROUPS_PER_PAGE
|
||||
page_groups = groups[start : start + GROUPS_PER_PAGE]
|
||||
|
||||
return Response(
|
||||
{
|
||||
"count": total,
|
||||
"page": page,
|
||||
"per_page": GROUPS_PER_PAGE,
|
||||
"has_next": start + GROUPS_PER_PAGE < total,
|
||||
"threshold": round(threshold * 100, 1),
|
||||
"algorithms": algorithms,
|
||||
"groups": [
|
||||
{
|
||||
"size": len(members),
|
||||
"members": [item_brief(item, request) for item in members],
|
||||
}
|
||||
for members in page_groups
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def can_delete(user, item):
|
||||
if user.is_superuser or user.role == user.ROLE_STAFF:
|
||||
return True
|
||||
return item.uploaded_by_id == user.id
|
||||
|
||||
|
||||
def remove_watched_file(path):
|
||||
"""Delete a file only when it lives inside the watched folder."""
|
||||
watched = Path(settings.WATCHED_FOLDER).resolve()
|
||||
try:
|
||||
resolved = Path(path).resolve()
|
||||
resolved.relative_to(watched)
|
||||
except (ValueError, OSError):
|
||||
return False
|
||||
if resolved.is_file():
|
||||
resolved.unlink()
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
class DeleteFilesView(APIView):
|
||||
"""Delete items (with every copy) or individual duplicate locations."""
|
||||
|
||||
permission_classes = [CanUpload]
|
||||
|
||||
def post(self, request):
|
||||
j_ids = request.data.get("j_ids") or []
|
||||
location_ids = request.data.get("location_ids") or []
|
||||
if not isinstance(j_ids, list) or not isinstance(location_ids, list):
|
||||
return Response(
|
||||
{"detail": "j_ids and location_ids must be lists."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
|
||||
deleted = []
|
||||
errors = []
|
||||
|
||||
numeric_ids = []
|
||||
for value in j_ids:
|
||||
text = str(value).strip()
|
||||
numeric = text[2:] if text.upper().startswith("J-") else text
|
||||
if numeric.isdigit():
|
||||
numeric_ids.append(int(numeric))
|
||||
|
||||
for item in MediaItem.objects.filter(pk__in=numeric_ids).prefetch_related(
|
||||
"locations"
|
||||
):
|
||||
if not can_delete(request.user, item):
|
||||
errors.append({"j_id": f"J-{item.id}", "error": "permission denied"})
|
||||
continue
|
||||
for location in item.locations.all():
|
||||
remove_watched_file(location.path)
|
||||
deleted.append(f"J-{item.id}")
|
||||
item.delete()
|
||||
|
||||
cache.delete(STORAGE_CACHE_KEY)
|
||||
cache.delete_many(TAG_CLOUD_CACHE_KEYS)
|
||||
|
||||
for location in MediaLocation.objects.filter(
|
||||
id__in=location_ids
|
||||
).select_related("item"):
|
||||
if not can_delete(request.user, location.item):
|
||||
errors.append(
|
||||
{"location": location.id, "error": "permission denied"}
|
||||
)
|
||||
continue
|
||||
location_id = location.id
|
||||
item = location.item
|
||||
remove_watched_file(location.path)
|
||||
location.delete()
|
||||
if item.locations.exists():
|
||||
deleted.append(f"location {location_id}")
|
||||
else:
|
||||
deleted.append(f"J-{item.id}")
|
||||
item.delete()
|
||||
|
||||
return Response({"deleted": deleted, "errors": errors})
|
||||
|
||||
|
||||
class ClearTempView(APIView):
|
||||
"""Remove staged files from the temp upload folder."""
|
||||
|
||||
permission_classes = [CanUpload]
|
||||
|
||||
def post(self, request):
|
||||
temp_dir = Path(settings.MEDIA_ROOT) / "uploads" / "temp"
|
||||
removed = 0
|
||||
if temp_dir.exists():
|
||||
for entry in temp_dir.iterdir():
|
||||
if entry.is_file():
|
||||
entry.unlink()
|
||||
removed += 1
|
||||
cache.delete(STORAGE_CACHE_KEY)
|
||||
return Response({"removed": removed})
|
||||
|
||||
|
||||
class TagCloudView(APIView):
|
||||
"""Most-used tags across the library (custom + e621 tags)."""
|
||||
|
||||
permission_classes = [AllowAny]
|
||||
|
||||
def get(self, request):
|
||||
cache_key = (
|
||||
"j621.library.tag_cloud.auth"
|
||||
if request.user.is_authenticated
|
||||
else "j621.library.tag_cloud.guest"
|
||||
)
|
||||
cached = cache.get(cache_key)
|
||||
if cached is not None:
|
||||
return Response(cached)
|
||||
|
||||
queryset = MediaItem.objects.all()
|
||||
if not request.user.is_authenticated:
|
||||
queryset = queryset.filter(hidden_from_guests=False)
|
||||
|
||||
counts = {}
|
||||
for item in queryset.only("tags", "e621_data").iterator(chunk_size=500):
|
||||
names = set()
|
||||
for tag in item.tags or []:
|
||||
name = str(tag).strip().lower()
|
||||
if name:
|
||||
names.add(name)
|
||||
data = item.e621_data or {}
|
||||
categories = data.get("tags") if isinstance(data, dict) else None
|
||||
if isinstance(categories, dict):
|
||||
for values in categories.values():
|
||||
if isinstance(values, list):
|
||||
names.update(
|
||||
str(tag).strip().lower()
|
||||
for tag in values
|
||||
if str(tag).strip()
|
||||
)
|
||||
for name in names:
|
||||
counts[name] = counts.get(name, 0) + 1
|
||||
|
||||
ranked = sorted(counts.items(), key=lambda entry: (-entry[1], entry[0]))
|
||||
payload = {
|
||||
"count": len(counts),
|
||||
"tags": [
|
||||
{"tag": tag, "count": count} for tag, count in ranked[:120]
|
||||
],
|
||||
}
|
||||
cache.set(cache_key, payload, 120)
|
||||
return Response(payload)
|
||||
|
||||
|
||||
class StorageView(APIView):
|
||||
"""Disk usage for the watched folder, media root and temp uploads."""
|
||||
|
||||
permission_classes = [IsAuthenticated]
|
||||
|
||||
def get(self, request):
|
||||
cached = cache.get(STORAGE_CACHE_KEY)
|
||||
if cached is not None:
|
||||
return Response(cached)
|
||||
|
||||
watched = Path(settings.WATCHED_FOLDER)
|
||||
media_root = Path(settings.MEDIA_ROOT)
|
||||
temp_dir = media_root / "uploads" / "temp"
|
||||
|
||||
usage = shutil.disk_usage(
|
||||
str(watched) if watched.exists() else str(Path(settings.BASE_DIR))
|
||||
)
|
||||
library_size = (
|
||||
sum(file.stat().st_size for file in watched.rglob("*") if file.is_file())
|
||||
if watched.exists()
|
||||
else 0
|
||||
)
|
||||
media_size = (
|
||||
sum(file.stat().st_size for file in media_root.rglob("*") if file.is_file())
|
||||
if media_root.exists()
|
||||
else 0
|
||||
)
|
||||
temp_files = (
|
||||
[file for file in temp_dir.rglob("*") if file.is_file()]
|
||||
if temp_dir.exists()
|
||||
else []
|
||||
)
|
||||
|
||||
payload = {
|
||||
"watched_folder": {
|
||||
"path": str(watched),
|
||||
"total": usage.total,
|
||||
"used": usage.used,
|
||||
"free": usage.free,
|
||||
"library_size": library_size,
|
||||
"percent_used": (
|
||||
round(usage.used / usage.total * 100, 1) if usage.total else 0
|
||||
),
|
||||
},
|
||||
"media": {"path": str(media_root), "size": media_size},
|
||||
"temp": {
|
||||
"path": str(temp_dir),
|
||||
"size": sum(file.stat().st_size for file in temp_files),
|
||||
"files": len(temp_files),
|
||||
},
|
||||
"library_items": MediaItem.objects.count(),
|
||||
}
|
||||
cache.set(STORAGE_CACHE_KEY, payload, STORAGE_CACHE_TTL)
|
||||
return Response(payload)
|
||||
@@ -0,0 +1,329 @@
|
||||
"""Staged uploads: files land in a temp folder until they are resolved.
|
||||
|
||||
Flow:
|
||||
- POST /api/uploads/ stage a file (computes MD5, detects duplicates)
|
||||
- GET /api/uploads/ list my staged uploads
|
||||
- GET /api/uploads/<id>/file/ serve the staged file (previews, IQDB)
|
||||
- POST /api/uploads/<id>/iqdb/ store IQDB candidates -> visual_match
|
||||
- POST /api/uploads/<id>/resolve/ link to a post or apply custom metadata
|
||||
- DELETE /api/uploads/<id>/ discard a staged upload / dismiss a record
|
||||
"""
|
||||
|
||||
import logging
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from django.conf import settings
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.core import signing
|
||||
from django.http import Http404
|
||||
from django.utils import timezone
|
||||
from rest_framework import mixins, status, viewsets
|
||||
from rest_framework.decorators import action
|
||||
from rest_framework.parsers import FormParser, JSONParser, MultiPartParser
|
||||
from rest_framework.permissions import AllowAny
|
||||
from rest_framework.response import Response
|
||||
|
||||
from . import services
|
||||
from .models import MediaItem, TempUpload
|
||||
from .permissions import CanUpload
|
||||
from .serializers import TempUploadSerializer
|
||||
from .tools import HASH_FIELDS, hashes_similarity
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def find_library_matches(path, limit=10, user=None):
|
||||
"""Library items visually similar to a staged file."""
|
||||
hashes = services.compute_visual_hashes(path)
|
||||
if not hashes:
|
||||
return []
|
||||
algorithms = list(HASH_FIELDS)
|
||||
threshold = settings.VISUAL_MATCH_THRESHOLD
|
||||
matches = []
|
||||
for item in MediaItem.objects.prefetch_related("locations"):
|
||||
similarity = hashes_similarity(
|
||||
hashes,
|
||||
{field: getattr(item, field, "") for field in algorithms},
|
||||
algorithms,
|
||||
threshold,
|
||||
)
|
||||
if similarity is None:
|
||||
continue
|
||||
location = item.locations.first()
|
||||
matches.append(
|
||||
{
|
||||
"j_id": f"J-{item.id}",
|
||||
"filename": Path(location.rel_path).name if location else item.md5,
|
||||
"similarity": round(similarity * 100, 1),
|
||||
"thumbnail_url": services.signed_media_url(item, user, "thumbnail"),
|
||||
}
|
||||
)
|
||||
matches.sort(key=lambda entry: entry["similarity"], reverse=True)
|
||||
return matches[:limit]
|
||||
|
||||
|
||||
def complete_temp_upload(temp, download_url=None):
|
||||
"""Index the upload into the library.
|
||||
|
||||
With a ``download_url`` the e621 original is fetched and the staged copy
|
||||
is dropped; otherwise the staged file itself moves into the library.
|
||||
"""
|
||||
folder = Path(settings.WATCHED_FOLDER)
|
||||
|
||||
if download_url:
|
||||
file_data = {}
|
||||
if isinstance(temp.e621_data, dict):
|
||||
file_data = temp.e621_data.get("file") or {}
|
||||
extension = Path(urlparse(download_url).path).suffix
|
||||
if not extension:
|
||||
extension = Path(temp.original_filename).suffix
|
||||
md5 = file_data.get("md5")
|
||||
name = f"{md5}{extension}" if md5 else (temp.original_filename or "download")
|
||||
destination = services.unique_destination(folder, name)
|
||||
try:
|
||||
services.download_file(download_url, destination)
|
||||
except Exception:
|
||||
destination.unlink(missing_ok=True)
|
||||
raise
|
||||
item, _, location, _ = services.index_file(destination, folder)
|
||||
services.rename_location_to_j_id(item, location)
|
||||
services.ensure_visual_hashes(item)
|
||||
if temp.file:
|
||||
temp.file.delete(save=False)
|
||||
else:
|
||||
if not temp.file:
|
||||
raise ValueError("staged file is missing")
|
||||
destination = services.unique_destination(
|
||||
folder, temp.original_filename or "upload"
|
||||
)
|
||||
with temp.file.open("rb") as source, open(destination, "wb") as target:
|
||||
shutil.copyfileobj(source, target)
|
||||
item, _, location, _ = services.index_file(destination, folder)
|
||||
services.rename_location_to_j_id(item, location)
|
||||
services.ensure_visual_hashes(item)
|
||||
temp.file.delete(save=False)
|
||||
|
||||
temp.library_item = item
|
||||
temp.status = TempUpload.STATUS_COMPLETED
|
||||
|
||||
update_fields = []
|
||||
if item.uploaded_by_id is None:
|
||||
item.uploaded_by = temp.user
|
||||
update_fields.append("uploaded_by")
|
||||
if temp.e621_post_id is not None:
|
||||
item.e621_post_id = temp.e621_post_id
|
||||
item.e621_data = temp.e621_data
|
||||
item.e621_match_status = MediaItem.E621_MATCHED
|
||||
item.e621_checked_at = timezone.now()
|
||||
update_fields += [
|
||||
"e621_post_id",
|
||||
"e621_data",
|
||||
"e621_match_status",
|
||||
"e621_checked_at",
|
||||
]
|
||||
if temp.custom_rating in {"s", "q", "e"} and not item.rating:
|
||||
item.rating = temp.custom_rating
|
||||
update_fields.append("rating")
|
||||
if temp.custom_tags:
|
||||
item.tags = temp.custom_tags
|
||||
update_fields.append("tags")
|
||||
if temp.custom_notes:
|
||||
item.notes = temp.custom_notes
|
||||
update_fields.append("notes")
|
||||
if update_fields:
|
||||
item.save(update_fields=update_fields + ["updated_at"])
|
||||
|
||||
temp.save()
|
||||
return item
|
||||
|
||||
|
||||
class TempUploadViewSet(
|
||||
mixins.ListModelMixin,
|
||||
mixins.RetrieveModelMixin,
|
||||
mixins.DestroyModelMixin,
|
||||
viewsets.GenericViewSet,
|
||||
):
|
||||
serializer_class = TempUploadSerializer
|
||||
permission_classes = [CanUpload]
|
||||
parser_classes = [MultiPartParser, FormParser, JSONParser]
|
||||
http_method_names = ["get", "post", "delete", "head", "options"]
|
||||
|
||||
def get_queryset(self):
|
||||
queryset = TempUpload.objects.select_related("library_item")
|
||||
user = self.request.user
|
||||
if not (user.is_staff or user.is_superuser):
|
||||
queryset = queryset.filter(user=user)
|
||||
return queryset
|
||||
|
||||
def create(self, request):
|
||||
upload = request.FILES.get("file")
|
||||
if upload is None:
|
||||
return Response(
|
||||
{"detail": "A file is required."}, status=status.HTTP_400_BAD_REQUEST
|
||||
)
|
||||
extension = Path(upload.name).suffix.lower()
|
||||
if extension not in services.ALLOWED_EXTENSIONS:
|
||||
return Response(
|
||||
{"detail": f"Unsupported file type: {extension or 'unknown'}"},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
|
||||
temp = TempUpload.objects.create(
|
||||
user=request.user,
|
||||
file=upload,
|
||||
original_filename=upload.name,
|
||||
size=upload.size,
|
||||
)
|
||||
temp.md5 = services.compute_md5(temp.file.path)
|
||||
|
||||
existing = MediaItem.objects.filter(md5=temp.md5).first()
|
||||
if existing is not None:
|
||||
temp.status = TempUpload.STATUS_COMPLETED
|
||||
temp.resolution = TempUpload.RESOLUTION_DUPLICATE
|
||||
temp.library_item = existing
|
||||
temp.file.delete(save=False)
|
||||
else:
|
||||
matches = find_library_matches(temp.file.path, user=request.user)
|
||||
if matches:
|
||||
temp.visual_matches = matches
|
||||
temp.status = TempUpload.STATUS_VISUAL_MATCH
|
||||
temp.save()
|
||||
return Response(
|
||||
self.get_serializer(temp).data, status=status.HTTP_201_CREATED
|
||||
)
|
||||
|
||||
@action(detail=True, methods=["get", "head"], permission_classes=[AllowAny])
|
||||
def file(self, request, pk=None):
|
||||
"""Serve the staged file; accepts a signed URL for media tags."""
|
||||
user = request.user if request.user.is_authenticated else None
|
||||
if user is None:
|
||||
signature = request.query_params.get("sig")
|
||||
if signature:
|
||||
try:
|
||||
payload = signing.loads(
|
||||
signature,
|
||||
salt=services.UPLOAD_FILE_SALT,
|
||||
max_age=86400,
|
||||
)
|
||||
except signing.BadSignature:
|
||||
payload = None
|
||||
if payload and str(payload.get("temp")) == str(pk):
|
||||
user = (
|
||||
get_user_model()
|
||||
.objects.filter(pk=payload.get("user"))
|
||||
.first()
|
||||
)
|
||||
if user is None or not user.can_upload:
|
||||
return Response(
|
||||
{"detail": "Authentication required."},
|
||||
status=status.HTTP_401_UNAUTHORIZED,
|
||||
)
|
||||
|
||||
temp = TempUpload.objects.filter(pk=pk).first()
|
||||
is_owner = temp is not None and temp.user_id == user.id
|
||||
if temp is None or not (
|
||||
is_owner or user.is_staff or user.is_superuser
|
||||
):
|
||||
raise Http404
|
||||
if not temp.file:
|
||||
raise Http404
|
||||
return services.serve_file(request, temp.file.path)
|
||||
|
||||
@action(detail=True, methods=["post"])
|
||||
def iqdb(self, request, pk=None):
|
||||
temp = self.get_object()
|
||||
results = request.data.get("results")
|
||||
if not isinstance(results, list):
|
||||
return Response(
|
||||
{"detail": "results must be a list."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
temp.iqdb_data = services.sanitize_iqdb_results(results)
|
||||
if temp.status == TempUpload.STATUS_PENDING and temp.iqdb_data:
|
||||
temp.status = TempUpload.STATUS_VISUAL_MATCH
|
||||
temp.save(update_fields=["iqdb_data", "status", "updated_at"])
|
||||
return Response(self.get_serializer(temp).data)
|
||||
|
||||
@action(detail=True, methods=["post"])
|
||||
def resolve(self, request, pk=None):
|
||||
temp = self.get_object()
|
||||
if temp.status == TempUpload.STATUS_COMPLETED:
|
||||
return Response(
|
||||
{"detail": "This upload is already in the library."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
|
||||
mode = str(request.data.get("mode") or "").strip()
|
||||
download_url = None
|
||||
if mode == "link":
|
||||
post = request.data.get("post")
|
||||
post_id = request.data.get("post_id")
|
||||
trimmed = services.trim_e621_post(post)
|
||||
has_numeric_id = post_id is not None and str(post_id).isdigit()
|
||||
if trimmed is None and not has_numeric_id:
|
||||
return Response(
|
||||
{"detail": "A post payload or post_id is required."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
if trimmed is not None and trimmed.get("id") is not None:
|
||||
temp.e621_post_id = int(trimmed["id"])
|
||||
elif has_numeric_id:
|
||||
temp.e621_post_id = int(post_id)
|
||||
temp.e621_data = trimmed
|
||||
temp.resolution = (
|
||||
TempUpload.RESOLUTION_AUTO_MD5
|
||||
if request.data.get("auto")
|
||||
else TempUpload.RESOLUTION_LINKED
|
||||
)
|
||||
rating = str(request.data.get("rating") or "")
|
||||
if rating in {"s", "q", "e"}:
|
||||
temp.custom_rating = rating
|
||||
tags = services.parse_tags(request.data.get("tags"))
|
||||
if tags is not None:
|
||||
temp.custom_tags = tags
|
||||
notes = request.data.get("notes")
|
||||
if notes is not None:
|
||||
temp.custom_notes = str(notes)
|
||||
|
||||
# Prefer the e621 original; fall back to the staged file when the
|
||||
# post has no URL or its file is byte-identical to the upload.
|
||||
file_data = (trimmed or {}).get("file") or {}
|
||||
candidate_url = str(
|
||||
request.data.get("file_url") or file_data.get("url") or ""
|
||||
).strip()
|
||||
post_md5 = str(file_data.get("md5") or "").strip().lower()
|
||||
if candidate_url and post_md5 and post_md5 == temp.md5.lower():
|
||||
candidate_url = ""
|
||||
download_url = candidate_url or None
|
||||
elif mode == "custom":
|
||||
rating = str(request.data.get("rating") or "")
|
||||
temp.custom_rating = rating if rating in {"s", "q", "e"} else ""
|
||||
temp.custom_tags = services.parse_tags(request.data.get("tags")) or []
|
||||
temp.custom_notes = str(request.data.get("notes") or "")
|
||||
temp.resolution = TempUpload.RESOLUTION_CUSTOM
|
||||
else:
|
||||
return Response(
|
||||
{"detail": "mode must be 'link' or 'custom'."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
|
||||
temp.save()
|
||||
try:
|
||||
complete_temp_upload(temp, download_url=download_url)
|
||||
except Exception as exc: # noqa: BLE001 - report completion failures
|
||||
logger.exception("Could not complete staged upload %s", temp.id)
|
||||
temp.status = TempUpload.STATUS_ERROR
|
||||
temp.save(update_fields=["status", "updated_at"])
|
||||
return Response(
|
||||
{"detail": f"Could not finish the upload: {exc}"},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
temp.refresh_from_db()
|
||||
return Response(self.get_serializer(temp).data)
|
||||
|
||||
def perform_destroy(self, instance):
|
||||
if instance.file:
|
||||
instance.file.delete(save=False)
|
||||
instance.delete()
|
||||
@@ -1,17 +1,49 @@
|
||||
from django.urls import include, path
|
||||
from rest_framework.routers import DefaultRouter
|
||||
|
||||
from .views import DownloadToLibraryView, MediaItemViewSet, UploadView
|
||||
from .tools import (
|
||||
ClearTempView,
|
||||
DeleteFilesView,
|
||||
ExactDuplicatesView,
|
||||
StorageView,
|
||||
TagCloudView,
|
||||
VisualGroupsView,
|
||||
VisualMatchesView,
|
||||
)
|
||||
from .uploads import TempUploadViewSet
|
||||
from .views import (
|
||||
ClientDownloadView,
|
||||
DownloadTaskViewSet,
|
||||
MatchTaskViewSet,
|
||||
MediaItemViewSet,
|
||||
)
|
||||
|
||||
router = DefaultRouter()
|
||||
router.register("files", MediaItemViewSet, basename="file")
|
||||
router.register("uploads", TempUploadViewSet, basename="upload")
|
||||
router.register("online/downloads", DownloadTaskViewSet, basename="download")
|
||||
router.register("matches", MatchTaskViewSet, basename="match")
|
||||
|
||||
urlpatterns = [
|
||||
path("", include(router.urls)),
|
||||
path("uploads/", UploadView.as_view(), name="upload"),
|
||||
path("online/file/", ClientDownloadView.as_view(), name="client_download"),
|
||||
path(
|
||||
"online/downloads/",
|
||||
DownloadToLibraryView.as_view(),
|
||||
name="download_to_library",
|
||||
"duplicates/md5/",
|
||||
ExactDuplicatesView.as_view(),
|
||||
name="duplicates_md5",
|
||||
),
|
||||
path(
|
||||
"duplicates/visual/",
|
||||
VisualMatchesView.as_view(),
|
||||
name="duplicates_visual",
|
||||
),
|
||||
path(
|
||||
"duplicates/visual_groups/",
|
||||
VisualGroupsView.as_view(),
|
||||
name="duplicates_groups",
|
||||
),
|
||||
path("delete/", DeleteFilesView.as_view(), name="delete_files"),
|
||||
path("temp/clear/", ClearTempView.as_view(), name="clear_temp"),
|
||||
path("tags/cloud/", TagCloudView.as_view(), name="tag_cloud"),
|
||||
path("storage/", StorageView.as_view(), name="storage_info"),
|
||||
]
|
||||
|
||||
+282
-92
@@ -3,20 +3,28 @@ from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from django.conf import settings
|
||||
from django.db.models import Min
|
||||
from django.http import Http404
|
||||
from django.core import signing
|
||||
from django.db.models import Min, Q
|
||||
from django.http import Http404, StreamingHttpResponse
|
||||
from django.shortcuts import get_object_or_404
|
||||
from django.utils import timezone
|
||||
from django.utils.text import get_valid_filename
|
||||
from rest_framework import mixins, status, viewsets
|
||||
from rest_framework.decorators import action
|
||||
from rest_framework.parsers import FormParser, MultiPartParser
|
||||
from rest_framework.permissions import AllowAny, IsAuthenticatedOrReadOnly
|
||||
from rest_framework.response import Response
|
||||
from rest_framework.views import APIView
|
||||
|
||||
from . import services
|
||||
from .models import MediaItem
|
||||
from . import e621, matching, services
|
||||
from .downloads import reap_stale_downloads, start_download_task
|
||||
from .matching import reap_stale_match_tasks, start_match_task
|
||||
from .models import DownloadTask, MatchTask, MediaItem
|
||||
from .permissions import CanUpload, IsUploaderOrStaffOrReadOnly
|
||||
from .serializers import MediaItemSerializer
|
||||
from .serializers import (
|
||||
DownloadTaskSerializer,
|
||||
MatchTaskSerializer,
|
||||
MediaItemSerializer,
|
||||
)
|
||||
|
||||
LIST_ORDERINGS = {"name", "-name", "size", "-size", "created_at", "-created_at"}
|
||||
MD5_RE = re.compile(r"[0-9a-fA-F]{32}")
|
||||
@@ -38,9 +46,30 @@ class MediaItemViewSet(
|
||||
)
|
||||
if not self.request.user.is_authenticated:
|
||||
queryset = queryset.filter(hidden_from_guests=False)
|
||||
j_ids = self.request.query_params.get("j_ids", "").strip()
|
||||
if j_ids:
|
||||
numeric_ids = []
|
||||
for value in j_ids.split(","):
|
||||
text = value.strip()
|
||||
number = text[2:] if text.upper().startswith("J-") else text
|
||||
if number.isdigit():
|
||||
numeric_ids.append(int(number))
|
||||
queryset = queryset.filter(pk__in=numeric_ids)
|
||||
search = self.request.query_params.get("search", "").strip()
|
||||
search_type = self.request.query_params.get("search_type", "filename").strip()
|
||||
if search:
|
||||
queryset = queryset.filter(locations__rel_path__icontains=search)
|
||||
words = [word for word in search.lower().split() if word]
|
||||
if search_type == "tags":
|
||||
for word in words:
|
||||
queryset = queryset.filter(search_tags__icontains=word)
|
||||
elif search_type == "both":
|
||||
query = Q(locations__rel_path__icontains=search)
|
||||
tag_query = Q()
|
||||
for word in words:
|
||||
tag_query &= Q(search_tags__icontains=word)
|
||||
queryset = queryset.filter(query | tag_query)
|
||||
else:
|
||||
queryset = queryset.filter(locations__rel_path__icontains=search)
|
||||
ratings = [
|
||||
value
|
||||
for value in self.request.query_params.get("rating", "").split(",")
|
||||
@@ -48,6 +77,28 @@ class MediaItemViewSet(
|
||||
]
|
||||
if ratings:
|
||||
queryset = queryset.filter(rating__in=ratings)
|
||||
statuses = []
|
||||
for value in self.request.query_params.getlist("status"):
|
||||
statuses.extend(
|
||||
part.strip() for part in value.split(",") if part.strip()
|
||||
)
|
||||
if statuses:
|
||||
status_query = Q()
|
||||
for status_value in set(statuses):
|
||||
if status_value == "custom":
|
||||
status_query |= Q(has_custom_data=True)
|
||||
elif status_value == "matched":
|
||||
status_query |= Q(e621_match_status=MediaItem.E621_MATCHED)
|
||||
elif status_value == "not_found":
|
||||
status_query |= Q(e621_match_status=MediaItem.E621_NOT_FOUND)
|
||||
elif status_value == "deleted":
|
||||
status_query |= Q(e621_match_status=MediaItem.E621_DELETED)
|
||||
elif status_value == "unknown":
|
||||
status_query |= Q(
|
||||
has_custom_data=False,
|
||||
e621_match_status=MediaItem.E621_UNKNOWN,
|
||||
)
|
||||
queryset = queryset.filter(status_query)
|
||||
ordering = self.request.query_params.get("ordering", "").strip()
|
||||
queryset = queryset.order_by(
|
||||
ordering if ordering in LIST_ORDERINGS else "-created_at"
|
||||
@@ -68,9 +119,32 @@ class MediaItemViewSet(
|
||||
self.check_object_permissions(self.request, obj)
|
||||
return obj
|
||||
|
||||
def _signed_media_item(self, request, action_name):
|
||||
"""Signed media URLs let <img>/<video> tags bypass guest filtering."""
|
||||
if request.user.is_authenticated:
|
||||
return None
|
||||
signature = request.query_params.get("sig")
|
||||
if not signature:
|
||||
return None
|
||||
try:
|
||||
payload = signing.loads(
|
||||
signature, salt=services.MEDIA_FILE_SALT, max_age=86400
|
||||
)
|
||||
except signing.BadSignature:
|
||||
return None
|
||||
if payload.get("action") != action_name:
|
||||
return None
|
||||
return MediaItem.objects.filter(pk=payload.get("item")).first()
|
||||
|
||||
def _media_object(self, request, action_name):
|
||||
item = self._signed_media_item(request, action_name)
|
||||
if item is not None:
|
||||
return item
|
||||
return self.get_object()
|
||||
|
||||
@action(detail=True, methods=["get"])
|
||||
def raw(self, request, pk=None):
|
||||
item = self.get_object()
|
||||
item = self._media_object(request, "raw")
|
||||
location = item.locations.first()
|
||||
if location is None:
|
||||
return Response(
|
||||
@@ -83,7 +157,7 @@ class MediaItemViewSet(
|
||||
|
||||
@action(detail=True, methods=["get"])
|
||||
def thumbnail(self, request, pk=None):
|
||||
item = self.get_object()
|
||||
item = self._media_object(request, "thumbnail")
|
||||
location = item.locations.first()
|
||||
if location is None:
|
||||
return Response(
|
||||
@@ -164,108 +238,224 @@ class MediaItemViewSet(
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class UploadView(APIView):
|
||||
permission_classes = [CanUpload]
|
||||
parser_classes = [MultiPartParser, FormParser]
|
||||
|
||||
def post(self, request):
|
||||
upload = request.FILES.get("file")
|
||||
if upload is None:
|
||||
return Response(
|
||||
{"detail": "A file is required."}, status=status.HTTP_400_BAD_REQUEST
|
||||
)
|
||||
folder = Path(settings.WATCHED_FOLDER)
|
||||
destination = services.unique_destination(folder, upload.name)
|
||||
with open(destination, "wb") as output:
|
||||
for chunk in upload.chunks():
|
||||
output.write(chunk)
|
||||
try:
|
||||
item, created_item, _ = services.index_file(destination, folder)
|
||||
except Exception as exc: # noqa: BLE001 - report indexing failures
|
||||
destination.unlink(missing_ok=True)
|
||||
return Response(
|
||||
{"detail": f"Could not index upload: {exc}"},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
|
||||
update_fields = []
|
||||
if created_item and item.uploaded_by_id is None:
|
||||
item.uploaded_by = request.user
|
||||
update_fields.append("uploaded_by")
|
||||
rating = request.data.get("rating") or ""
|
||||
if rating in {"s", "q", "e"} and item.rating != rating:
|
||||
item.rating = rating
|
||||
update_fields.append("rating")
|
||||
tags = services.parse_tags(request.data.get("tags"))
|
||||
if tags is not None:
|
||||
item.tags = tags
|
||||
update_fields.append("tags")
|
||||
notes = request.data.get("notes")
|
||||
if notes is not None:
|
||||
item.notes = str(notes)
|
||||
update_fields.append("notes")
|
||||
if update_fields:
|
||||
item.save(update_fields=update_fields + ["updated_at"])
|
||||
|
||||
return Response(
|
||||
MediaItemSerializer(item, context={"request": request}).data,
|
||||
status=status.HTTP_201_CREATED,
|
||||
def _can_match(self, request, item):
|
||||
user = request.user
|
||||
return bool(
|
||||
user.is_superuser
|
||||
or user.role == user.ROLE_STAFF
|
||||
or item.uploaded_by_id == user.id
|
||||
)
|
||||
|
||||
@action(detail=True, methods=["post"], permission_classes=[CanUpload])
|
||||
def match(self, request, pk=None):
|
||||
"""Link this item to e621: MD5 lookup, or an explicit post id."""
|
||||
item = self.get_object()
|
||||
if not self._can_match(request, item):
|
||||
return Response(
|
||||
{"detail": "Only the uploader or staff can match this item."},
|
||||
status=status.HTTP_403_FORBIDDEN,
|
||||
)
|
||||
extra = {}
|
||||
post_id = request.data.get("post_id")
|
||||
try:
|
||||
if post_id not in (None, ""):
|
||||
text = str(post_id).strip()
|
||||
if not text.isdigit():
|
||||
return Response(
|
||||
{"detail": "Post ID must be a number."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
extra["md5_match"] = matching.link_item_to_post(
|
||||
item, int(text), request.user
|
||||
)
|
||||
else:
|
||||
matching.check_item(item, request.user)
|
||||
except e621.E621NotFound:
|
||||
return Response(
|
||||
{"detail": "That post does not exist on e621."},
|
||||
status=status.HTTP_404_NOT_FOUND,
|
||||
)
|
||||
except e621.E621Error as exc:
|
||||
return Response(
|
||||
{"detail": str(exc)}, status=status.HTTP_502_BAD_GATEWAY
|
||||
)
|
||||
item.refresh_from_db()
|
||||
data = dict(self.get_serializer(item).data)
|
||||
data.update(extra)
|
||||
return Response(data)
|
||||
|
||||
class DownloadToLibraryView(APIView):
|
||||
"""Fetch an e621 file into the watched folder and index it."""
|
||||
@action(detail=True, methods=["post"], permission_classes=[CanUpload])
|
||||
def unlink(self, request, pk=None):
|
||||
"""Forget this item's e621 link so it can be matched again."""
|
||||
item = self.get_object()
|
||||
if not self._can_match(request, item):
|
||||
return Response(
|
||||
{"detail": "Only the uploader or staff can unlink this item."},
|
||||
status=status.HTTP_403_FORBIDDEN,
|
||||
)
|
||||
matching.unlink_item(item)
|
||||
return Response(self.get_serializer(item).data)
|
||||
|
||||
|
||||
class DownloadTaskViewSet(
|
||||
mixins.ListModelMixin,
|
||||
mixins.RetrieveModelMixin,
|
||||
viewsets.GenericViewSet,
|
||||
):
|
||||
"""Background 'Download to Library' jobs."""
|
||||
|
||||
serializer_class = DownloadTaskSerializer
|
||||
permission_classes = [CanUpload]
|
||||
http_method_names = ["get", "post", "head", "options"]
|
||||
|
||||
def post(self, request):
|
||||
def get_queryset(self):
|
||||
reap_stale_downloads()
|
||||
queryset = DownloadTask.objects.select_related("library_item")
|
||||
user = self.request.user
|
||||
if not (user.is_staff or user.is_superuser):
|
||||
queryset = queryset.filter(user=user)
|
||||
return queryset
|
||||
|
||||
def create(self, request):
|
||||
url = str(request.data.get("url") or "").strip()
|
||||
post_id = request.data.get("post_id")
|
||||
filename = str(request.data.get("filename") or "").strip()
|
||||
post_payload = request.data.get("post")
|
||||
if not url.startswith(("http://", "https://")):
|
||||
return Response(
|
||||
{"detail": "A valid file URL is required."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
folder = Path(settings.WATCHED_FOLDER)
|
||||
name = (
|
||||
filename
|
||||
or Path(urlparse(url).path).name
|
||||
or f"post-{post_id or 'download'}"
|
||||
trimmed = services.trim_e621_post(request.data.get("post"))
|
||||
resolved_post_id = None
|
||||
if trimmed is not None and trimmed.get("id") is not None:
|
||||
resolved_post_id = int(trimmed["id"])
|
||||
elif post_id is not None and str(post_id).isdigit():
|
||||
resolved_post_id = int(post_id)
|
||||
|
||||
task = DownloadTask.objects.create(
|
||||
user=request.user,
|
||||
post_id=resolved_post_id,
|
||||
url=url,
|
||||
filename=filename,
|
||||
e621_data=trimmed,
|
||||
)
|
||||
destination = services.unique_destination(folder, name)
|
||||
try:
|
||||
services.download_file(url, destination)
|
||||
item, _, _ = services.index_file(destination, folder)
|
||||
except Exception as exc: # noqa: BLE001 - report download failures
|
||||
destination.unlink(missing_ok=True)
|
||||
start_download_task(task.id)
|
||||
return Response(
|
||||
self.get_serializer(task).data, status=status.HTTP_202_ACCEPTED
|
||||
)
|
||||
|
||||
@action(detail=True, methods=["post"])
|
||||
def cancel(self, request, pk=None):
|
||||
task = self.get_object()
|
||||
if task.status not in {
|
||||
DownloadTask.STATUS_PENDING,
|
||||
DownloadTask.STATUS_DOWNLOADING,
|
||||
}:
|
||||
return Response(
|
||||
{"detail": f"Download failed: {exc}"},
|
||||
{"detail": "Task is not running."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
DownloadTask.objects.filter(id=task.id).update(
|
||||
cancelled=True, updated_at=timezone.now()
|
||||
)
|
||||
return Response({"success": True})
|
||||
|
||||
|
||||
class MatchTaskViewSet(
|
||||
mixins.ListModelMixin,
|
||||
mixins.RetrieveModelMixin,
|
||||
viewsets.GenericViewSet,
|
||||
):
|
||||
"""Background library-wide e621 match scans."""
|
||||
|
||||
serializer_class = MatchTaskSerializer
|
||||
permission_classes = [CanUpload]
|
||||
http_method_names = ["get", "post", "head", "options"]
|
||||
|
||||
def get_queryset(self):
|
||||
reap_stale_match_tasks()
|
||||
queryset = MatchTask.objects.all()
|
||||
user = self.request.user
|
||||
if not (user.is_staff or user.is_superuser):
|
||||
queryset = queryset.filter(user=user)
|
||||
return queryset
|
||||
|
||||
def create(self, request):
|
||||
scope = str(request.data.get("scope") or MatchTask.SCOPE_MISSING).strip()
|
||||
if scope not in {MatchTask.SCOPE_MISSING, MatchTask.SCOPE_ALL}:
|
||||
scope = MatchTask.SCOPE_MISSING
|
||||
task = MatchTask.objects.create(user=request.user, scope=scope)
|
||||
start_match_task(task.id)
|
||||
return Response(
|
||||
self.get_serializer(task).data, status=status.HTTP_202_ACCEPTED
|
||||
)
|
||||
|
||||
@action(detail=True, methods=["post"])
|
||||
def cancel(self, request, pk=None):
|
||||
task = self.get_object()
|
||||
if task.status not in {MatchTask.STATUS_PENDING, MatchTask.STATUS_RUNNING}:
|
||||
return Response(
|
||||
{"detail": "Task is not running."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
MatchTask.objects.filter(id=task.id).update(
|
||||
cancelled=True, updated_at=timezone.now()
|
||||
)
|
||||
return Response({"success": True})
|
||||
|
||||
|
||||
class ClientDownloadView(APIView):
|
||||
"""Stream an e621 file straight to the browser (no library write)."""
|
||||
|
||||
permission_classes = [AllowAny]
|
||||
|
||||
def get(self, request):
|
||||
url = str(request.query_params.get("url") or "").strip()
|
||||
filename = str(request.query_params.get("filename") or "").strip()
|
||||
parsed = urlparse(url)
|
||||
if (
|
||||
parsed.scheme not in {"http", "https"}
|
||||
or parsed.hostname not in settings.E621_MEDIA_HOSTS
|
||||
):
|
||||
return Response(
|
||||
{"detail": "URL not allowed."},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
|
||||
update_fields = []
|
||||
if item.uploaded_by_id is None:
|
||||
item.uploaded_by = request.user
|
||||
update_fields.append("uploaded_by")
|
||||
import requests
|
||||
|
||||
trimmed = services.trim_e621_post(post_payload)
|
||||
if trimmed is not None:
|
||||
if trimmed.get("id") is not None:
|
||||
item.e621_post_id = int(trimmed["id"])
|
||||
elif post_id and str(post_id).isdigit():
|
||||
item.e621_post_id = int(post_id)
|
||||
item.e621_data = trimmed
|
||||
update_fields += ["e621_post_id", "e621_data"]
|
||||
if not item.rating and trimmed.get("rating") in {"s", "q", "e"}:
|
||||
item.rating = trimmed["rating"]
|
||||
update_fields.append("rating")
|
||||
try:
|
||||
upstream = requests.get(
|
||||
url,
|
||||
headers={"User-Agent": settings.USER_AGENT},
|
||||
stream=True,
|
||||
timeout=60,
|
||||
)
|
||||
upstream.raise_for_status()
|
||||
except requests.RequestException as exc:
|
||||
return Response(
|
||||
{"detail": f"Could not fetch the file: {exc}"},
|
||||
status=status.HTTP_502_BAD_GATEWAY,
|
||||
)
|
||||
|
||||
if update_fields:
|
||||
item.save(update_fields=update_fields + ["updated_at"])
|
||||
return Response(
|
||||
MediaItemSerializer(item, context={"request": request}).data,
|
||||
status=status.HTTP_201_CREATED,
|
||||
content_type = upstream.headers.get(
|
||||
"Content-Type", "application/octet-stream"
|
||||
)
|
||||
name = get_valid_filename(
|
||||
filename or Path(parsed.path).name or "download"
|
||||
)
|
||||
|
||||
def stream():
|
||||
try:
|
||||
for chunk in upstream.iter_content(chunk_size=64 * 1024):
|
||||
if chunk:
|
||||
yield chunk
|
||||
finally:
|
||||
upstream.close()
|
||||
|
||||
response = StreamingHttpResponse(stream(), content_type=content_type)
|
||||
response["Content-Disposition"] = f'attachment; filename="{name}"'
|
||||
length = upstream.headers.get("Content-Length")
|
||||
if length:
|
||||
response["Content-Length"] = length
|
||||
return response
|
||||
|
||||
@@ -52,6 +52,7 @@ INSTALLED_APPS = [
|
||||
"django_filters",
|
||||
"apps.accounts",
|
||||
"apps.library",
|
||||
"apps.follows",
|
||||
"apps.core",
|
||||
]
|
||||
|
||||
@@ -144,6 +145,15 @@ WATCHED_FOLDER = str(WATCHED_FOLDER)
|
||||
|
||||
E621_BASE_URL = os.getenv("E621_BASE_URL", "https://e621.net").rstrip("/")
|
||||
USER_AGENT = os.getenv("USER_AGENT", "J621/0.1 (by J621 on e621)")
|
||||
# Hosts the client-download proxy is allowed to stream from.
|
||||
E621_MEDIA_HOSTS = [
|
||||
host.strip()
|
||||
for host in os.getenv(
|
||||
"E621_MEDIA_HOSTS",
|
||||
"static1.e621.net,static2.e621.net,static3.e621.net",
|
||||
).split(",")
|
||||
if host.strip()
|
||||
]
|
||||
|
||||
# Guest visibility: e621's anonymous default blacklist is mirrored into the
|
||||
# cache by `manage.py refresh_guest_blacklist`. This fallback is used until
|
||||
@@ -157,6 +167,9 @@ GUEST_BLACKLIST_FALLBACK = [
|
||||
]
|
||||
GUEST_BLACKLIST_TTL = int(os.getenv("GUEST_BLACKLIST_TTL", "3600"))
|
||||
|
||||
# Similarity threshold for flagging staged uploads that match library items.
|
||||
VISUAL_MATCH_THRESHOLD = float(os.getenv("VISUAL_MATCH_THRESHOLD", "0.9"))
|
||||
|
||||
# Redis cache (run via docker compose at the repo root), shared by web
|
||||
# workers and management commands (e.g. the mirrored guest blacklist).
|
||||
CACHES = {
|
||||
|
||||
@@ -6,8 +6,10 @@ from django.urls import include, path
|
||||
urlpatterns = [
|
||||
path("admin/", admin.site.urls),
|
||||
path("api/auth/", include("apps.accounts.urls")),
|
||||
path("api/", include("apps.accounts.api_urls")),
|
||||
path("api/", include("apps.core.urls")),
|
||||
path("api/", include("apps.library.urls")),
|
||||
path("api/", include("apps.follows.urls")),
|
||||
]
|
||||
|
||||
if settings.DEBUG:
|
||||
|
||||
@@ -4,5 +4,6 @@ django-filter>=25.1
|
||||
Pillow>=11.0
|
||||
python-dotenv>=1.0
|
||||
requests>=2.32
|
||||
imagehash>=4.3
|
||||
PyMySQL>=1.1
|
||||
redis>=5.0
|
||||
|
||||
+55
-1
@@ -1,4 +1,4 @@
|
||||
import { useEffect, type ReactNode } from "react";
|
||||
import { useEffect, useState, type ReactNode } from "react";
|
||||
import {
|
||||
Navigate,
|
||||
Route,
|
||||
@@ -10,13 +10,21 @@ import {
|
||||
import { AppShell } from "@/components/AppShell";
|
||||
import { FullPageSpinner } from "@/components/ui";
|
||||
import AccountPage from "@/features/account/AccountPage";
|
||||
import { AgeGate } from "@/features/age/AgeGate";
|
||||
import LoginPage from "@/features/auth/LoginPage";
|
||||
import RegisterPage from "@/features/auth/RegisterPage";
|
||||
import DetailPage from "@/features/detail/DetailPage";
|
||||
import DeletePage from "@/features/library/DeletePage";
|
||||
import DuplicatesPage from "@/features/library/DuplicatesPage";
|
||||
import FollowedPage from "@/features/follows/FollowedPage";
|
||||
import LibraryPage from "@/features/library/LibraryPage";
|
||||
import Md5Redirect from "@/features/library/Md5Redirect";
|
||||
import OnlinePage from "@/features/online/OnlinePage";
|
||||
import PoolDetailPage from "@/features/pools/PoolDetailPage";
|
||||
import PoolsPage from "@/features/pools/PoolsPage";
|
||||
import UploadPage from "@/features/upload/UploadPage";
|
||||
import UsersPage from "@/features/users/UsersPage";
|
||||
import { isAgeVerified, markAgeVerified } from "@/lib/age";
|
||||
import { useAuth } from "@/store/auth";
|
||||
|
||||
function RequireAuth({ children }: { children: ReactNode }) {
|
||||
@@ -36,11 +44,23 @@ function PostRedirect() {
|
||||
export default function App() {
|
||||
const ready = useAuth((state) => state.ready);
|
||||
const loadUser = useAuth((state) => state.loadUser);
|
||||
const [ageVerified, setAgeVerified] = useState(isAgeVerified);
|
||||
|
||||
useEffect(() => {
|
||||
void loadUser();
|
||||
}, [loadUser]);
|
||||
|
||||
if (!ageVerified) {
|
||||
return (
|
||||
<AgeGate
|
||||
onEnter={() => {
|
||||
markAgeVerified();
|
||||
setAgeVerified(true);
|
||||
}}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
if (!ready) {
|
||||
return <FullPageSpinner />;
|
||||
}
|
||||
@@ -53,6 +73,40 @@ export default function App() {
|
||||
<Route path="/view/:md5" element={<Md5Redirect />} />
|
||||
<Route path="/online" element={<OnlinePage />} />
|
||||
<Route path="/online/view/:postId" element={<PostRedirect />} />
|
||||
<Route path="/pools" element={<PoolsPage />} />
|
||||
<Route path="/pools/:poolId" element={<PoolDetailPage />} />
|
||||
<Route
|
||||
path="/followed"
|
||||
element={
|
||||
<RequireAuth>
|
||||
<FollowedPage />
|
||||
</RequireAuth>
|
||||
}
|
||||
/>
|
||||
<Route
|
||||
path="/duplicates"
|
||||
element={
|
||||
<RequireAuth>
|
||||
<DuplicatesPage />
|
||||
</RequireAuth>
|
||||
}
|
||||
/>
|
||||
<Route
|
||||
path="/delete"
|
||||
element={
|
||||
<RequireAuth>
|
||||
<DeletePage />
|
||||
</RequireAuth>
|
||||
}
|
||||
/>
|
||||
<Route
|
||||
path="/users"
|
||||
element={
|
||||
<RequireAuth>
|
||||
<UsersPage />
|
||||
</RequireAuth>
|
||||
}
|
||||
/>
|
||||
<Route
|
||||
path="/upload"
|
||||
element={
|
||||
|
||||
@@ -28,8 +28,8 @@ export function AppShell() {
|
||||
const statusQuery = useQuery({
|
||||
queryKey: ["status"],
|
||||
queryFn: () => api<SystemStatus>("/api/status/"),
|
||||
refetchInterval: 30_000,
|
||||
staleTime: 15_000,
|
||||
refetchInterval: 5_000,
|
||||
staleTime: 4_000,
|
||||
});
|
||||
const status = statusQuery.data;
|
||||
|
||||
@@ -60,7 +60,18 @@ export function AppShell() {
|
||||
const navItems = [
|
||||
{ to: "/", label: "Library" },
|
||||
{ to: "/online", label: "Online" },
|
||||
...(user?.can_upload ? [{ to: "/upload", label: "Upload" }] : []),
|
||||
{ to: "/pools", label: "Pools" },
|
||||
...(user ? [{ to: "/followed", label: "Followed" }] : []),
|
||||
...(user?.can_upload
|
||||
? [
|
||||
{ to: "/upload", label: "Upload" },
|
||||
{ to: "/duplicates", label: "Duplicates" },
|
||||
{ to: "/delete", label: "Delete" },
|
||||
]
|
||||
: []),
|
||||
...(user && (user.is_staff || user.is_superuser || user.role === "staff")
|
||||
? [{ to: "/users", label: "Users" }]
|
||||
: []),
|
||||
];
|
||||
|
||||
const navigationType = useNavigationType();
|
||||
@@ -120,9 +131,17 @@ export function AppShell() {
|
||||
title="Account settings"
|
||||
className="flex items-center gap-2 transition hover:opacity-80"
|
||||
>
|
||||
<span className="flex h-7 w-7 items-center justify-center rounded-full bg-ctp-surface1 font-mono text-xs font-semibold text-ctp-lavender">
|
||||
{user.username.slice(0, 1).toUpperCase()}
|
||||
</span>
|
||||
{user.avatar_url ? (
|
||||
<img
|
||||
src={user.avatar_url}
|
||||
alt={user.username}
|
||||
className="h-7 w-7 rounded-full border border-ctp-surface1 object-cover"
|
||||
/>
|
||||
) : (
|
||||
<span className="flex h-7 w-7 items-center justify-center rounded-full bg-ctp-surface1 font-mono text-xs font-semibold text-ctp-lavender">
|
||||
{user.username.slice(0, 1).toUpperCase()}
|
||||
</span>
|
||||
)}
|
||||
<span className="hidden text-sm text-ctp-subtext1 sm:block">
|
||||
{user.username}
|
||||
</span>
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import {
|
||||
Copy,
|
||||
FolderOpen,
|
||||
Globe,
|
||||
LogIn,
|
||||
@@ -6,7 +7,9 @@ import {
|
||||
Menu,
|
||||
Search,
|
||||
Settings,
|
||||
Trash2,
|
||||
Upload,
|
||||
Users,
|
||||
type LucideIcon,
|
||||
} from "lucide-react";
|
||||
import { useMemo, useState } from "react";
|
||||
@@ -59,6 +62,18 @@ function CommandPaletteDialog({ onClose }: { onClose: () => void }) {
|
||||
icon: Upload,
|
||||
run: () => navigate("/upload"),
|
||||
},
|
||||
{
|
||||
id: "duplicates",
|
||||
label: "Find duplicates",
|
||||
icon: Copy,
|
||||
run: () => navigate("/duplicates"),
|
||||
},
|
||||
{
|
||||
id: "delete",
|
||||
label: "Delete files & storage",
|
||||
icon: Trash2,
|
||||
run: () => navigate("/delete"),
|
||||
},
|
||||
]
|
||||
: []),
|
||||
{
|
||||
@@ -94,6 +109,14 @@ function CommandPaletteDialog({ onClose }: { onClose: () => void }) {
|
||||
icon: Settings,
|
||||
run: () => navigate("/account"),
|
||||
});
|
||||
if (user.is_staff || user.is_superuser || user.role === "staff") {
|
||||
list.push({
|
||||
id: "users",
|
||||
label: "Manage users",
|
||||
icon: Users,
|
||||
run: () => navigate("/users"),
|
||||
});
|
||||
}
|
||||
list.push({
|
||||
id: "logout",
|
||||
label: `Log out (${user.username})`,
|
||||
|
||||
@@ -18,9 +18,7 @@ const ratingLabels: Record<string, string> = {
|
||||
|
||||
export function MediaCard({ item }: { item: MediaItem }) {
|
||||
const preview =
|
||||
item.kind === "video"
|
||||
? `/api/files/${item.j_id}/thumbnail/`
|
||||
: `/api/files/${item.j_id}/raw/`;
|
||||
item.kind === "video" ? item.thumbnail_url : item.raw_url;
|
||||
const rating = item.display_rating;
|
||||
|
||||
return (
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { Laptop, Monitor, Terminal, type LucideIcon } from "lucide-react";
|
||||
import { useEffect, useRef, useState } from "react";
|
||||
|
||||
import { cn } from "@/lib/cn";
|
||||
import type { SystemStatus } from "@/lib/types";
|
||||
@@ -16,17 +17,45 @@ function formatMs(value: number): string {
|
||||
return `${value.toFixed(1)}ms`;
|
||||
}
|
||||
|
||||
function formatClock(at: number): string {
|
||||
return new Date(at).toLocaleTimeString();
|
||||
}
|
||||
|
||||
export function StatusPill({ status }: { status?: SystemStatus }) {
|
||||
const requestCount = useE621Metrics((state) => state.requestCount);
|
||||
const totalMs = useE621Metrics((state) => state.totalMs);
|
||||
const last = useE621Metrics((state) => state.last);
|
||||
const history = useE621Metrics((state) => state.history);
|
||||
const [open, setOpen] = useState(false);
|
||||
const containerRef = useRef<HTMLSpanElement>(null);
|
||||
|
||||
useEffect(() => {
|
||||
if (!open) return;
|
||||
function handlePointerDown(event: MouseEvent) {
|
||||
if (
|
||||
containerRef.current &&
|
||||
!containerRef.current.contains(event.target as Node)
|
||||
) {
|
||||
setOpen(false);
|
||||
}
|
||||
}
|
||||
function handleKeyDown(event: KeyboardEvent) {
|
||||
if (event.key === "Escape") setOpen(false);
|
||||
}
|
||||
document.addEventListener("mousedown", handlePointerDown);
|
||||
document.addEventListener("keydown", handleKeyDown);
|
||||
return () => {
|
||||
document.removeEventListener("mousedown", handlePointerDown);
|
||||
document.removeEventListener("keydown", handleKeyDown);
|
||||
};
|
||||
}, [open]);
|
||||
|
||||
if (!status) return null;
|
||||
const OsIcon = osIcons[status.os.icon] ?? Monitor;
|
||||
|
||||
return (
|
||||
<span
|
||||
className="hidden select-none items-center gap-1.5 rounded-full border border-ctp-surface0 bg-ctp-mantle px-2.5 py-1 font-mono text-[11px] text-ctp-overlay1 lg:flex"
|
||||
ref={containerRef}
|
||||
className="relative hidden select-none items-center gap-1.5 rounded-full border border-ctp-surface0 bg-ctp-mantle px-2.5 py-1 font-mono text-[11px] text-ctp-overlay1 lg:flex"
|
||||
title={`${status.os.name} ${status.os.release} · built from ${status.git_hash}`}
|
||||
>
|
||||
<span
|
||||
@@ -47,14 +76,19 @@ export function StatusPill({ status }: { status?: SystemStatus }) {
|
||||
{requestCount > 0 ? (
|
||||
<>
|
||||
<span className="text-ctp-surface2">·</span>
|
||||
<span
|
||||
className="text-ctp-teal"
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setOpen((value) => !value)}
|
||||
title={`${requestCount} e621 request${
|
||||
requestCount === 1 ? "" : "s"
|
||||
} this session · last ${last ? `${formatMs(last.ms)} (${last.path})` : "—"}`}
|
||||
} this session — click for details`}
|
||||
className={cn(
|
||||
"text-ctp-teal transition hover:underline",
|
||||
open && "underline",
|
||||
)}
|
||||
>
|
||||
e621: {formatMs(totalMs)}
|
||||
</span>
|
||||
</button>
|
||||
</>
|
||||
) : null}
|
||||
{status.e621_time_ms !== null ? (
|
||||
@@ -65,6 +99,47 @@ export function StatusPill({ status }: { status?: SystemStatus }) {
|
||||
</span>
|
||||
</>
|
||||
) : null}
|
||||
|
||||
{open && requestCount > 0 ? (
|
||||
<span className="absolute right-0 top-full z-50 mt-2 block w-80 cursor-default overflow-hidden rounded-lg border border-ctp-surface1 bg-ctp-surface0 text-left shadow-[0_4px_12px_rgba(17,17,27,0.6)]">
|
||||
<span className="flex items-center justify-between gap-2 border-b border-ctp-surface1 px-3 py-2 text-[11px] text-ctp-overlay0">
|
||||
<span className="font-sans">e621 requests this session</span>
|
||||
<span className="font-mono">
|
||||
{requestCount} · {formatMs(totalMs)}
|
||||
</span>
|
||||
</span>
|
||||
<span className="block max-h-72 overflow-y-auto py-1">
|
||||
{history.map((sample, index) => (
|
||||
<span
|
||||
key={`${sample.at}-${index}`}
|
||||
className="flex items-center gap-2 px-3 py-1 font-mono text-[11px]"
|
||||
>
|
||||
<span className="shrink-0 text-ctp-overlay0">
|
||||
{formatClock(sample.at)}
|
||||
</span>
|
||||
<span
|
||||
className="min-w-0 flex-1 truncate text-ctp-subtext0"
|
||||
title={sample.path}
|
||||
>
|
||||
{sample.path}
|
||||
</span>
|
||||
<span
|
||||
className={cn(
|
||||
"shrink-0",
|
||||
sample.ms >= 2000
|
||||
? "text-ctp-peach"
|
||||
: sample.ms >= 800
|
||||
? "text-ctp-yellow"
|
||||
: "text-ctp-teal",
|
||||
)}
|
||||
>
|
||||
{formatMs(sample.ms)}
|
||||
</span>
|
||||
</span>
|
||||
))}
|
||||
</span>
|
||||
</span>
|
||||
) : null}
|
||||
</span>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { cn } from "@/lib/cn";
|
||||
import { TagFollowToggle } from "@/features/follows/TagFollowToggle";
|
||||
|
||||
const TAG_STYLES: Record<string, string> = {
|
||||
general:
|
||||
@@ -26,11 +27,13 @@ export function TagChip({
|
||||
category,
|
||||
count,
|
||||
onClick,
|
||||
followable = false,
|
||||
}: {
|
||||
tag: string;
|
||||
category?: string;
|
||||
count?: number;
|
||||
onClick?: () => void;
|
||||
followable?: boolean;
|
||||
}) {
|
||||
const style = TAG_STYLES[category ?? "unknown"] ?? TAG_STYLES.unknown;
|
||||
const className = cn(
|
||||
@@ -39,6 +42,21 @@ export function TagChip({
|
||||
);
|
||||
const label = count !== undefined ? `${tag} (${count})` : tag;
|
||||
|
||||
if (followable) {
|
||||
return (
|
||||
<span className={cn(className, "inline-flex items-center gap-1")}>
|
||||
{onClick ? (
|
||||
<button type="button" onClick={onClick} className="text-left">
|
||||
{label}
|
||||
</button>
|
||||
) : (
|
||||
<span>{label}</span>
|
||||
)}
|
||||
<TagFollowToggle tag={tag} />
|
||||
</span>
|
||||
);
|
||||
}
|
||||
|
||||
if (!onClick) {
|
||||
return <span className={className}>{label}</span>;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import { Ban, ShieldAlert } from "lucide-react";
|
||||
import { useState } from "react";
|
||||
|
||||
import { Button } from "@/components/ui";
|
||||
|
||||
export function AgeGate({ onEnter }: { onEnter: () => void }) {
|
||||
const [declined, setDeclined] = useState(false);
|
||||
|
||||
if (declined) {
|
||||
return (
|
||||
<div className="flex min-h-screen items-center justify-center bg-ctp-crust px-4">
|
||||
<div className="w-full max-w-md rounded-lg border border-ctp-red/40 bg-ctp-base p-6 text-center">
|
||||
<Ban className="mx-auto h-7 w-7 text-ctp-red" />
|
||||
<h1 className="mt-3 text-lg font-semibold text-ctp-text">
|
||||
Access restricted
|
||||
</h1>
|
||||
<p className="mt-2 text-sm leading-relaxed text-ctp-subtext0">
|
||||
You have to be 18 years or older to view this archive. Come back
|
||||
when that applies.
|
||||
</p>
|
||||
<Button
|
||||
variant="secondary"
|
||||
className="mt-5"
|
||||
onClick={() => setDeclined(false)}
|
||||
>
|
||||
Go back
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex min-h-screen items-center justify-center bg-ctp-crust px-4">
|
||||
<div className="w-full max-w-lg">
|
||||
<div className="flex items-center gap-3">
|
||||
<span className="rounded bg-ctp-mauve px-2.5 py-1.5 font-mono text-sm font-bold tracking-wide text-ctp-crust">
|
||||
J621
|
||||
</span>
|
||||
<span className="font-mono text-[11px] uppercase tracking-widest text-ctp-overlay0">
|
||||
Self-hosted media archive
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<div className="mt-6 rounded-lg border border-ctp-surface0 bg-ctp-base p-6">
|
||||
<div className="flex items-start gap-3">
|
||||
<ShieldAlert className="mt-0.5 h-5 w-5 shrink-0 text-ctp-peach" />
|
||||
<h1 className="text-lg font-semibold leading-snug text-ctp-text">
|
||||
You have to be 18+ to access this site.
|
||||
</h1>
|
||||
</div>
|
||||
|
||||
<p className="mt-3 text-sm leading-relaxed text-ctp-subtext0">
|
||||
This archive contains adult material. By entering you confirm that
|
||||
you are at least 18 years old and that viewing adult content is
|
||||
legal where you live.
|
||||
</p>
|
||||
|
||||
<div className="mt-6 flex flex-col gap-3 sm:flex-row">
|
||||
<Button className="flex-1" autoFocus onClick={onEnter}>
|
||||
I am 18 or older — Enter
|
||||
</Button>
|
||||
<Button variant="secondary" onClick={() => setDeclined(true)}>
|
||||
Leave
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p className="mt-4 text-center font-mono text-[11px] text-ctp-overlay0">
|
||||
Local-first library · e621 integration · nothing leaves this server
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,527 @@
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Eye, StarOff } from "lucide-react";
|
||||
import { useState } from "react";
|
||||
import { Link } from "react-router-dom";
|
||||
|
||||
import { Button, EmptyState, Spinner } from "@/components/ui";
|
||||
import { api } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import { formatDate } from "@/lib/format";
|
||||
import type {
|
||||
FollowCloud,
|
||||
FollowCoverData,
|
||||
FollowedPool,
|
||||
FollowedTag,
|
||||
FollowFeed,
|
||||
Paginated,
|
||||
} from "@/lib/types";
|
||||
|
||||
const ratingPill: Record<string, string> = {
|
||||
s: "bg-ctp-green text-ctp-crust",
|
||||
q: "bg-ctp-peach text-ctp-crust",
|
||||
e: "bg-ctp-red text-ctp-crust",
|
||||
};
|
||||
|
||||
function invalidateFollows(queryClient: ReturnType<typeof useQueryClient>) {
|
||||
void queryClient.invalidateQueries({ queryKey: ["follows-tags"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["follows-pools"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["follows-feed"] });
|
||||
}
|
||||
|
||||
function unseenLabel(count: number): string {
|
||||
return count > 99 ? "99+ unseen" : `${count} unseen`;
|
||||
}
|
||||
|
||||
function FollowCard({
|
||||
title,
|
||||
meta,
|
||||
cover,
|
||||
unseen,
|
||||
syncedAt,
|
||||
selected,
|
||||
onSelect,
|
||||
onSeen,
|
||||
onUnfollow,
|
||||
}: {
|
||||
title: string;
|
||||
meta: string;
|
||||
cover: FollowCoverData | null;
|
||||
unseen: number;
|
||||
syncedAt: string | null;
|
||||
selected: boolean;
|
||||
onSelect: () => void;
|
||||
onSeen: () => void;
|
||||
onUnfollow: () => void;
|
||||
}) {
|
||||
const metaLine = [meta, syncedAt ? `synced ${formatDate(syncedAt)}` : null]
|
||||
.filter(Boolean)
|
||||
.join(" · ");
|
||||
|
||||
return (
|
||||
<div
|
||||
className={cn(
|
||||
"overflow-hidden rounded-lg border bg-ctp-base transition",
|
||||
selected
|
||||
? "border-ctp-mauve ring-2 ring-ctp-mauve/30"
|
||||
: "border-ctp-surface0 hover:border-ctp-surface2",
|
||||
)}
|
||||
>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onSelect}
|
||||
className="relative block aspect-[4/3] w-full overflow-hidden bg-ctp-mantle"
|
||||
title={selected ? "Clear the feed filter" : "Show only this in the feed"}
|
||||
>
|
||||
{cover?.preview_url ? (
|
||||
<img
|
||||
src={cover.preview_url}
|
||||
alt={title}
|
||||
loading="lazy"
|
||||
className="h-full w-full object-cover"
|
||||
/>
|
||||
) : (
|
||||
<span className="flex h-full w-full items-center justify-center text-xs text-ctp-overlay0">
|
||||
No cover yet — run a sync
|
||||
</span>
|
||||
)}
|
||||
{unseen > 0 ? (
|
||||
<span className="absolute right-2 top-2 rounded-full bg-ctp-red px-2 py-0.5 font-mono text-[10px] font-bold text-ctp-crust">
|
||||
{unseenLabel(unseen)}
|
||||
</span>
|
||||
) : null}
|
||||
{cover?.rating ? (
|
||||
<span
|
||||
className={cn(
|
||||
"absolute left-2 top-2 rounded-full px-1.5 py-0.5 font-mono text-[10px] font-semibold uppercase",
|
||||
ratingPill[cover.rating] ?? "bg-ctp-surface1 text-ctp-text",
|
||||
)}
|
||||
>
|
||||
{cover.rating}
|
||||
</span>
|
||||
) : null}
|
||||
</button>
|
||||
|
||||
<div className="p-3">
|
||||
<div className="flex items-start justify-between gap-2">
|
||||
<div className="min-w-0">
|
||||
<p
|
||||
className="truncate font-mono text-sm text-ctp-text"
|
||||
title={title}
|
||||
>
|
||||
{title}
|
||||
</p>
|
||||
{metaLine ? (
|
||||
<p className="truncate text-[11px] text-ctp-overlay0">
|
||||
{metaLine}
|
||||
</p>
|
||||
) : null}
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onUnfollow}
|
||||
title="Unfollow"
|
||||
className="shrink-0 rounded p-1 text-ctp-overlay0 transition hover:bg-ctp-red/15 hover:text-ctp-red"
|
||||
>
|
||||
<StarOff className="h-4 w-4" />
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="mt-2 flex items-center gap-2">
|
||||
<Button
|
||||
variant="secondary"
|
||||
className="px-2 py-1 text-xs"
|
||||
disabled={unseen === 0}
|
||||
onClick={onSeen}
|
||||
>
|
||||
<Eye className="h-3.5 w-3.5" /> Mark seen
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function BlacklistCloudPanel() {
|
||||
const query = useQuery({
|
||||
queryKey: ["follows-cloud"],
|
||||
queryFn: () => api<FollowCloud>("/api/follows/cloud/"),
|
||||
refetchInterval: (q) =>
|
||||
q.state.data?.status === "building" ? 2000 : false,
|
||||
});
|
||||
const cloud = query.data;
|
||||
|
||||
return (
|
||||
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
|
||||
<div className="flex flex-wrap items-center justify-between gap-2">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Blacklisted tags
|
||||
</h2>
|
||||
<p className="text-xs text-ctp-overlay0">
|
||||
{cloud?.source === "user"
|
||||
? "From your e621 blacklist"
|
||||
: "From e621's anonymous default blacklist"}
|
||||
{cloud ? ` · ${cloud.blacklist_count} entries` : ""}
|
||||
{cloud ? ` · ${cloud.posts} feed post(s) scanned` : ""}
|
||||
{cloud?.computed_at ? ` · computed ${formatDate(cloud.computed_at)}` : ""}
|
||||
{cloud?.status === "building" ? (
|
||||
<span className="ml-2 inline-flex items-center gap-1.5">
|
||||
<Spinner className="h-3 w-3" /> building…
|
||||
</span>
|
||||
) : null}
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{query.isPending ? (
|
||||
<div className="mt-3 flex justify-center py-4">
|
||||
<Spinner className="h-4 w-4" />
|
||||
</div>
|
||||
) : cloud && cloud.tags.length > 0 ? (
|
||||
<div className="mt-3 flex flex-wrap gap-1.5">
|
||||
{cloud.tags.map(([tag, count]) => (
|
||||
<span
|
||||
key={tag}
|
||||
className="rounded border border-ctp-red/30 bg-ctp-red/10 px-1.5 py-0.5 font-mono text-[11px] text-ctp-red"
|
||||
>
|
||||
{tag} ({count})
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
) : (
|
||||
<p className="mt-3 text-xs text-ctp-overlay0">
|
||||
No blacklisted tags in your feeds.
|
||||
</p>
|
||||
)}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
export default function FollowedPage() {
|
||||
const queryClient = useQueryClient();
|
||||
const [feedSource, setFeedSource] = useState<{
|
||||
kind: "all" | "tag" | "pool";
|
||||
id?: number;
|
||||
label?: string;
|
||||
}>({ kind: "all" });
|
||||
const [unseenOnly, setUnseenOnly] = useState(false);
|
||||
const [page, setPage] = useState(1);
|
||||
|
||||
const tagsQuery = useQuery({
|
||||
queryKey: ["follows-tags"],
|
||||
queryFn: () => api<Paginated<FollowedTag>>("/api/follows/tags/"),
|
||||
});
|
||||
const poolsQuery = useQuery({
|
||||
queryKey: ["follows-pools"],
|
||||
queryFn: () => api<Paginated<FollowedPool>>("/api/follows/pools/"),
|
||||
});
|
||||
|
||||
const feedQuery = useQuery({
|
||||
queryKey: [
|
||||
"follows-feed",
|
||||
feedSource.kind,
|
||||
feedSource.id ?? null,
|
||||
unseenOnly,
|
||||
page,
|
||||
],
|
||||
queryFn: () => {
|
||||
const params = new URLSearchParams({
|
||||
page: String(page),
|
||||
per_page: "48",
|
||||
});
|
||||
if (feedSource.kind === "tag") params.set("tag", String(feedSource.id));
|
||||
if (feedSource.kind === "pool") params.set("pool", String(feedSource.id));
|
||||
if (unseenOnly) params.set("unseen", "1");
|
||||
return api<FollowFeed>(`/api/follows/feed/?${params.toString()}`);
|
||||
},
|
||||
});
|
||||
|
||||
const seenFollow = useMutation({
|
||||
mutationFn: ({ kind, id }: { kind: "tag" | "pool"; id: number }) =>
|
||||
api(`/api/follows/${kind === "tag" ? "tags" : "pools"}/${id}/seen/`, {
|
||||
method: "POST",
|
||||
}),
|
||||
onSuccess: () => invalidateFollows(queryClient),
|
||||
});
|
||||
|
||||
const seenPost = useMutation({
|
||||
mutationFn: (postId: number) =>
|
||||
api(`/api/follows/posts/${postId}/seen/`, { method: "POST" }),
|
||||
onSuccess: () => invalidateFollows(queryClient),
|
||||
});
|
||||
|
||||
const unfollow = useMutation({
|
||||
mutationFn: ({ kind, id }: { kind: "tag" | "pool"; id: number }) =>
|
||||
api(`/api/follows/${kind === "tag" ? "tags" : "pools"}/${id}/`, {
|
||||
method: "DELETE",
|
||||
}),
|
||||
onSuccess: () => invalidateFollows(queryClient),
|
||||
});
|
||||
|
||||
function selectSource(source: {
|
||||
kind: "all" | "tag" | "pool";
|
||||
id?: number;
|
||||
label?: string;
|
||||
}) {
|
||||
setPage(1);
|
||||
setFeedSource((current) =>
|
||||
current.kind === source.kind && current.id === source.id
|
||||
? { kind: "all" }
|
||||
: source,
|
||||
);
|
||||
}
|
||||
|
||||
const tags = tagsQuery.data?.results ?? [];
|
||||
const pools = poolsQuery.data?.results ?? [];
|
||||
const feed = feedQuery.data;
|
||||
|
||||
return (
|
||||
<div className="mx-auto flex w-full max-w-6xl flex-col gap-6">
|
||||
<header>
|
||||
<h1 className="text-lg font-semibold">Followed</h1>
|
||||
<p className="mt-1 text-sm text-ctp-overlay0">
|
||||
New posts from your followed e621 tags and pools. Follow tags with
|
||||
the + on any tag chip in the online or library detail views, and
|
||||
pools from a pool page. The periodic commands refresh these feeds
|
||||
and their covers.
|
||||
</p>
|
||||
</header>
|
||||
|
||||
<section className="flex flex-col gap-3">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Tags{" "}
|
||||
<span className="font-mono text-[11px] text-ctp-overlay0">
|
||||
{tags.length}
|
||||
</span>
|
||||
</h2>
|
||||
{tagsQuery.isPending ? (
|
||||
<div className="flex justify-center py-8">
|
||||
<Spinner className="h-5 w-5" />
|
||||
</div>
|
||||
) : tags.length === 0 ? (
|
||||
<p className="text-xs text-ctp-overlay0">
|
||||
No followed tags yet — use the + on any tag chip.
|
||||
</p>
|
||||
) : (
|
||||
<div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
|
||||
{tags.map((follow) => (
|
||||
<FollowCard
|
||||
key={follow.id}
|
||||
title={follow.tag}
|
||||
meta={
|
||||
follow.cover_data?.post_id
|
||||
? `latest #${follow.cover_data.post_id}`
|
||||
: ""
|
||||
}
|
||||
cover={follow.cover_data}
|
||||
unseen={follow.unseen_count}
|
||||
syncedAt={follow.last_synced_at}
|
||||
selected={
|
||||
feedSource.kind === "tag" && feedSource.id === follow.id
|
||||
}
|
||||
onSelect={() =>
|
||||
selectSource({
|
||||
kind: "tag",
|
||||
id: follow.id,
|
||||
label: `tag:${follow.tag}`,
|
||||
})
|
||||
}
|
||||
onSeen={() => seenFollow.mutate({ kind: "tag", id: follow.id })}
|
||||
onUnfollow={() => unfollow.mutate({ kind: "tag", id: follow.id })}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
|
||||
<section className="flex flex-col gap-3">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Pools{" "}
|
||||
<span className="font-mono text-[11px] text-ctp-overlay0">
|
||||
{pools.length}
|
||||
</span>
|
||||
</h2>
|
||||
{poolsQuery.isPending ? (
|
||||
<div className="flex justify-center py-8">
|
||||
<Spinner className="h-5 w-5" />
|
||||
</div>
|
||||
) : pools.length === 0 ? (
|
||||
<p className="text-xs text-ctp-overlay0">
|
||||
No followed pools yet — open a pool and use "Follow pool".
|
||||
</p>
|
||||
) : (
|
||||
<div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
|
||||
{pools.map((follow) => (
|
||||
<FollowCard
|
||||
key={follow.id}
|
||||
title={follow.name || `Pool #${follow.pool_id}`}
|
||||
meta={`#${follow.pool_id}${
|
||||
follow.cover_data?.post_id
|
||||
? ` · latest #${follow.cover_data.post_id}`
|
||||
: ""
|
||||
}`}
|
||||
cover={follow.cover_data}
|
||||
unseen={follow.unseen_count}
|
||||
syncedAt={follow.last_synced_at}
|
||||
selected={
|
||||
feedSource.kind === "pool" && feedSource.id === follow.id
|
||||
}
|
||||
onSelect={() =>
|
||||
selectSource({
|
||||
kind: "pool",
|
||||
id: follow.id,
|
||||
label: follow.name || `Pool #${follow.pool_id}`,
|
||||
})
|
||||
}
|
||||
onSeen={() =>
|
||||
seenFollow.mutate({ kind: "pool", id: follow.id })
|
||||
}
|
||||
onUnfollow={() =>
|
||||
unfollow.mutate({ kind: "pool", id: follow.id })
|
||||
}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
|
||||
<section className="flex flex-col gap-3">
|
||||
<div className="flex flex-wrap items-center justify-between gap-2">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Feed{" "}
|
||||
<span className="font-mono text-[11px] text-ctp-overlay0">
|
||||
{feed ? `${feed.count}` : "…"}
|
||||
{feedSource.kind !== "all" ? ` · showing ${feedSource.label}` : ""}
|
||||
</span>
|
||||
</h2>
|
||||
<div className="flex items-center gap-2">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => {
|
||||
setPage(1);
|
||||
setUnseenOnly((value) => !value);
|
||||
}}
|
||||
className={cn(
|
||||
"rounded-full border px-2.5 py-1 text-xs font-medium transition",
|
||||
unseenOnly
|
||||
? "border-ctp-blue/40 bg-ctp-blue/15 text-ctp-blue"
|
||||
: "border-ctp-surface1 text-ctp-subtext0 hover:border-ctp-surface2 hover:text-ctp-text",
|
||||
)}
|
||||
>
|
||||
Unseen only
|
||||
</button>
|
||||
{feedSource.kind !== "all" ? (
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => {
|
||||
setPage(1);
|
||||
setFeedSource({ kind: "all" });
|
||||
}}
|
||||
className="text-xs text-ctp-blue hover:underline"
|
||||
>
|
||||
Clear filter
|
||||
</button>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{feedQuery.isPending ? (
|
||||
<div className="flex justify-center py-12">
|
||||
<Spinner className="h-5 w-5" />
|
||||
</div>
|
||||
) : !feed || feed.results.length === 0 ? (
|
||||
<EmptyState
|
||||
title="Nothing in the feed"
|
||||
description={
|
||||
unseenOnly
|
||||
? "Everything here is marked seen."
|
||||
: "Follow tags or pools, or run a sync command to fetch posts."
|
||||
}
|
||||
/>
|
||||
) : (
|
||||
<div className="grid grid-cols-2 gap-3 sm:grid-cols-3 xl:grid-cols-4">
|
||||
{feed.results.map((entry) => (
|
||||
<Link
|
||||
key={entry.post_id}
|
||||
to={`/detail/${entry.post_id}`}
|
||||
onClick={() => {
|
||||
if (!entry.seen) seenPost.mutate(entry.post_id);
|
||||
}}
|
||||
className="group block overflow-hidden rounded-lg border border-ctp-surface0 bg-ctp-base transition hover:border-ctp-surface2"
|
||||
>
|
||||
<span className="relative block aspect-square overflow-hidden bg-ctp-mantle">
|
||||
{entry.data.preview?.url ? (
|
||||
<img
|
||||
src={entry.data.preview.url}
|
||||
alt={`Post #${entry.post_id}`}
|
||||
loading="lazy"
|
||||
className="h-full w-full object-cover transition group-hover:scale-[1.03]"
|
||||
/>
|
||||
) : (
|
||||
<span className="flex h-full w-full items-center justify-center text-[10px] text-ctp-overlay0">
|
||||
No preview
|
||||
</span>
|
||||
)}
|
||||
{!entry.seen ? (
|
||||
<span
|
||||
className="absolute left-2 top-2 h-2.5 w-2.5 rounded-full bg-ctp-blue ring-2 ring-ctp-crust"
|
||||
title="Unseen"
|
||||
/>
|
||||
) : null}
|
||||
{entry.data.rating ? (
|
||||
<span
|
||||
className={cn(
|
||||
"absolute right-2 top-2 rounded-full px-1.5 py-0.5 font-mono text-[10px] font-semibold uppercase",
|
||||
ratingPill[entry.data.rating] ??
|
||||
"bg-ctp-surface1 text-ctp-text",
|
||||
)}
|
||||
>
|
||||
{entry.data.rating}
|
||||
</span>
|
||||
) : null}
|
||||
</span>
|
||||
<span className="block px-2 py-1.5">
|
||||
<span className="block font-mono text-[11px] text-ctp-subtext0">
|
||||
#{entry.post_id}
|
||||
<span className="text-ctp-overlay0">
|
||||
{" "}
|
||||
▲ {entry.data.score?.total ?? 0}
|
||||
{entry.data.fav_count
|
||||
? ` · ${entry.data.fav_count} ★`
|
||||
: ""}
|
||||
</span>
|
||||
</span>
|
||||
<span className="block truncate text-[10px] text-ctp-overlay0">
|
||||
{entry.sources.map((source) => source.label).join(", ")}
|
||||
</span>
|
||||
</span>
|
||||
</Link>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{feed && (page > 1 || feed.has_next) ? (
|
||||
<div className="flex items-center justify-center gap-3">
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={page <= 1 || feedQuery.isFetching}
|
||||
onClick={() => setPage((current) => Math.max(1, current - 1))}
|
||||
>
|
||||
Previous
|
||||
</Button>
|
||||
<span className="font-mono text-xs text-ctp-subtext0">
|
||||
Page {page}
|
||||
</span>
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={!feed.has_next || feedQuery.isFetching}
|
||||
onClick={() => setPage((current) => current + 1)}
|
||||
>
|
||||
Next
|
||||
</Button>
|
||||
</div>
|
||||
) : null}
|
||||
</section>
|
||||
|
||||
<BlacklistCloudPanel />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Check, Star } from "lucide-react";
|
||||
import { useState } from "react";
|
||||
|
||||
import { Button, Spinner } from "@/components/ui";
|
||||
import { api, errorMessage } from "@/lib/api";
|
||||
import type { FollowedPool, Paginated } from "@/lib/types";
|
||||
import { useAuth } from "@/store/auth";
|
||||
|
||||
export function PoolFollowButton({
|
||||
poolId,
|
||||
name,
|
||||
}: {
|
||||
poolId: number;
|
||||
name?: string;
|
||||
}) {
|
||||
const user = useAuth((state) => state.user);
|
||||
const queryClient = useQueryClient();
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
// What the button should show once the request and refetch settle.
|
||||
const [target, setTarget] = useState<boolean | null>(null);
|
||||
|
||||
const query = useQuery({
|
||||
queryKey: ["follows-pools"],
|
||||
queryFn: () => api<Paginated<FollowedPool>>("/api/follows/pools/"),
|
||||
enabled: Boolean(user),
|
||||
staleTime: 30_000,
|
||||
});
|
||||
|
||||
const follow = query.data?.results.find((entry) => entry.pool_id === poolId);
|
||||
const followed = Boolean(follow);
|
||||
|
||||
const mutation = useMutation({
|
||||
mutationFn: () =>
|
||||
follow
|
||||
? api(`/api/follows/pools/${follow.id}/`, { method: "DELETE" })
|
||||
: api<FollowedPool>("/api/follows/pools/", {
|
||||
method: "POST",
|
||||
json: { pool_id: poolId },
|
||||
}),
|
||||
onSuccess: () => {
|
||||
setError(null);
|
||||
void queryClient.invalidateQueries({ queryKey: ["follows-pools"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["follows-feed"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["follows-cloud"] });
|
||||
},
|
||||
onError: (err) => {
|
||||
setTarget(null);
|
||||
setError(errorMessage(err));
|
||||
},
|
||||
});
|
||||
|
||||
if (!user) return null;
|
||||
|
||||
const busy = mutation.isPending || (target !== null && followed !== target);
|
||||
const label = name ? `"${name}"` : `#${poolId}`;
|
||||
|
||||
return (
|
||||
<div className="flex flex-col items-end gap-1">
|
||||
<Button
|
||||
variant={followed ? "secondary" : "primary"}
|
||||
disabled={busy || query.isPending}
|
||||
onClick={() => {
|
||||
if (busy) return;
|
||||
setTarget(!followed);
|
||||
mutation.mutate();
|
||||
}}
|
||||
title={followed ? `Unfollow pool ${label}` : `Follow pool ${label}`}
|
||||
>
|
||||
{busy ? (
|
||||
<Spinner className="h-3.5 w-3.5" />
|
||||
) : followed ? (
|
||||
<Check className="h-4 w-4" />
|
||||
) : (
|
||||
<Star className="h-4 w-4" />
|
||||
)}
|
||||
{followed ? "Following" : "Follow pool"}
|
||||
</Button>
|
||||
{error ? <p className="text-xs text-ctp-red">{error}</p> : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Check, Plus } from "lucide-react";
|
||||
import { useState, type MouseEvent } from "react";
|
||||
|
||||
import { api, errorMessage } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import type { FollowedTag, Paginated } from "@/lib/types";
|
||||
import { useAuth } from "@/store/auth";
|
||||
|
||||
export function TagFollowToggle({ tag }: { tag: string }) {
|
||||
const user = useAuth((state) => state.user);
|
||||
const queryClient = useQueryClient();
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
// What the chip should show once the request and refetch settle.
|
||||
const [target, setTarget] = useState<boolean | null>(null);
|
||||
|
||||
const query = useQuery({
|
||||
queryKey: ["follows-tags"],
|
||||
queryFn: () => api<Paginated<FollowedTag>>("/api/follows/tags/"),
|
||||
enabled: Boolean(user),
|
||||
staleTime: 30_000,
|
||||
});
|
||||
|
||||
const normalized = tag.trim().toLowerCase();
|
||||
const follow = query.data?.results.find((entry) => entry.tag === normalized);
|
||||
const followed = Boolean(follow);
|
||||
|
||||
const mutation = useMutation({
|
||||
mutationFn: () =>
|
||||
follow
|
||||
? api(`/api/follows/tags/${follow.id}/`, { method: "DELETE" })
|
||||
: api<FollowedTag>("/api/follows/tags/", {
|
||||
method: "POST",
|
||||
json: { tag: normalized },
|
||||
}),
|
||||
onSuccess: () => {
|
||||
setError(null);
|
||||
void queryClient.invalidateQueries({ queryKey: ["follows-tags"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["follows-feed"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["follows-cloud"] });
|
||||
},
|
||||
onError: (err) => {
|
||||
setTarget(null);
|
||||
setError(errorMessage(err));
|
||||
},
|
||||
});
|
||||
|
||||
// Keep the spinner up until the refetched list reflects the new state.
|
||||
// A stale target that matches `followed` is harmless: the next click
|
||||
// recomputes it before mutating.
|
||||
const busy = mutation.isPending || (target !== null && followed !== target);
|
||||
|
||||
if (!user) return null;
|
||||
|
||||
function handleClick(event: MouseEvent<HTMLButtonElement>) {
|
||||
// Chips often sit inside clickable labels/links; the toggle is its own
|
||||
// action and must not trigger them.
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
if (busy) return;
|
||||
setTarget(!followed);
|
||||
mutation.mutate();
|
||||
}
|
||||
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
onClick={handleClick}
|
||||
disabled={busy || query.isPending}
|
||||
title={error ?? (followed ? `Unfollow ${tag}` : `Follow ${tag}`)}
|
||||
aria-label={followed ? `Unfollow ${tag}` : `Follow ${tag}`}
|
||||
className={cn(
|
||||
"-mr-0.5 inline-flex h-3.5 w-3.5 shrink-0 items-center justify-center rounded-sm transition",
|
||||
error ? "text-ctp-red" : "opacity-70 hover:opacity-100",
|
||||
query.isPending && "opacity-40",
|
||||
)}
|
||||
>
|
||||
{busy ? (
|
||||
<span
|
||||
aria-hidden="true"
|
||||
className="h-3 w-3 animate-spin rounded-full border-[1.5px] border-current border-t-transparent"
|
||||
/>
|
||||
) : followed ? (
|
||||
<Check className="h-3 w-3" strokeWidth={3} />
|
||||
) : (
|
||||
<Plus className="h-3 w-3" strokeWidth={3} />
|
||||
)}
|
||||
</button>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { useState } from "react";
|
||||
import { Link } from "react-router-dom";
|
||||
|
||||
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
|
||||
import { api, errorMessage } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import { formatBytes } from "@/lib/format";
|
||||
import type { MediaItem, Paginated, StorageInfo } from "@/lib/types";
|
||||
|
||||
function parseJIds(text: string): string[] {
|
||||
return text
|
||||
.split(/[\s,]+/)
|
||||
.map((value) => value.trim().toUpperCase())
|
||||
.filter((value) => /^J-\d+$/.test(value));
|
||||
}
|
||||
|
||||
export default function DeletePage() {
|
||||
const queryClient = useQueryClient();
|
||||
const [jIdsInput, setJIdsInput] = useState("");
|
||||
const [loadedIds, setLoadedIds] = useState<string[]>([]);
|
||||
const [selected, setSelected] = useState<string[]>([]);
|
||||
const [status, setStatus] = useState<string | null>(null);
|
||||
|
||||
const storageQuery = useQuery({
|
||||
queryKey: ["storage"],
|
||||
queryFn: () => api<StorageInfo>("/api/storage/"),
|
||||
});
|
||||
|
||||
const filesQuery = useQuery({
|
||||
queryKey: ["delete-files", loadedIds],
|
||||
queryFn: () =>
|
||||
api<Paginated<MediaItem>>(`/api/files/?j_ids=${loadedIds.join(",")}`),
|
||||
enabled: loadedIds.length > 0,
|
||||
});
|
||||
|
||||
const deleteMutation = useMutation({
|
||||
mutationFn: (jIds: string[]) =>
|
||||
api<{ deleted: string[]; errors: { j_id?: string; error: string }[] }>(
|
||||
"/api/delete/",
|
||||
{ method: "POST", json: { j_ids: jIds } },
|
||||
),
|
||||
onSuccess: (result) => {
|
||||
setSelected([]);
|
||||
setLoadedIds([]);
|
||||
setStatus(
|
||||
`Deleted ${result.deleted.length} item(s)${
|
||||
result.errors.length > 0 ? ` · ${result.errors.length} failed` : ""
|
||||
}.`,
|
||||
);
|
||||
void queryClient.invalidateQueries({ queryKey: ["storage"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["duplicates-md5"] });
|
||||
},
|
||||
onError: (error) => setStatus(errorMessage(error)),
|
||||
});
|
||||
|
||||
const tempMutation = useMutation({
|
||||
mutationFn: () =>
|
||||
api<{ removed: number }>("/api/temp/clear/", { method: "POST" }),
|
||||
onSuccess: (result) => {
|
||||
setStatus(`Removed ${result.removed} temp file(s).`);
|
||||
void queryClient.invalidateQueries({ queryKey: ["storage"] });
|
||||
},
|
||||
onError: (error) => setStatus(errorMessage(error)),
|
||||
});
|
||||
|
||||
function toggleSelected(jId: string) {
|
||||
setSelected((current) =>
|
||||
current.includes(jId)
|
||||
? current.filter((value) => value !== jId)
|
||||
: [...current, jId],
|
||||
);
|
||||
}
|
||||
|
||||
const storage = storageQuery.data;
|
||||
|
||||
return (
|
||||
<div className="mx-auto flex w-full max-w-6xl flex-col gap-6">
|
||||
<header>
|
||||
<h1 className="text-lg font-semibold">Delete & storage</h1>
|
||||
<p className="mt-1 text-sm text-ctp-overlay0">
|
||||
Inspect disk usage and remove files from the library.
|
||||
</p>
|
||||
</header>
|
||||
|
||||
{status ? (
|
||||
<p className="rounded-md border border-ctp-surface1 bg-ctp-base px-3 py-2 text-xs text-ctp-subtext0">
|
||||
{status}
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
<section className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
|
||||
{storageQuery.isPending ? (
|
||||
<Spinner className="h-5 w-5" />
|
||||
) : storage ? (
|
||||
<>
|
||||
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
|
||||
Library size
|
||||
</p>
|
||||
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
|
||||
{formatBytes(storage.watched_folder.library_size)}
|
||||
</p>
|
||||
<p className="mt-1 truncate font-mono text-[10px] text-ctp-overlay0" title={storage.watched_folder.path}>
|
||||
{storage.watched_folder.path}
|
||||
</p>
|
||||
<p className="font-mono text-[10px] text-ctp-overlay0">
|
||||
{storage.library_items} items
|
||||
</p>
|
||||
</div>
|
||||
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
|
||||
Disk
|
||||
</p>
|
||||
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
|
||||
{formatBytes(storage.watched_folder.used)} /{" "}
|
||||
{formatBytes(storage.watched_folder.total)}
|
||||
</p>
|
||||
<div className="mt-2 h-1.5 w-full overflow-hidden rounded-full bg-ctp-surface0">
|
||||
<div
|
||||
className={cn(
|
||||
"h-full",
|
||||
storage.watched_folder.percent_used >= 95
|
||||
? "bg-ctp-red"
|
||||
: storage.watched_folder.percent_used >= 80
|
||||
? "bg-ctp-peach"
|
||||
: "bg-ctp-blue",
|
||||
)}
|
||||
style={{ width: `${storage.watched_folder.percent_used}%` }}
|
||||
/>
|
||||
</div>
|
||||
<p className="mt-1 font-mono text-[10px] text-ctp-overlay0">
|
||||
{storage.watched_folder.percent_used}% used ·{" "}
|
||||
{formatBytes(storage.watched_folder.free)} free
|
||||
</p>
|
||||
</div>
|
||||
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
|
||||
Media folder
|
||||
</p>
|
||||
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
|
||||
{formatBytes(storage.media.size)}
|
||||
</p>
|
||||
<p className="mt-1 truncate font-mono text-[10px] text-ctp-overlay0" title={storage.media.path}>
|
||||
{storage.media.path}
|
||||
</p>
|
||||
</div>
|
||||
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
|
||||
Temp uploads
|
||||
</p>
|
||||
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
|
||||
{formatBytes(storage.temp.size)} · {storage.temp.files} file(s)
|
||||
</p>
|
||||
<Button
|
||||
variant="secondary"
|
||||
className="mt-2 px-2 py-1 text-xs"
|
||||
disabled={tempMutation.isPending || storage.temp.files === 0}
|
||||
onClick={() => tempMutation.mutate()}
|
||||
>
|
||||
{tempMutation.isPending ? "Clearing…" : "Clear temp folder"}
|
||||
</Button>
|
||||
</div>
|
||||
</>
|
||||
) : (
|
||||
<EmptyState title="Could not load storage info" />
|
||||
)}
|
||||
</section>
|
||||
|
||||
<section className="flex flex-col gap-3">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Delete files by J-ID
|
||||
</h2>
|
||||
<textarea
|
||||
className={cn(inputClass, "min-h-20 resize-y font-mono")}
|
||||
placeholder="J-12, J-34 J-56 comma, space or newline separated"
|
||||
value={jIdsInput}
|
||||
onChange={(event) => setJIdsInput(event.target.value)}
|
||||
/>
|
||||
<div className="flex items-center gap-3">
|
||||
<Button
|
||||
variant="secondary"
|
||||
onClick={() => {
|
||||
setStatus(null);
|
||||
setSelected([]);
|
||||
setLoadedIds(parseJIds(jIdsInput));
|
||||
}}
|
||||
>
|
||||
Load files
|
||||
</Button>
|
||||
{selected.length > 0 ? (
|
||||
<Button
|
||||
variant="danger"
|
||||
disabled={deleteMutation.isPending}
|
||||
onClick={() => deleteMutation.mutate(selected)}
|
||||
>
|
||||
{deleteMutation.isPending
|
||||
? "Deleting…"
|
||||
: `Delete ${selected.length} selected`}
|
||||
</Button>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
{loadedIds.length === 0 ? (
|
||||
<p className="text-xs text-ctp-overlay0">
|
||||
Enter one or more J-IDs to preview them here.
|
||||
</p>
|
||||
) : filesQuery.isPending ? (
|
||||
<Spinner className="h-5 w-5" />
|
||||
) : (filesQuery.data?.results.length ?? 0) === 0 ? (
|
||||
<EmptyState title="No matching library items" />
|
||||
) : (
|
||||
<div className="grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
|
||||
{filesQuery.data?.results.map((item) => {
|
||||
const isSelected = selected.includes(item.j_id);
|
||||
return (
|
||||
<div
|
||||
key={item.j_id}
|
||||
className={cn(
|
||||
"overflow-hidden rounded-lg border bg-ctp-base",
|
||||
isSelected ? "border-ctp-mauve" : "border-ctp-surface0",
|
||||
)}
|
||||
>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => toggleSelected(item.j_id)}
|
||||
className="block w-full text-left"
|
||||
>
|
||||
<div className="relative aspect-square overflow-hidden bg-ctp-mantle">
|
||||
<img
|
||||
src={item.thumbnail_url}
|
||||
alt={item.filename}
|
||||
loading="lazy"
|
||||
className="h-full w-full object-cover"
|
||||
/>
|
||||
<span
|
||||
className={cn(
|
||||
"absolute left-2 top-2 flex h-5 w-5 items-center justify-center rounded border text-[11px]",
|
||||
isSelected
|
||||
? "border-ctp-mauve bg-ctp-mauve text-ctp-crust"
|
||||
: "border-ctp-surface1 bg-ctp-crust/70 text-transparent",
|
||||
)}
|
||||
>
|
||||
✓
|
||||
</span>
|
||||
</div>
|
||||
</button>
|
||||
<div className="px-2 py-1.5">
|
||||
<Link
|
||||
to={`/detail/${item.j_id}`}
|
||||
className="font-mono text-[10px] text-ctp-blue hover:underline"
|
||||
>
|
||||
{item.j_id}
|
||||
</Link>
|
||||
<p className="truncate font-mono text-[11px] text-ctp-subtext1" title={item.filename}>
|
||||
{item.filename}
|
||||
</p>
|
||||
<p className="font-mono text-[10px] text-ctp-overlay0">
|
||||
{formatBytes(item.size)}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,473 @@
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { useMemo, useState } from "react";
|
||||
import { Link } from "react-router-dom";
|
||||
|
||||
import { Button, Spinner, inputClass } from "@/components/ui";
|
||||
import { api, errorMessage } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import { formatBytes } from "@/lib/format";
|
||||
import type {
|
||||
DuplicateEntry,
|
||||
VisualGroupsResponse,
|
||||
VisualMatchesResponse,
|
||||
} from "@/lib/types";
|
||||
|
||||
const ALGORITHMS = [
|
||||
{ value: "ahash", label: "aHash" },
|
||||
{ value: "dhash", label: "dHash" },
|
||||
{ value: "phash", label: "pHash" },
|
||||
{ value: "whash", label: "wHash" },
|
||||
];
|
||||
|
||||
interface DeleteResult {
|
||||
deleted: string[];
|
||||
errors: { j_id?: string; location?: number; error: string }[];
|
||||
}
|
||||
|
||||
function MemberCard({
|
||||
entry,
|
||||
selected,
|
||||
onToggle,
|
||||
}: {
|
||||
entry: DuplicateEntry;
|
||||
selected: boolean;
|
||||
onToggle: () => void;
|
||||
}) {
|
||||
const preview =
|
||||
entry.thumbnail_url ?? `/api/files/${entry.j_id}/thumbnail/`;
|
||||
return (
|
||||
<div
|
||||
className={cn(
|
||||
"overflow-hidden rounded-lg border bg-ctp-base",
|
||||
selected ? "border-ctp-mauve" : "border-ctp-surface0",
|
||||
)}
|
||||
>
|
||||
<button type="button" onClick={onToggle} className="block w-full text-left">
|
||||
<div className="relative aspect-square overflow-hidden bg-ctp-mantle">
|
||||
<img
|
||||
src={preview}
|
||||
alt={entry.j_id}
|
||||
loading="lazy"
|
||||
className="h-full w-full object-cover"
|
||||
/>
|
||||
<span
|
||||
className={cn(
|
||||
"absolute left-2 top-2 flex h-5 w-5 items-center justify-center rounded border text-[11px]",
|
||||
selected
|
||||
? "border-ctp-mauve bg-ctp-mauve text-ctp-crust"
|
||||
: "border-ctp-surface1 bg-ctp-crust/70 text-transparent",
|
||||
)}
|
||||
>
|
||||
✓
|
||||
</span>
|
||||
{entry.similarity !== undefined ? (
|
||||
<span className="absolute right-2 top-2 rounded-sm bg-ctp-crust/80 px-1.5 py-0.5 font-mono text-[10px] text-ctp-peach">
|
||||
{entry.similarity}%
|
||||
</span>
|
||||
) : null}
|
||||
</div>
|
||||
</button>
|
||||
<div className="px-2 py-1.5">
|
||||
<Link
|
||||
to={`/detail/${entry.j_id}`}
|
||||
className="font-mono text-[10px] text-ctp-blue hover:underline"
|
||||
>
|
||||
{entry.j_id}
|
||||
</Link>
|
||||
<p
|
||||
className="truncate font-mono text-[11px] text-ctp-subtext1"
|
||||
title={entry.filename}
|
||||
>
|
||||
{entry.filename}
|
||||
</p>
|
||||
<p className="font-mono text-[10px] text-ctp-overlay0">
|
||||
{formatBytes(entry.size)}
|
||||
{entry.location_count > 1 ? ` · ${entry.location_count} copies` : ""}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default function DuplicatesPage() {
|
||||
const queryClient = useQueryClient();
|
||||
const [threshold, setThreshold] = useState(80);
|
||||
const [algorithms, setAlgorithms] = useState<string[]>(
|
||||
ALGORITHMS.map((algorithm) => algorithm.value),
|
||||
);
|
||||
const [groupsPage, setGroupsPage] = useState(1);
|
||||
const [groupsEnabled, setGroupsEnabled] = useState(false);
|
||||
const [selected, setSelected] = useState<string[]>([]);
|
||||
const [dismissed, setDismissed] = useState<string[]>([]);
|
||||
const [similarTarget, setSimilarTarget] = useState("");
|
||||
const [status, setStatus] = useState<string | null>(null);
|
||||
|
||||
const exactQuery = useQuery({
|
||||
queryKey: ["duplicates-md5"],
|
||||
queryFn: () =>
|
||||
api<{ count: number; groups: DuplicateEntry[] }>("/api/duplicates/md5/"),
|
||||
});
|
||||
|
||||
const groupsQuery = useQuery({
|
||||
queryKey: [
|
||||
"duplicate-groups",
|
||||
groupsEnabled,
|
||||
threshold,
|
||||
algorithms,
|
||||
groupsPage,
|
||||
],
|
||||
queryFn: () =>
|
||||
api<VisualGroupsResponse>("/api/duplicates/visual_groups/", {
|
||||
method: "POST",
|
||||
json: { threshold, algorithms, page: groupsPage },
|
||||
}),
|
||||
enabled: groupsEnabled && algorithms.length > 0,
|
||||
});
|
||||
|
||||
const similarMutation = useMutation({
|
||||
mutationFn: (jId: string) =>
|
||||
api<VisualMatchesResponse>("/api/duplicates/visual/", {
|
||||
method: "POST",
|
||||
json: { j_id: jId, threshold, algorithms },
|
||||
}),
|
||||
onError: (error) => setStatus(errorMessage(error)),
|
||||
});
|
||||
|
||||
const deleteMutation = useMutation({
|
||||
mutationFn: (payload: { j_ids?: string[]; location_ids?: number[] }) =>
|
||||
api<DeleteResult>("/api/delete/", { method: "POST", json: payload }),
|
||||
onSuccess: (result) => {
|
||||
setSelected([]);
|
||||
setStatus(
|
||||
`Deleted ${result.deleted.length} entr${
|
||||
result.deleted.length === 1 ? "y" : "ies"
|
||||
}${result.errors.length > 0 ? ` · ${result.errors.length} failed` : ""}.`,
|
||||
);
|
||||
void queryClient.invalidateQueries({ queryKey: ["duplicates-md5"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["duplicate-groups"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||
},
|
||||
onError: (error) => setStatus(errorMessage(error)),
|
||||
});
|
||||
|
||||
function toggleSelected(jId: string) {
|
||||
setSelected((current) =>
|
||||
current.includes(jId)
|
||||
? current.filter((value) => value !== jId)
|
||||
: [...current, jId],
|
||||
);
|
||||
}
|
||||
|
||||
function toggleAlgorithm(value: string) {
|
||||
setGroupsPage(1);
|
||||
setAlgorithms((current) =>
|
||||
current.includes(value)
|
||||
? current.filter((algorithm) => algorithm !== value)
|
||||
: [...current, value],
|
||||
);
|
||||
}
|
||||
|
||||
const groups = useMemo(
|
||||
() =>
|
||||
(groupsQuery.data?.groups ?? []).filter(
|
||||
(group) =>
|
||||
!dismissed.includes(
|
||||
group.members
|
||||
.map((member) => member.j_id)
|
||||
.sort()
|
||||
.join("|"),
|
||||
),
|
||||
),
|
||||
[groupsQuery.data, dismissed],
|
||||
);
|
||||
|
||||
return (
|
||||
<div className="mx-auto flex w-full max-w-6xl flex-col gap-8">
|
||||
<header>
|
||||
<h1 className="text-lg font-semibold">Duplicates</h1>
|
||||
<p className="mt-1 text-sm text-ctp-overlay0">
|
||||
Clean up exact copies and visually similar files.
|
||||
</p>
|
||||
</header>
|
||||
|
||||
{status ? (
|
||||
<p className="rounded-md border border-ctp-surface1 bg-ctp-base px-3 py-2 text-xs text-ctp-subtext0">
|
||||
{status}
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
<section className="flex flex-col gap-3">
|
||||
<header className="flex items-center justify-between gap-2">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Exact duplicates
|
||||
</h2>
|
||||
<span className="font-mono text-[11px] text-ctp-overlay0">
|
||||
{exactQuery.data?.count ?? 0} item(s)
|
||||
</span>
|
||||
</header>
|
||||
{exactQuery.isPending ? (
|
||||
<Spinner className="h-5 w-5" />
|
||||
) : (exactQuery.data?.count ?? 0) === 0 ? (
|
||||
<p className="rounded-lg border border-dashed border-ctp-surface1 px-3 py-6 text-center text-xs text-ctp-overlay0">
|
||||
No items exist at multiple paths.
|
||||
</p>
|
||||
) : (
|
||||
<div className="flex flex-col gap-3">
|
||||
{exactQuery.data?.groups.map((group) => (
|
||||
<div
|
||||
key={group.j_id}
|
||||
className="flex flex-col gap-3 rounded-lg border border-ctp-surface0 bg-ctp-base p-3 lg:flex-row"
|
||||
>
|
||||
<div className="w-32 shrink-0">
|
||||
<div className="aspect-square overflow-hidden rounded-md bg-ctp-mantle">
|
||||
<img
|
||||
src={
|
||||
group.thumbnail_url ??
|
||||
`/api/files/${group.j_id}/thumbnail/`
|
||||
}
|
||||
alt={group.j_id}
|
||||
loading="lazy"
|
||||
className="h-full w-full object-cover"
|
||||
/>
|
||||
</div>
|
||||
<Link
|
||||
to={`/detail/${group.j_id}`}
|
||||
className="font-mono text-[10px] text-ctp-blue hover:underline"
|
||||
>
|
||||
{group.j_id}
|
||||
</Link>
|
||||
</div>
|
||||
<div className="min-w-0 flex-1">
|
||||
<p className="truncate font-mono text-xs text-ctp-subtext1">
|
||||
{group.filename}
|
||||
</p>
|
||||
<ul className="mt-2 flex flex-col gap-1.5">
|
||||
{(group.locations ?? []).map((location) => (
|
||||
<li
|
||||
key={location.id}
|
||||
className="flex items-center justify-between gap-3 rounded-md border border-ctp-surface0 px-2 py-1"
|
||||
>
|
||||
<span className="truncate font-mono text-[11px] text-ctp-overlay1" title={location.rel_path}>
|
||||
{location.rel_path}
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() =>
|
||||
deleteMutation.mutate({ location_ids: [location.id] })
|
||||
}
|
||||
className="shrink-0 rounded-md px-1.5 py-0.5 font-mono text-[10px] text-ctp-red transition hover:bg-ctp-red/15"
|
||||
>
|
||||
delete copy
|
||||
</button>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
<div className="shrink-0">
|
||||
<Button
|
||||
variant="danger"
|
||||
className="px-2 py-1 text-xs"
|
||||
onClick={() => deleteMutation.mutate({ j_ids: [group.j_id] })}
|
||||
>
|
||||
Delete item
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
|
||||
<section className="flex flex-col gap-3">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Visual similarity
|
||||
</h2>
|
||||
<div className="flex flex-col gap-3 rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||
<div className="flex flex-wrap items-center gap-4">
|
||||
<label className="flex flex-1 items-center gap-3 text-xs text-ctp-subtext0">
|
||||
Similarity {threshold}%
|
||||
<input
|
||||
type="range"
|
||||
min={50}
|
||||
max={100}
|
||||
value={threshold}
|
||||
onChange={(event) => {
|
||||
setGroupsPage(1);
|
||||
setThreshold(Number(event.target.value));
|
||||
}}
|
||||
className="flex-1"
|
||||
/>
|
||||
</label>
|
||||
<div className="flex flex-wrap items-center gap-2">
|
||||
{ALGORITHMS.map((algorithm) => (
|
||||
<label
|
||||
key={algorithm.value}
|
||||
className={cn(
|
||||
"cursor-pointer rounded-full border px-2.5 py-1 font-mono text-[11px] transition",
|
||||
algorithms.includes(algorithm.value)
|
||||
? "border-ctp-mauve/40 bg-ctp-mauve/15 text-ctp-mauve"
|
||||
: "border-ctp-surface1 text-ctp-subtext0 hover:border-ctp-surface2",
|
||||
)}
|
||||
>
|
||||
<input
|
||||
type="checkbox"
|
||||
className="hidden"
|
||||
checked={algorithms.includes(algorithm.value)}
|
||||
onChange={() => toggleAlgorithm(algorithm.value)}
|
||||
/>
|
||||
{algorithm.label}
|
||||
</label>
|
||||
))}
|
||||
</div>
|
||||
<Button
|
||||
onClick={() => {
|
||||
setGroupsEnabled(true);
|
||||
void groupsQuery.refetch();
|
||||
}}
|
||||
disabled={algorithms.length === 0 || groupsQuery.isFetching}
|
||||
>
|
||||
{groupsQuery.isFetching ? "Comparing…" : "Find groups"}
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap items-end gap-2 border-t border-ctp-surface0 pt-3">
|
||||
<label className="flex flex-col gap-1 text-xs text-ctp-subtext0">
|
||||
Find files similar to
|
||||
<input
|
||||
className={cn(inputClass, "w-32 font-mono")}
|
||||
placeholder="J-123"
|
||||
value={similarTarget}
|
||||
onChange={(event) => setSimilarTarget(event.target.value)}
|
||||
/>
|
||||
</label>
|
||||
<Button
|
||||
variant="secondary"
|
||||
onClick={() => similarMutation.mutate(similarTarget.trim())}
|
||||
disabled={!similarTarget.trim() || similarMutation.isPending}
|
||||
>
|
||||
{similarMutation.isPending ? "Searching…" : "Search"}
|
||||
</Button>
|
||||
{similarMutation.isError ? (
|
||||
<p className="text-xs text-ctp-red">
|
||||
{errorMessage(similarMutation.error)}
|
||||
</p>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
{similarMutation.data ? (
|
||||
similarMutation.data.count === 0 ? (
|
||||
<p className="text-xs text-ctp-overlay0">
|
||||
No matches at {similarMutation.data.threshold}% similarity.
|
||||
</p>
|
||||
) : (
|
||||
<div className="grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
|
||||
{similarMutation.data.matches.map((entry) => (
|
||||
<MemberCard
|
||||
key={entry.j_id}
|
||||
entry={entry}
|
||||
selected={selected.includes(entry.j_id)}
|
||||
onToggle={() => toggleSelected(entry.j_id)}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
)
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
{groupsEnabled ? (
|
||||
groupsQuery.isPending ? (
|
||||
<Spinner className="h-5 w-5" />
|
||||
) : groups.length === 0 ? (
|
||||
<p className="rounded-lg border border-dashed border-ctp-surface1 px-3 py-6 text-center text-xs text-ctp-overlay0">
|
||||
No groups at {threshold}% similarity.
|
||||
</p>
|
||||
) : (
|
||||
<div className="flex flex-col gap-4">
|
||||
{groups.map((group) => {
|
||||
const groupKey = group.members
|
||||
.map((member) => member.j_id)
|
||||
.sort()
|
||||
.join("|");
|
||||
return (
|
||||
<div
|
||||
key={groupKey}
|
||||
className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3"
|
||||
>
|
||||
<header className="flex items-center justify-between gap-2">
|
||||
<span className="font-mono text-[11px] text-ctp-overlay0">
|
||||
{group.size} similar items
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setDismissed((current) => [...current, groupKey])}
|
||||
className="rounded-md px-1.5 py-0.5 font-mono text-[10px] text-ctp-overlay0 transition hover:bg-ctp-surface0 hover:text-ctp-text"
|
||||
>
|
||||
dismiss
|
||||
</button>
|
||||
</header>
|
||||
<div className="mt-3 grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
|
||||
{group.members.map((member) => (
|
||||
<MemberCard
|
||||
key={member.j_id}
|
||||
entry={member}
|
||||
selected={selected.includes(member.j_id)}
|
||||
onToggle={() => toggleSelected(member.j_id)}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
|
||||
<div className="flex items-center justify-between gap-3">
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={groupsPage <= 1 || groupsQuery.isFetching}
|
||||
onClick={() => setGroupsPage((page) => Math.max(1, page - 1))}
|
||||
>
|
||||
Previous
|
||||
</Button>
|
||||
<span className="font-mono text-xs text-ctp-overlay0">
|
||||
Page {groupsQuery.data?.page ?? groupsPage} ·{" "}
|
||||
{groupsQuery.data?.count ?? 0} group(s)
|
||||
</span>
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={!groupsQuery.data?.has_next || groupsQuery.isFetching}
|
||||
onClick={() => setGroupsPage((page) => page + 1)}
|
||||
>
|
||||
Next
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
) : null}
|
||||
|
||||
{selected.length > 0 ? (
|
||||
<div className="sticky bottom-16 z-20 flex items-center justify-between gap-3 rounded-lg border border-ctp-mauve/40 bg-ctp-surface0 px-3 py-2 shadow-[0_4px_12px_rgba(17,17,27,0.6)]">
|
||||
<span className="text-xs text-ctp-subtext1">
|
||||
{selected.length} selected
|
||||
</span>
|
||||
<div className="flex items-center gap-2">
|
||||
<Button
|
||||
variant="ghost"
|
||||
className="px-2 py-1 text-xs"
|
||||
onClick={() => setSelected([])}
|
||||
>
|
||||
Clear
|
||||
</Button>
|
||||
<Button
|
||||
variant="danger"
|
||||
className="px-2 py-1 text-xs"
|
||||
disabled={deleteMutation.isPending}
|
||||
onClick={() => deleteMutation.mutate({ j_ids: selected })}
|
||||
>
|
||||
{deleteMutation.isPending ? "Deleting…" : "Delete selected"}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
) : null}
|
||||
</section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,282 @@
|
||||
import { useMutation, useQueryClient } from "@tanstack/react-query";
|
||||
import { ScanSearch } from "lucide-react";
|
||||
import { useState } from "react";
|
||||
import { Link } from "react-router-dom";
|
||||
|
||||
import { Button, Spinner } from "@/components/ui";
|
||||
import { api, errorMessage } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import {
|
||||
e621ErrorMessage,
|
||||
effectiveCredentials,
|
||||
iqdbSearch,
|
||||
RATING_LABELS,
|
||||
type E621IqdbResult,
|
||||
} from "@/lib/e621";
|
||||
import type { MediaItem } from "@/lib/types";
|
||||
import { useE621 } from "@/store/e621";
|
||||
|
||||
const ratingPill: Record<string, string> = {
|
||||
s: "bg-ctp-green text-ctp-crust",
|
||||
q: "bg-ctp-peach text-ctp-crust",
|
||||
e: "bg-ctp-red text-ctp-crust",
|
||||
};
|
||||
|
||||
type Notice = { kind: "ok" | "warn" | "error"; text: string } | null;
|
||||
|
||||
export function IqdbCard({
|
||||
item,
|
||||
canLink,
|
||||
}: {
|
||||
item: MediaItem;
|
||||
canLink: boolean;
|
||||
}) {
|
||||
const queryClient = useQueryClient();
|
||||
const credentials = useE621((state) => state.credentials);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [candidates, setCandidates] = useState<E621IqdbResult[] | null>(null);
|
||||
const [selected, setSelected] = useState<E621IqdbResult | null>(null);
|
||||
const [notice, setNotice] = useState<Notice>(null);
|
||||
|
||||
const isImage = item.kind === "image";
|
||||
const configured = Boolean(credentials?.configured);
|
||||
|
||||
async function search() {
|
||||
setBusy(true);
|
||||
setNotice(null);
|
||||
setSelected(null);
|
||||
setCandidates(null);
|
||||
try {
|
||||
const response = await fetch(item.raw_url);
|
||||
if (!response.ok) {
|
||||
throw new Error(`Could not load the file (HTTP ${response.status}).`);
|
||||
}
|
||||
const blob = await response.blob();
|
||||
const results = await iqdbSearch(
|
||||
effectiveCredentials(credentials),
|
||||
blob,
|
||||
item.filename,
|
||||
);
|
||||
setCandidates(results);
|
||||
if (results.length === 0) {
|
||||
setNotice({ kind: "warn", text: "No similar posts found on e621." });
|
||||
}
|
||||
} catch (error) {
|
||||
setNotice({ kind: "error", text: e621ErrorMessage(error) });
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}
|
||||
|
||||
const linkMutation = useMutation({
|
||||
mutationFn: (postId: number) =>
|
||||
api<MediaItem & { md5_match?: boolean }>(
|
||||
`/api/files/${item.j_id}/match/`,
|
||||
{ method: "POST", json: { post_id: postId } },
|
||||
),
|
||||
onSuccess: (data) => {
|
||||
void queryClient.invalidateQueries({ queryKey: ["file", item.j_id] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["tag-cloud"] });
|
||||
setSelected(null);
|
||||
setNotice(
|
||||
data.md5_match
|
||||
? {
|
||||
kind: "ok",
|
||||
text: `Linked to post #${data.e621_post_id} — MD5 matches.`,
|
||||
}
|
||||
: {
|
||||
kind: "warn",
|
||||
text: `Linked to post #${data.e621_post_id}, but its MD5 differs from this file.`,
|
||||
},
|
||||
);
|
||||
},
|
||||
onError: (error) =>
|
||||
setNotice({ kind: "error", text: errorMessage(error) }),
|
||||
});
|
||||
|
||||
return (
|
||||
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
|
||||
<div className="flex items-center justify-between gap-2">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">IQDB search</h2>
|
||||
{candidates && candidates.length > 0 ? (
|
||||
<span className="font-mono text-[11px] text-ctp-overlay0">
|
||||
{candidates.length} candidates
|
||||
</span>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
<p className="mt-2 text-xs text-ctp-overlay0">
|
||||
Reverse-search e621 with this file to find its original post.
|
||||
</p>
|
||||
|
||||
<Button
|
||||
variant="secondary"
|
||||
className="mt-3 w-full"
|
||||
disabled={busy || !isImage || !configured}
|
||||
onClick={() => void search()}
|
||||
>
|
||||
{busy ? (
|
||||
<Spinner className="h-3.5 w-3.5" />
|
||||
) : (
|
||||
<ScanSearch className="h-4 w-4" />
|
||||
)}
|
||||
{busy ? "Searching…" : "Search e621"}
|
||||
</Button>
|
||||
|
||||
{!configured ? (
|
||||
<p className="mt-2 text-[11px] text-ctp-overlay0">
|
||||
Configure e621 credentials in Account to use IQDB.
|
||||
</p>
|
||||
) : !isImage ? (
|
||||
<p className="mt-2 text-[11px] text-ctp-overlay0">
|
||||
IQDB works on images only.
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
{notice ? (
|
||||
<p
|
||||
className={cn(
|
||||
"mt-2 text-xs",
|
||||
notice.kind === "ok"
|
||||
? "text-ctp-green"
|
||||
: notice.kind === "warn"
|
||||
? "text-ctp-yellow"
|
||||
: "text-ctp-red",
|
||||
)}
|
||||
>
|
||||
{notice.text}
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
{candidates && candidates.length > 0 ? (
|
||||
<div className="mt-3 grid grid-cols-3 gap-1.5">
|
||||
{candidates.map((candidate) => {
|
||||
const exact = candidate.md5 !== null && candidate.md5 === item.md5;
|
||||
return (
|
||||
<button
|
||||
key={`${candidate.post_id}-${candidate.md5}`}
|
||||
type="button"
|
||||
disabled={linkMutation.isPending || candidate.post_id === null}
|
||||
onClick={() => setSelected(candidate)}
|
||||
className={cn(
|
||||
"overflow-hidden rounded-md border bg-ctp-mantle text-left transition",
|
||||
selected?.post_id === candidate.post_id
|
||||
? "border-ctp-mauve ring-2 ring-ctp-mauve/40"
|
||||
: "border-ctp-surface0 hover:border-ctp-surface2",
|
||||
)}
|
||||
>
|
||||
<span className="relative block aspect-square overflow-hidden">
|
||||
{candidate.preview_url ? (
|
||||
<img
|
||||
src={candidate.preview_url}
|
||||
alt={`Post #${candidate.post_id}`}
|
||||
loading="lazy"
|
||||
className="h-full w-full object-cover"
|
||||
/>
|
||||
) : (
|
||||
<span className="flex h-full w-full items-center justify-center text-[10px] text-ctp-overlay0">
|
||||
No preview
|
||||
</span>
|
||||
)}
|
||||
{candidate.rating ? (
|
||||
<span
|
||||
className={cn(
|
||||
"absolute left-1 top-1 rounded-full px-1 py-0.5 font-mono text-[9px] font-semibold uppercase",
|
||||
ratingPill[candidate.rating] ??
|
||||
"bg-ctp-surface1 text-ctp-text",
|
||||
)}
|
||||
>
|
||||
{candidate.rating}
|
||||
</span>
|
||||
) : null}
|
||||
{exact ? (
|
||||
<span className="absolute right-1 top-1 rounded-full bg-ctp-green/90 px-1 py-0.5 font-mono text-[9px] font-semibold uppercase text-ctp-crust">
|
||||
MD5
|
||||
</span>
|
||||
) : null}
|
||||
</span>
|
||||
<span className="block px-1 py-0.5 font-mono text-[10px] text-ctp-overlay0">
|
||||
#{candidate.post_id}
|
||||
{candidate.score !== null
|
||||
? ` · ${Math.round(candidate.score)}%`
|
||||
: ""}
|
||||
</span>
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{selected ? (
|
||||
<div className="mt-3 rounded-md border border-ctp-surface0 bg-ctp-surface0/40 p-3">
|
||||
<div className="flex items-center justify-between gap-2">
|
||||
<p className="font-mono text-xs text-ctp-subtext1">
|
||||
Post #{selected.post_id}
|
||||
</p>
|
||||
{selected.md5 === item.md5 ? (
|
||||
<span className="rounded-full bg-ctp-green/15 px-1.5 py-0.5 font-mono text-[9px] font-semibold uppercase text-ctp-green">
|
||||
MD5 match
|
||||
</span>
|
||||
) : null}
|
||||
</div>
|
||||
<p className="mt-1 text-xs text-ctp-overlay0">
|
||||
{selected.rating
|
||||
? (RATING_LABELS[selected.rating] ?? selected.rating)
|
||||
: "Unknown rating"}
|
||||
{selected.score_total !== null
|
||||
? ` · ▲ ${selected.score_total}`
|
||||
: ""}
|
||||
{selected.fav_count !== null ? ` · ${selected.fav_count} favs` : ""}
|
||||
{selected.width && selected.height
|
||||
? ` · ${selected.width}×${selected.height}`
|
||||
: ""}
|
||||
</p>
|
||||
{selected.tags_preview.length > 0 ? (
|
||||
<div className="mt-2 flex flex-wrap gap-1">
|
||||
{selected.tags_preview.map((tag) => (
|
||||
<span
|
||||
key={tag}
|
||||
className="rounded border border-ctp-surface1 px-1 py-0.5 font-mono text-[10px] text-ctp-subtext0"
|
||||
>
|
||||
{tag}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
) : null}
|
||||
<div className="mt-3 flex items-center gap-3">
|
||||
<Button
|
||||
disabled={
|
||||
!canLink || linkMutation.isPending || selected.post_id === null
|
||||
}
|
||||
onClick={() =>
|
||||
selected.post_id !== null &&
|
||||
linkMutation.mutate(selected.post_id)
|
||||
}
|
||||
>
|
||||
{linkMutation.isPending ? "Linking…" : "Link selected post"}
|
||||
</Button>
|
||||
<Link
|
||||
to={`/detail/${selected.post_id}`}
|
||||
target="_blank"
|
||||
className="text-xs text-ctp-blue hover:underline"
|
||||
>
|
||||
View post
|
||||
</Link>
|
||||
</div>
|
||||
<p className="mt-2 text-[11px] text-ctp-overlay0">
|
||||
{canLink
|
||||
? "Linking stores this post's metadata on the item."
|
||||
: "Only the uploader or staff can link posts."}
|
||||
</p>
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{candidates && candidates.length > 0 && !selected ? (
|
||||
<p className="mt-2 text-[11px] text-ctp-overlay0">
|
||||
Select a candidate first, then confirm the link.
|
||||
</p>
|
||||
) : null}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -1,18 +1,24 @@
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { ArrowLeft, Download } from "lucide-react";
|
||||
import { ArrowLeft, Download, Trash2 } from "lucide-react";
|
||||
import { useEffect, useState } from "react";
|
||||
import { Link, useNavigate } from "react-router-dom";
|
||||
|
||||
import { Button, EmptyState, Spinner, inputClass, linkButtonClass } from "@/components/ui";
|
||||
import { TagChip } from "@/components/TagChip";
|
||||
import { DetailShell } from "@/features/detail/DetailShell";
|
||||
import { IqdbCard } from "@/features/library/IqdbCard";
|
||||
import { api, errorMessage } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import { isTypingTarget } from "@/lib/dom";
|
||||
import { renderDText } from "@/lib/dtext";
|
||||
import { RATING_LABELS, TAG_CATEGORY_LABELS } from "@/lib/e621";
|
||||
import { formatBytes, formatDate } from "@/lib/format";
|
||||
import type { FileNeighbors, MediaItem, Rating } from "@/lib/types";
|
||||
import type {
|
||||
E621MatchStatus,
|
||||
FileNeighbors,
|
||||
MediaItem,
|
||||
Rating,
|
||||
} from "@/lib/types";
|
||||
import { useAuth } from "@/store/auth";
|
||||
import { useUi } from "@/store/ui";
|
||||
|
||||
@@ -45,6 +51,188 @@ const ratingLabel: Record<string, string> = {
|
||||
e: "Explicit",
|
||||
};
|
||||
|
||||
const matchStatusMeta: Record<
|
||||
E621MatchStatus,
|
||||
{ label: string; className: string }
|
||||
> = {
|
||||
unknown: {
|
||||
label: "Not checked",
|
||||
className: "border-ctp-surface2 bg-ctp-surface0 text-ctp-subtext1",
|
||||
},
|
||||
matched: {
|
||||
label: "Matched",
|
||||
className: "border-ctp-green/40 bg-ctp-green/15 text-ctp-green",
|
||||
},
|
||||
not_found: {
|
||||
label: "Not found on e621",
|
||||
className: "border-ctp-yellow/40 bg-ctp-yellow/15 text-ctp-yellow",
|
||||
},
|
||||
deleted: {
|
||||
label: "Deleted on e621",
|
||||
className: "border-ctp-red/40 bg-ctp-red/15 text-ctp-red",
|
||||
},
|
||||
};
|
||||
|
||||
function MatchCard({ item, canEdit }: { item: MediaItem; canEdit: boolean }) {
|
||||
const queryClient = useQueryClient();
|
||||
const [postId, setPostId] = useState("");
|
||||
const [notice, setNotice] = useState<{
|
||||
kind: "ok" | "warn" | "error";
|
||||
text: string;
|
||||
} | null>(null);
|
||||
|
||||
function refresh() {
|
||||
void queryClient.invalidateQueries({ queryKey: ["file", item.j_id] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["tag-cloud"] });
|
||||
}
|
||||
|
||||
const checkMutation = useMutation({
|
||||
mutationFn: () =>
|
||||
api<MediaItem>(`/api/files/${item.j_id}/match/`, {
|
||||
method: "POST",
|
||||
json: {},
|
||||
}),
|
||||
onSuccess: (data) => {
|
||||
refresh();
|
||||
if (data.e621_match_status === "matched") {
|
||||
setNotice({ kind: "ok", text: `Matched to post #${data.e621_post_id}.` });
|
||||
} else if (data.e621_match_status === "deleted") {
|
||||
setNotice({
|
||||
kind: "error",
|
||||
text: "The linked post no longer exists on e621.",
|
||||
});
|
||||
} else {
|
||||
setNotice({
|
||||
kind: "warn",
|
||||
text: "No e621 post with this file's MD5 was found.",
|
||||
});
|
||||
}
|
||||
},
|
||||
onError: (error) =>
|
||||
setNotice({ kind: "error", text: errorMessage(error) }),
|
||||
});
|
||||
|
||||
const linkMutation = useMutation({
|
||||
mutationFn: (id: number) =>
|
||||
api<MediaItem & { md5_match?: boolean }>(`/api/files/${item.j_id}/match/`, {
|
||||
method: "POST",
|
||||
json: { post_id: id },
|
||||
}),
|
||||
onSuccess: (data) => {
|
||||
refresh();
|
||||
setPostId("");
|
||||
setNotice(
|
||||
data.md5_match
|
||||
? {
|
||||
kind: "ok",
|
||||
text: `Linked to post #${data.e621_post_id} — MD5 matches.`,
|
||||
}
|
||||
: {
|
||||
kind: "warn",
|
||||
text: `Linked to post #${data.e621_post_id}, but its MD5 differs from this file.`,
|
||||
},
|
||||
);
|
||||
},
|
||||
onError: (error) =>
|
||||
setNotice({ kind: "error", text: errorMessage(error) }),
|
||||
});
|
||||
|
||||
const unlinkMutation = useMutation({
|
||||
mutationFn: () =>
|
||||
api<MediaItem>(`/api/files/${item.j_id}/unlink/`, { method: "POST" }),
|
||||
onSuccess: () => {
|
||||
refresh();
|
||||
setNotice({ kind: "ok", text: "e621 link removed." });
|
||||
},
|
||||
onError: (error) =>
|
||||
setNotice({ kind: "error", text: errorMessage(error) }),
|
||||
});
|
||||
|
||||
const meta = matchStatusMeta[item.e621_match_status];
|
||||
|
||||
return (
|
||||
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
|
||||
<div className="flex items-center justify-between gap-2">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">e621 match</h2>
|
||||
<span
|
||||
className={cn(
|
||||
"rounded-full border px-2 py-0.5 text-[11px] font-medium",
|
||||
meta.className,
|
||||
)}
|
||||
>
|
||||
{meta.label}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<p className="mt-2 text-xs text-ctp-overlay0">
|
||||
{item.e621_checked_at
|
||||
? `Last checked ${formatDate(item.e621_checked_at)}`
|
||||
: "This item has not been checked against e621 yet."}
|
||||
</p>
|
||||
|
||||
{canEdit ? (
|
||||
<div className="mt-3 flex flex-col gap-2">
|
||||
<div className="flex gap-2">
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={checkMutation.isPending}
|
||||
onClick={() => checkMutation.mutate()}
|
||||
>
|
||||
{checkMutation.isPending
|
||||
? "Checking…"
|
||||
: item.e621_post_id
|
||||
? "Recheck"
|
||||
: "Check MD5"}
|
||||
</Button>
|
||||
{item.e621_post_id ? (
|
||||
<Button
|
||||
variant="ghost"
|
||||
disabled={unlinkMutation.isPending}
|
||||
onClick={() => unlinkMutation.mutate()}
|
||||
>
|
||||
Unlink
|
||||
</Button>
|
||||
) : null}
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
<input
|
||||
className={inputClass}
|
||||
placeholder="Post ID"
|
||||
inputMode="numeric"
|
||||
value={postId}
|
||||
onChange={(event) =>
|
||||
setPostId(event.target.value.replace(/[^0-9]/g, ""))
|
||||
}
|
||||
/>
|
||||
<Button
|
||||
disabled={!postId || linkMutation.isPending}
|
||||
onClick={() => linkMutation.mutate(Number(postId))}
|
||||
>
|
||||
Link
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{notice ? (
|
||||
<p
|
||||
className={cn(
|
||||
"mt-2 text-xs",
|
||||
notice.kind === "ok"
|
||||
? "text-ctp-green"
|
||||
: notice.kind === "warn"
|
||||
? "text-ctp-yellow"
|
||||
: "text-ctp-red",
|
||||
)}
|
||||
>
|
||||
{notice.text}
|
||||
</p>
|
||||
) : null}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
function E621MetadataCard({ item }: { item: MediaItem }) {
|
||||
const data = item.e621_data;
|
||||
if (!item.e621_post_id || !data) return null;
|
||||
@@ -77,6 +265,13 @@ function E621MetadataCard({ item }: { item: MediaItem }) {
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{item.e621_match_status === "deleted" ? (
|
||||
<p className="mt-3 rounded border border-ctp-red/40 bg-ctp-red/10 px-2.5 py-1.5 text-xs text-ctp-red">
|
||||
This post has been deleted on e621 — the metadata below is a cached
|
||||
copy.
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
<dl className="mt-3 grid grid-cols-2 gap-x-4 gap-y-2 text-xs sm:grid-cols-4">
|
||||
<div>
|
||||
<dt className="text-ctp-overlay0">Rating</dt>
|
||||
@@ -112,7 +307,7 @@ function E621MetadataCard({ item }: { item: MediaItem }) {
|
||||
</p>
|
||||
<div className="flex flex-wrap gap-1.5">
|
||||
{tags.map((tag) => (
|
||||
<TagChip key={tag} tag={tag} category={category} />
|
||||
<TagChip key={tag} tag={tag} category={category} followable />
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
@@ -296,15 +491,21 @@ function CustomDataCard({ item, canEdit }: { item: MediaItem; canEdit: boolean }
|
||||
|
||||
export function LibraryDetail({ jId }: { jId: string }) {
|
||||
const navigate = useNavigate();
|
||||
const queryClient = useQueryClient();
|
||||
const ordering = useUi((state) => state.ordering);
|
||||
const user = useAuth((state) => state.user);
|
||||
const rawUrl = `/api/files/${jId}/raw/`;
|
||||
const [confirmDelete, setConfirmDelete] = useState(false);
|
||||
|
||||
const query = useQuery({
|
||||
queryKey: ["file", jId],
|
||||
queryFn: () => api<MediaItem>(`/api/files/${jId}/`),
|
||||
});
|
||||
|
||||
const rawUrl = query.data?.raw_url ?? `/api/files/${jId}/raw/`;
|
||||
const downloadUrl = rawUrl.includes("?")
|
||||
? `${rawUrl}&download=1`
|
||||
: `${rawUrl}?download=1`;
|
||||
|
||||
const neighborsQuery = useQuery({
|
||||
queryKey: ["neighbors", jId, ordering],
|
||||
queryFn: () =>
|
||||
@@ -313,13 +514,27 @@ export function LibraryDetail({ jId }: { jId: string }) {
|
||||
),
|
||||
});
|
||||
|
||||
const deleteMutation = useMutation({
|
||||
mutationFn: () =>
|
||||
api<{ deleted: string[]; errors: { j_id?: string; error: string }[] }>(
|
||||
"/api/delete/",
|
||||
{ method: "POST", json: { j_ids: [jId] } },
|
||||
),
|
||||
onSuccess: () => {
|
||||
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["duplicates-md5"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["storage"] });
|
||||
navigate("/");
|
||||
},
|
||||
});
|
||||
|
||||
useEffect(() => {
|
||||
function handleKeyDown(event: KeyboardEvent) {
|
||||
if (event.ctrlKey || event.metaKey || event.altKey) return;
|
||||
if (isTypingTarget(event.target)) return;
|
||||
if (event.key === "d" || event.key === "D") {
|
||||
event.preventDefault();
|
||||
window.location.href = `${rawUrl}?download=1`;
|
||||
window.location.href = downloadUrl;
|
||||
return;
|
||||
}
|
||||
const neighbors = neighborsQuery.data;
|
||||
@@ -331,7 +546,7 @@ export function LibraryDetail({ jId }: { jId: string }) {
|
||||
}
|
||||
window.addEventListener("keydown", handleKeyDown);
|
||||
return () => window.removeEventListener("keydown", handleKeyDown);
|
||||
}, [neighborsQuery.data, navigate, rawUrl]);
|
||||
}, [neighborsQuery.data, navigate, downloadUrl]);
|
||||
|
||||
if (query.isPending) {
|
||||
return (
|
||||
@@ -361,6 +576,12 @@ export function LibraryDetail({ jId }: { jId: string }) {
|
||||
user.is_staff ||
|
||||
user.role === "staff"),
|
||||
);
|
||||
const canDelete = Boolean(
|
||||
user &&
|
||||
(user.is_superuser ||
|
||||
user.role === "staff" ||
|
||||
user.id === item.uploaded_by_id),
|
||||
);
|
||||
|
||||
return (
|
||||
<DetailShell
|
||||
@@ -378,9 +599,45 @@ export function LibraryDetail({ jId }: { jId: string }) {
|
||||
}
|
||||
actions={
|
||||
<div className="flex flex-wrap items-center gap-2">
|
||||
<a href={`${rawUrl}?download=1`} className={linkButtonClass}>
|
||||
<a href={downloadUrl} className={linkButtonClass}>
|
||||
<Download className="h-4 w-4" /> Download
|
||||
</a>
|
||||
{canDelete ? (
|
||||
confirmDelete ? (
|
||||
<>
|
||||
<span className="text-xs text-ctp-subtext0">
|
||||
Delete this item and all its copies?
|
||||
</span>
|
||||
<Button
|
||||
variant="danger"
|
||||
disabled={deleteMutation.isPending}
|
||||
onClick={() => deleteMutation.mutate()}
|
||||
>
|
||||
{deleteMutation.isPending ? "Deleting…" : "Confirm delete"}
|
||||
</Button>
|
||||
<Button
|
||||
variant="ghost"
|
||||
onClick={() => setConfirmDelete(false)}
|
||||
disabled={deleteMutation.isPending}
|
||||
>
|
||||
Cancel
|
||||
</Button>
|
||||
</>
|
||||
) : (
|
||||
<Button
|
||||
variant="danger"
|
||||
onClick={() => setConfirmDelete(true)}
|
||||
title="Delete this item from the library"
|
||||
>
|
||||
<Trash2 className="h-4 w-4" /> Delete
|
||||
</Button>
|
||||
)
|
||||
) : null}
|
||||
{deleteMutation.isError ? (
|
||||
<p className="text-sm text-ctp-red">
|
||||
{errorMessage(deleteMutation.error)}
|
||||
</p>
|
||||
) : null}
|
||||
</div>
|
||||
}
|
||||
media={
|
||||
@@ -439,6 +696,12 @@ export function LibraryDetail({ jId }: { jId: string }) {
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<MatchCard key={item.j_id} item={item} canEdit={canEdit} />
|
||||
|
||||
{user ? (
|
||||
<IqdbCard key={item.j_id} item={item} canLink={canEdit} />
|
||||
) : null}
|
||||
|
||||
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Locations ({item.locations.length})
|
||||
|
||||
@@ -5,6 +5,7 @@ import { useState, useEffect, type FormEvent } from "react";
|
||||
import { CollapsibleSidebar } from "@/components/CollapsibleSidebar";
|
||||
import { MediaCard } from "@/components/MediaCard";
|
||||
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
|
||||
import { MatchPanel } from "@/features/library/MatchPanel";
|
||||
import { api } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import { isTypingTarget } from "@/lib/dom";
|
||||
@@ -41,13 +42,44 @@ const ratingFilters = [
|
||||
},
|
||||
];
|
||||
|
||||
const statusFilters = [
|
||||
{
|
||||
value: "matched",
|
||||
label: "Matched",
|
||||
active: "border-ctp-green/40 bg-ctp-green/15 text-ctp-green",
|
||||
},
|
||||
{
|
||||
value: "not_found",
|
||||
label: "Not found",
|
||||
active: "border-ctp-yellow/40 bg-ctp-yellow/15 text-ctp-yellow",
|
||||
},
|
||||
{
|
||||
value: "deleted",
|
||||
label: "Deleted",
|
||||
active: "border-ctp-red/40 bg-ctp-red/15 text-ctp-red",
|
||||
},
|
||||
{
|
||||
value: "custom",
|
||||
label: "Custom",
|
||||
active: "border-ctp-mauve/40 bg-ctp-mauve/15 text-ctp-mauve",
|
||||
},
|
||||
{
|
||||
value: "unknown",
|
||||
label: "Unknown",
|
||||
active: "border-ctp-surface2 bg-ctp-surface0 text-ctp-subtext1",
|
||||
},
|
||||
];
|
||||
|
||||
export default function LibraryPage() {
|
||||
const [searchInput, setSearchInput] = useState("");
|
||||
const [search, setSearch] = useState("");
|
||||
const [searchType, setSearchType] = useState("filename");
|
||||
const [page, setPage] = useState(1);
|
||||
|
||||
const ratings = useUi((state) => state.ratings);
|
||||
const setRatings = useUi((state) => state.setRatings);
|
||||
const statuses = useUi((state) => state.statuses);
|
||||
const setStatuses = useUi((state) => state.setStatuses);
|
||||
const perPage = useUi((state) => state.perPage);
|
||||
const setPerPage = useUi((state) => state.setPerPage);
|
||||
const ordering = useUi((state) => state.ordering);
|
||||
@@ -68,19 +100,35 @@ export default function LibraryPage() {
|
||||
}, []);
|
||||
|
||||
const query = useQuery({
|
||||
queryKey: ["files", { search, ratings, ordering, perPage, page }],
|
||||
queryKey: [
|
||||
"files",
|
||||
{ search, searchType, ratings, statuses, ordering, perPage, page },
|
||||
],
|
||||
queryFn: () => {
|
||||
const params = new URLSearchParams({
|
||||
page: String(page),
|
||||
per_page: String(perPage),
|
||||
ordering,
|
||||
});
|
||||
if (search) params.set("search", search);
|
||||
if (search) {
|
||||
params.set("search", search);
|
||||
params.set("search_type", searchType);
|
||||
}
|
||||
if (ratings.length) params.set("rating", ratings.join(","));
|
||||
if (statuses.length) params.set("status", statuses.join(","));
|
||||
return api<Paginated<MediaItem>>(`/api/files/?${params.toString()}`);
|
||||
},
|
||||
});
|
||||
|
||||
const tagCloudQuery = useQuery({
|
||||
queryKey: ["tag-cloud"],
|
||||
queryFn: () =>
|
||||
api<{ count: number; tags: { tag: string; count: number }[] }>(
|
||||
"/api/tags/cloud/",
|
||||
),
|
||||
staleTime: 60_000,
|
||||
});
|
||||
|
||||
const pageCount = query.data
|
||||
? Math.max(1, Math.ceil(query.data.count / perPage))
|
||||
: 1;
|
||||
@@ -100,10 +148,28 @@ export default function LibraryPage() {
|
||||
);
|
||||
}
|
||||
|
||||
function toggleStatus(value: string) {
|
||||
setPage(1);
|
||||
setStatuses(
|
||||
statuses.includes(value)
|
||||
? statuses.filter((status) => status !== value)
|
||||
: [...statuses, value],
|
||||
);
|
||||
}
|
||||
|
||||
function searchTag(tag: string) {
|
||||
setSearchInput(tag);
|
||||
setSearch(tag);
|
||||
setSearchType("tags");
|
||||
setPage(1);
|
||||
}
|
||||
|
||||
function clearFilters() {
|
||||
setSearchInput("");
|
||||
setSearch("");
|
||||
setSearchType("filename");
|
||||
setRatings([]);
|
||||
setStatuses([]);
|
||||
setPage(1);
|
||||
}
|
||||
|
||||
@@ -118,11 +184,22 @@ export default function LibraryPage() {
|
||||
<input
|
||||
id="library-search"
|
||||
className={cn(inputClass, "pl-8")}
|
||||
placeholder="Filename…"
|
||||
placeholder={
|
||||
searchType === "tags" ? "Tags…" : "Filename…"
|
||||
}
|
||||
value={searchInput}
|
||||
onChange={(event) => setSearchInput(event.target.value)}
|
||||
/>
|
||||
</div>
|
||||
<select
|
||||
className={inputClass}
|
||||
value={searchType}
|
||||
onChange={(event) => setSearchType(event.target.value)}
|
||||
>
|
||||
<option value="filename">Filename</option>
|
||||
<option value="tags">Tags</option>
|
||||
<option value="both">Filename or tags</option>
|
||||
</select>
|
||||
<div className="flex gap-2">
|
||||
<Button type="submit" className="flex-1">
|
||||
Search
|
||||
@@ -133,6 +210,29 @@ export default function LibraryPage() {
|
||||
</div>
|
||||
</form>
|
||||
|
||||
<div className="flex flex-col gap-2">
|
||||
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||
Status
|
||||
</span>
|
||||
<div className="flex flex-wrap gap-1.5">
|
||||
{statusFilters.map((option) => (
|
||||
<button
|
||||
key={option.value}
|
||||
type="button"
|
||||
onClick={() => toggleStatus(option.value)}
|
||||
className={cn(
|
||||
"rounded-full border px-2.5 py-1 text-xs font-medium transition",
|
||||
statuses.includes(option.value)
|
||||
? option.active
|
||||
: "border-ctp-surface1 text-ctp-subtext0 hover:border-ctp-surface2 hover:text-ctp-text",
|
||||
)}
|
||||
>
|
||||
{option.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col gap-2">
|
||||
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||
Rating
|
||||
@@ -156,6 +256,31 @@ export default function LibraryPage() {
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col gap-2">
|
||||
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||
Tag cloud
|
||||
</span>
|
||||
<div className="flex flex-wrap gap-1.5">
|
||||
{(tagCloudQuery.data?.tags ?? []).map((entry) => (
|
||||
<button
|
||||
key={entry.tag}
|
||||
type="button"
|
||||
onClick={() => searchTag(entry.tag)}
|
||||
title={`Search ${entry.tag}`}
|
||||
className="rounded border border-ctp-blue/30 bg-ctp-blue/10 px-1.5 py-0.5 font-mono text-[11px] text-ctp-blue transition hover:bg-ctp-blue/20"
|
||||
>
|
||||
{entry.tag} ({entry.count})
|
||||
</button>
|
||||
))}
|
||||
{tagCloudQuery.isPending ? (
|
||||
<Spinner className="h-3.5 w-3.5" />
|
||||
) : null}
|
||||
{tagCloudQuery.data && tagCloudQuery.data.tags.length === 0 ? (
|
||||
<p className="text-xs text-ctp-overlay0">No tags yet.</p>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<label className="flex flex-col gap-2">
|
||||
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||
Sort
|
||||
@@ -209,11 +334,14 @@ export default function LibraryPage() {
|
||||
? `${query.data.count} item${query.data.count === 1 ? "" : "s"}`
|
||||
: "…"}
|
||||
</p>
|
||||
{pageCount > 1 ? (
|
||||
<p className="font-mono text-xs text-ctp-overlay0">
|
||||
Page {page} / {pageCount}
|
||||
</p>
|
||||
) : null}
|
||||
<div className="flex items-center gap-3">
|
||||
{pageCount > 1 ? (
|
||||
<p className="font-mono text-xs text-ctp-overlay0">
|
||||
Page {page} / {pageCount}
|
||||
</p>
|
||||
) : null}
|
||||
{user?.can_upload ? <MatchPanel /> : null}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{!user ? (
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { RefreshCw, X } from "lucide-react";
|
||||
import { useEffect, useState } from "react";
|
||||
|
||||
import { Button, Spinner } from "@/components/ui";
|
||||
import { api, errorMessage } from "@/lib/api";
|
||||
import type { MatchTask } from "@/lib/types";
|
||||
|
||||
type Scope = "missing" | "all";
|
||||
|
||||
function summary(task: MatchTask): string {
|
||||
const parts = [
|
||||
`${task.matched} matched`,
|
||||
`${task.not_found} not found`,
|
||||
`${task.deleted} deleted`,
|
||||
];
|
||||
if (task.failed > 0) parts.push(`${task.failed} failed`);
|
||||
return parts.join(" · ");
|
||||
}
|
||||
|
||||
export function MatchPanel() {
|
||||
const queryClient = useQueryClient();
|
||||
const [taskId, setTaskId] = useState<string | null>(null);
|
||||
const [menuOpen, setMenuOpen] = useState(false);
|
||||
|
||||
const taskQuery = useQuery({
|
||||
queryKey: ["match-task", taskId],
|
||||
queryFn: () => api<MatchTask>(`/api/matches/${taskId}/`),
|
||||
enabled: Boolean(taskId),
|
||||
refetchInterval: (query) => {
|
||||
const status = query.state.data?.status;
|
||||
return status === "pending" || status === "running" ? 1000 : false;
|
||||
},
|
||||
});
|
||||
|
||||
const task = taskQuery.data;
|
||||
const running = task?.status === "pending" || task?.status === "running";
|
||||
const finishedTaskId = task && !running ? task.task_id : null;
|
||||
|
||||
useEffect(() => {
|
||||
if (!finishedTaskId) return;
|
||||
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["tag-cloud"] });
|
||||
}, [finishedTaskId, queryClient]);
|
||||
|
||||
const startMutation = useMutation({
|
||||
mutationFn: (scope: Scope) =>
|
||||
api<MatchTask>("/api/matches/", { method: "POST", json: { scope } }),
|
||||
onSuccess: (data) => {
|
||||
setTaskId(data.task_id);
|
||||
setMenuOpen(false);
|
||||
},
|
||||
});
|
||||
|
||||
const cancelMutation = useMutation({
|
||||
mutationFn: () =>
|
||||
api<{ success: boolean }>(`/api/matches/${taskId}/cancel/`, {
|
||||
method: "POST",
|
||||
}),
|
||||
});
|
||||
|
||||
if (running && task) {
|
||||
const percent = task.total
|
||||
? Math.round((task.processed * 100) / task.total)
|
||||
: 0;
|
||||
return (
|
||||
<div className="flex items-center gap-3">
|
||||
<div className="w-36">
|
||||
<div className="h-1.5 overflow-hidden rounded-full bg-ctp-surface0">
|
||||
<div
|
||||
className="h-full bg-ctp-mauve transition-all"
|
||||
style={{ width: `${percent}%` }}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<span className="font-mono text-xs text-ctp-subtext0">
|
||||
{task.processed}/{task.total} · {task.matched} matched
|
||||
</span>
|
||||
<Button
|
||||
variant="ghost"
|
||||
disabled={cancelMutation.isPending}
|
||||
onClick={() => cancelMutation.mutate()}
|
||||
>
|
||||
Cancel
|
||||
</Button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (task) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 text-xs">
|
||||
{task.status === "complete" ? (
|
||||
<span className="text-ctp-green">Scan complete — {summary(task)}</span>
|
||||
) : task.status === "cancelled" ? (
|
||||
<span className="text-ctp-subtext0">Scan cancelled.</span>
|
||||
) : (
|
||||
<span className="text-ctp-red">
|
||||
{task.error || "The scan failed."}
|
||||
</span>
|
||||
)}
|
||||
<button
|
||||
type="button"
|
||||
className="text-ctp-overlay0 transition hover:text-ctp-text"
|
||||
onClick={() => setTaskId(null)}
|
||||
title="Dismiss"
|
||||
>
|
||||
<X className="h-3.5 w-3.5" />
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="relative">
|
||||
<Button variant="secondary" onClick={() => setMenuOpen((open) => !open)}>
|
||||
{startMutation.isPending ? (
|
||||
<Spinner className="h-3.5 w-3.5" />
|
||||
) : (
|
||||
<RefreshCw className="h-4 w-4" />
|
||||
)}
|
||||
Match e621
|
||||
</Button>
|
||||
|
||||
{menuOpen ? (
|
||||
<div className="absolute right-0 top-full z-20 mt-2 w-72 rounded-lg border border-ctp-surface1 bg-ctp-base p-1.5 shadow-lg">
|
||||
<button
|
||||
type="button"
|
||||
className="flex w-full flex-col rounded-md px-2.5 py-2 text-left transition hover:bg-ctp-surface0"
|
||||
onClick={() => startMutation.mutate("missing")}
|
||||
>
|
||||
<span className="text-sm text-ctp-text">
|
||||
Check unchecked items
|
||||
</span>
|
||||
<span className="text-xs text-ctp-overlay0">
|
||||
MD5 lookups for items never checked against e621.
|
||||
</span>
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
className="flex w-full flex-col rounded-md px-2.5 py-2 text-left transition hover:bg-ctp-surface0"
|
||||
onClick={() => startMutation.mutate("all")}
|
||||
>
|
||||
<span className="text-sm text-ctp-text">Recheck the library</span>
|
||||
<span className="text-xs text-ctp-overlay0">
|
||||
Refreshes matched metadata and flags deleted posts too.
|
||||
</span>
|
||||
</button>
|
||||
{startMutation.isError ? (
|
||||
<p className="px-2.5 py-1.5 text-xs text-ctp-red">
|
||||
{errorMessage(startMutation.error)}
|
||||
</p>
|
||||
) : null}
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -7,13 +7,13 @@ import {
|
||||
Heart,
|
||||
HeartOff,
|
||||
} from "lucide-react";
|
||||
import { useMemo } from "react";
|
||||
import { useMemo, useEffect, useState } from "react";
|
||||
import { Link, useLocation, useNavigate } from "react-router-dom";
|
||||
|
||||
import { Button, EmptyState, Spinner, linkButtonClass } from "@/components/ui";
|
||||
import { TagChip } from "@/components/TagChip";
|
||||
import { DetailShell } from "@/features/detail/DetailShell";
|
||||
import { api } from "@/lib/api";
|
||||
import { api, errorMessage } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import { renderDText } from "@/lib/dtext";
|
||||
import {
|
||||
@@ -27,7 +27,7 @@ import {
|
||||
TAG_CATEGORY_ORDER,
|
||||
} from "@/lib/e621";
|
||||
import { formatBytes, formatDate } from "@/lib/format";
|
||||
import type { MediaItem } from "@/lib/types";
|
||||
import type { DownloadTask } from "@/lib/types";
|
||||
import { useAuth } from "@/store/auth";
|
||||
import { useE621 } from "@/store/e621";
|
||||
|
||||
@@ -116,9 +116,11 @@ export function OnlineDetail({ postId }: { postId: number }) {
|
||||
},
|
||||
});
|
||||
|
||||
const [downloadTaskId, setDownloadTaskId] = useState<string | null>(null);
|
||||
|
||||
const downloadMutation = useMutation({
|
||||
mutationFn: (target: NonNullable<typeof post>) =>
|
||||
api<MediaItem>("/api/online/downloads/", {
|
||||
api<DownloadTask>("/api/online/downloads/", {
|
||||
method: "POST",
|
||||
json: {
|
||||
post_id: target.id,
|
||||
@@ -127,13 +129,51 @@ export function OnlineDetail({ postId }: { postId: number }) {
|
||||
post: target,
|
||||
},
|
||||
}),
|
||||
onSuccess: (item) => {
|
||||
void queryClient.invalidateQueries({ queryKey: ["library-lookup"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||
navigate(`/detail/${item.j_id}`);
|
||||
onSuccess: (task) => {
|
||||
setDownloadTaskId(task.task_id);
|
||||
void queryClient.invalidateQueries({ queryKey: ["status"] });
|
||||
},
|
||||
});
|
||||
|
||||
const taskQuery = useQuery({
|
||||
queryKey: ["download-task", downloadTaskId],
|
||||
queryFn: () =>
|
||||
api<DownloadTask>(`/api/online/downloads/${downloadTaskId}/`),
|
||||
enabled: Boolean(downloadTaskId),
|
||||
refetchInterval: (query) => {
|
||||
const status = query.state.data?.status;
|
||||
return status === "pending" || status === "downloading" ? 700 : false;
|
||||
},
|
||||
});
|
||||
const downloadTask = taskQuery.data;
|
||||
const downloadStatus = downloadTask?.status;
|
||||
const downloadActive =
|
||||
downloadStatus === "pending" || downloadStatus === "downloading";
|
||||
|
||||
useEffect(() => {
|
||||
if (!downloadStatus) return;
|
||||
if (
|
||||
downloadStatus === "complete" ||
|
||||
downloadStatus === "error" ||
|
||||
downloadStatus === "cancelled"
|
||||
) {
|
||||
void queryClient.invalidateQueries({ queryKey: ["status"] });
|
||||
}
|
||||
if (downloadStatus === "complete") {
|
||||
void queryClient.invalidateQueries({ queryKey: ["library-lookup"] });
|
||||
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||
}
|
||||
}, [downloadStatus, queryClient]);
|
||||
|
||||
function cancelDownload() {
|
||||
if (!downloadTaskId) return;
|
||||
void api(`/api/online/downloads/${downloadTaskId}/cancel/`, {
|
||||
method: "POST",
|
||||
}).then(() => {
|
||||
void queryClient.invalidateQueries({ queryKey: ["status"] });
|
||||
});
|
||||
}
|
||||
|
||||
if (query.isPending) {
|
||||
return (
|
||||
<div className="flex justify-center py-24">
|
||||
@@ -163,6 +203,11 @@ export function OnlineDetail({ postId }: { postId: number }) {
|
||||
const imageSrc =
|
||||
post.sample.has && post.sample.url ? post.sample.url : post.file.url;
|
||||
const canUpload = Boolean(user?.can_upload);
|
||||
const clientDownloadUrl = post.file.url
|
||||
? `/api/online/file/?url=${encodeURIComponent(post.file.url)}&filename=${encodeURIComponent(
|
||||
`${post.file.md5}.${post.file.ext}`,
|
||||
)}`
|
||||
: null;
|
||||
|
||||
return (
|
||||
<DetailShell
|
||||
@@ -213,7 +258,10 @@ export function OnlineDetail({ postId }: { postId: number }) {
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={
|
||||
!canUpload || downloadMutation.isPending || !post.file.url
|
||||
!canUpload ||
|
||||
downloadMutation.isPending ||
|
||||
downloadActive ||
|
||||
!post.file.url
|
||||
}
|
||||
onClick={() => downloadMutation.mutate(post)}
|
||||
title={
|
||||
@@ -223,12 +271,22 @@ export function OnlineDetail({ postId }: { postId: number }) {
|
||||
}
|
||||
>
|
||||
<Download className="h-4 w-4" />
|
||||
{downloadMutation.isPending
|
||||
{downloadMutation.isPending || downloadActive
|
||||
? "Downloading…"
|
||||
: "Download to Library"}
|
||||
</Button>
|
||||
)}
|
||||
|
||||
{clientDownloadUrl ? (
|
||||
<a
|
||||
href={clientDownloadUrl}
|
||||
className={linkButtonClass}
|
||||
title="Save the original file straight to your computer"
|
||||
>
|
||||
<Download className="h-4 w-4" /> Download to client
|
||||
</a>
|
||||
) : null}
|
||||
|
||||
<a
|
||||
href={`https://e621.net/posts/${post.id}`}
|
||||
target="_blank"
|
||||
@@ -239,6 +297,59 @@ export function OnlineDetail({ postId }: { postId: number }) {
|
||||
</a>
|
||||
</div>
|
||||
|
||||
{downloadTask &&
|
||||
(downloadTask.status === "pending" ||
|
||||
downloadTask.status === "downloading") ? (
|
||||
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||
<div className="flex flex-wrap items-center justify-between gap-3 text-xs">
|
||||
<span className="text-ctp-subtext1">
|
||||
Downloading to the library… {downloadTask.progress}%
|
||||
</span>
|
||||
<span className="font-mono text-ctp-overlay0">
|
||||
{formatBytes(downloadTask.downloaded)}
|
||||
{downloadTask.total > 0
|
||||
? ` / ${formatBytes(downloadTask.total)}`
|
||||
: ""}
|
||||
{downloadTask.speed
|
||||
? ` · ${(downloadTask.speed / 1024 / 1024).toFixed(1)} MB/s`
|
||||
: ""}
|
||||
</span>
|
||||
</div>
|
||||
<div className="mt-2 h-1.5 w-full overflow-hidden rounded-full bg-ctp-surface0">
|
||||
<div
|
||||
className="h-full bg-ctp-mauve transition-[width]"
|
||||
style={{ width: `${downloadTask.progress}%` }}
|
||||
/>
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
onClick={cancelDownload}
|
||||
className="mt-2 rounded-md px-1.5 py-0.5 font-mono text-[10px] text-ctp-overlay0 transition hover:bg-ctp-surface0 hover:text-ctp-text"
|
||||
>
|
||||
cancel
|
||||
</button>
|
||||
</div>
|
||||
) : downloadTask?.status === "complete" ? (
|
||||
<p className="flex items-center gap-2 rounded-md border border-ctp-green/30 bg-ctp-green/10 px-3 py-2 text-sm text-ctp-green">
|
||||
<CheckCircle2 className="h-4 w-4 shrink-0" />
|
||||
Downloaded to the library
|
||||
{downloadTask.library_j_id ? (
|
||||
<Link
|
||||
to={`/detail/${downloadTask.library_j_id}`}
|
||||
className="font-mono text-ctp-blue hover:underline"
|
||||
>
|
||||
{downloadTask.library_j_id}
|
||||
</Link>
|
||||
) : null}
|
||||
</p>
|
||||
) : downloadTask?.status === "cancelled" ? (
|
||||
<p className="text-xs text-ctp-overlay0">Download cancelled.</p>
|
||||
) : downloadTask?.status === "error" ? (
|
||||
<p className="rounded-md border border-ctp-red/30 bg-ctp-red/10 px-3 py-2 text-sm text-ctp-red">
|
||||
{downloadTask.error || "Download failed."}
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
{favoriteMutation.isError ? (
|
||||
<p className="rounded-md border border-ctp-red/30 bg-ctp-red/10 px-3 py-2 text-sm text-ctp-red">
|
||||
{e621ErrorMessage(favoriteMutation.error)}
|
||||
@@ -246,7 +357,7 @@ export function OnlineDetail({ postId }: { postId: number }) {
|
||||
) : null}
|
||||
{downloadMutation.isError ? (
|
||||
<p className="rounded-md border border-ctp-red/30 bg-ctp-red/10 px-3 py-2 text-sm text-ctp-red">
|
||||
{e621ErrorMessage(downloadMutation.error)}
|
||||
{errorMessage(downloadMutation.error)}
|
||||
</p>
|
||||
) : null}
|
||||
</>
|
||||
@@ -292,6 +403,7 @@ export function OnlineDetail({ postId }: { postId: number }) {
|
||||
key={tag}
|
||||
tag={tag}
|
||||
category={category}
|
||||
followable
|
||||
onClick={() =>
|
||||
navigate(`/online?tags=${encodeURIComponent(tag)}`)
|
||||
}
|
||||
|
||||
@@ -31,7 +31,7 @@ export function PostCard({
|
||||
to={`/detail/${post.id}`}
|
||||
state={from ? { from } : undefined}
|
||||
className={cn(
|
||||
"group overflow-hidden rounded-lg border bg-ctp-base transition",
|
||||
"group block overflow-hidden rounded-lg border bg-ctp-base transition",
|
||||
ratingBorder[post.rating] ?? "border-ctp-surface0",
|
||||
)}
|
||||
>
|
||||
|
||||
@@ -0,0 +1,314 @@
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { ArrowLeft, Archive, Eye, EyeOff } from "lucide-react";
|
||||
import { useMemo, useState } from "react";
|
||||
import { Link, useLocation, useParams } from "react-router-dom";
|
||||
|
||||
import { Button, EmptyState, Spinner } from "@/components/ui";
|
||||
import { PostCard } from "@/features/online/PostCard";
|
||||
import { PoolFollowButton } from "@/features/follows/PoolFollowButton";
|
||||
import { api } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import { renderDText } from "@/lib/dtext";
|
||||
import {
|
||||
e621ErrorMessage,
|
||||
effectiveCredentials,
|
||||
fetchPool,
|
||||
fetchPostsByIds,
|
||||
postIsBlacklisted,
|
||||
type E621Pool,
|
||||
type E621Post,
|
||||
} from "@/lib/e621";
|
||||
import { formatDate } from "@/lib/format";
|
||||
import { useE621 } from "@/store/e621";
|
||||
|
||||
const CHUNK = 100;
|
||||
|
||||
async function fetchPostsChunked(
|
||||
credentials: ReturnType<typeof effectiveCredentials>,
|
||||
ids: number[],
|
||||
signal?: AbortSignal,
|
||||
): Promise<E621Post[]> {
|
||||
const posts: E621Post[] = [];
|
||||
for (let index = 0; index < ids.length; index += CHUNK) {
|
||||
const chunk = ids.slice(index, index + CHUNK);
|
||||
posts.push(...(await fetchPostsByIds(credentials, chunk, signal)));
|
||||
}
|
||||
return posts;
|
||||
}
|
||||
|
||||
const categoryBadge: Record<string, string> = {
|
||||
series: "border-ctp-mauve/40 bg-ctp-mauve/15 text-ctp-mauve",
|
||||
collection: "border-ctp-blue/40 bg-ctp-blue/15 text-ctp-blue",
|
||||
};
|
||||
|
||||
export default function PoolDetailPage() {
|
||||
const { poolId = "" } = useParams();
|
||||
const id = Number(poolId);
|
||||
const location = useLocation();
|
||||
const credentials = useE621((state) => state.credentials);
|
||||
const blacklist = useE621((state) => state.blacklist);
|
||||
const blacklistSet = useMemo(() => new Set(blacklist), [blacklist]);
|
||||
|
||||
const [visibleCount, setVisibleCount] = useState(CHUNK);
|
||||
const [showBlacklisted, setShowBlacklisted] = useState(false);
|
||||
|
||||
const poolQuery = useQuery({
|
||||
queryKey: ["e621", "pool", id, credentials?.username ?? ""],
|
||||
queryFn: ({ signal }) =>
|
||||
fetchPool(effectiveCredentials(credentials), id, signal),
|
||||
enabled: Number.isFinite(id) && id > 0,
|
||||
staleTime: 5 * 60_000,
|
||||
});
|
||||
const pool: E621Pool | undefined = poolQuery.data;
|
||||
|
||||
const postIds = useMemo(
|
||||
() => (pool?.post_ids ?? []).slice(0, visibleCount),
|
||||
[pool, visibleCount],
|
||||
);
|
||||
|
||||
const postsQuery = useQuery({
|
||||
queryKey: [
|
||||
"e621",
|
||||
"pool-posts",
|
||||
id,
|
||||
postIds.join(","),
|
||||
credentials?.username ?? "",
|
||||
],
|
||||
queryFn: ({ signal }) =>
|
||||
fetchPostsChunked(
|
||||
effectiveCredentials(credentials),
|
||||
postIds,
|
||||
signal,
|
||||
),
|
||||
enabled: postIds.length > 0,
|
||||
staleTime: 5 * 60_000,
|
||||
});
|
||||
|
||||
// Keep the pool's own post order, dropping ids e621 no longer serves.
|
||||
const orderedPosts = useMemo(() => {
|
||||
const byId = new Map<number, E621Post>();
|
||||
for (const post of postsQuery.data ?? []) byId.set(post.id, post);
|
||||
return postIds
|
||||
.map((postIdValue) => byId.get(postIdValue))
|
||||
.filter((post): post is E621Post => Boolean(post));
|
||||
}, [postsQuery.data, postIds]);
|
||||
|
||||
const md5s = useMemo(
|
||||
() => orderedPosts.map((post) => post.file.md5),
|
||||
[orderedPosts],
|
||||
);
|
||||
const lookupQuery = useQuery({
|
||||
queryKey: ["library-lookup", md5s.join(",")],
|
||||
queryFn: () =>
|
||||
api<{ found: string[] }>("/api/files/lookup/", {
|
||||
method: "POST",
|
||||
json: { md5s },
|
||||
}),
|
||||
enabled: md5s.length > 0,
|
||||
staleTime: 60_000,
|
||||
});
|
||||
const inLibrary = useMemo(
|
||||
() => new Set(lookupQuery.data?.found ?? []),
|
||||
[lookupQuery.data],
|
||||
);
|
||||
|
||||
const { shownPosts, hiddenCount } = useMemo(() => {
|
||||
if (blacklistSet.size === 0 || showBlacklisted) {
|
||||
return { shownPosts: orderedPosts, hiddenCount: 0 };
|
||||
}
|
||||
const shown: E621Post[] = [];
|
||||
let hidden = 0;
|
||||
for (const post of orderedPosts) {
|
||||
if (postIsBlacklisted(post, blacklistSet).length > 0) {
|
||||
hidden += 1;
|
||||
} else {
|
||||
shown.push(post);
|
||||
}
|
||||
}
|
||||
return { shownPosts: shown, hiddenCount: hidden };
|
||||
}, [orderedPosts, blacklistSet, showBlacklisted]);
|
||||
|
||||
if (!Number.isFinite(id) || id <= 0) {
|
||||
return (
|
||||
<EmptyState
|
||||
title="Pool not found"
|
||||
description={
|
||||
<Link to="/pools" className="text-ctp-blue hover:underline">
|
||||
Back to pools
|
||||
</Link>
|
||||
}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
if (poolQuery.isPending) {
|
||||
return (
|
||||
<div className="flex justify-center py-24">
|
||||
<Spinner className="h-6 w-6" />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (poolQuery.isError || !pool) {
|
||||
return (
|
||||
<EmptyState
|
||||
title="Could not load this pool"
|
||||
description={
|
||||
<>
|
||||
{e621ErrorMessage(poolQuery.error)}{" "}
|
||||
<Link to="/pools" className="text-ctp-blue hover:underline">
|
||||
Back to pools
|
||||
</Link>
|
||||
</>
|
||||
}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
const remaining = pool.post_ids.length - postIds.length;
|
||||
|
||||
return (
|
||||
<div className="mx-auto flex w-full max-w-6xl flex-col gap-6">
|
||||
<header className="flex flex-wrap items-start justify-between gap-4">
|
||||
<div className="min-w-0">
|
||||
<Link
|
||||
to="/pools"
|
||||
className="inline-flex items-center gap-1 text-sm text-ctp-subtext0 transition hover:text-ctp-text"
|
||||
>
|
||||
<ArrowLeft className="h-4 w-4" /> Pools
|
||||
</Link>
|
||||
<h1 className="mt-2 break-words text-lg font-semibold">
|
||||
{pool.name}
|
||||
</h1>
|
||||
<p className="mt-1 flex flex-wrap items-center gap-2 text-xs text-ctp-overlay0">
|
||||
<span
|
||||
className={cn(
|
||||
"rounded-full border px-1.5 py-0.5 font-mono text-[10px] font-medium capitalize",
|
||||
categoryBadge[pool.category] ??
|
||||
"border-ctp-surface1 text-ctp-subtext0",
|
||||
)}
|
||||
>
|
||||
{pool.category}
|
||||
</span>
|
||||
<span className="font-mono">#{pool.id}</span>
|
||||
<span>{pool.post_count} posts</span>
|
||||
<span>by {pool.creator_name || `#${pool.creator_id}`}</span>
|
||||
<span>· updated {formatDate(pool.updated_at)}</span>
|
||||
{!pool.is_active ? (
|
||||
<span className="flex items-center gap-1 text-ctp-red">
|
||||
<Archive className="h-3 w-3" /> deleted
|
||||
</span>
|
||||
) : null}
|
||||
</p>
|
||||
</div>
|
||||
<PoolFollowButton poolId={pool.id} name={pool.name} />
|
||||
</header>
|
||||
|
||||
{pool.description.trim() ? (
|
||||
<section className="rounded-lg border border-ctp-surface0 bg-ctp-base p-4">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Description
|
||||
</h2>
|
||||
<div className="mt-2 text-sm leading-relaxed text-ctp-subtext0">
|
||||
{renderDText(pool.description)}
|
||||
</div>
|
||||
</section>
|
||||
) : null}
|
||||
|
||||
<section className="flex flex-col gap-3">
|
||||
<div className="flex flex-wrap items-center justify-between gap-2">
|
||||
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||
Posts{" "}
|
||||
<span className="font-mono text-[11px] text-ctp-overlay0">
|
||||
{shownPosts.length}
|
||||
{hiddenCount > 0 ? ` of ${orderedPosts.length}` : ""}
|
||||
</span>
|
||||
</h2>
|
||||
{hiddenCount > 0 || showBlacklisted ? (
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setShowBlacklisted((value) => !value)}
|
||||
className="inline-flex items-center gap-1.5 text-xs text-ctp-overlay0 transition hover:text-ctp-text"
|
||||
>
|
||||
{showBlacklisted ? (
|
||||
<>
|
||||
<EyeOff className="h-3.5 w-3.5" /> Hide blacklisted
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<Eye className="h-3.5 w-3.5" /> Reveal {hiddenCount}{" "}
|
||||
blacklisted
|
||||
</>
|
||||
)}
|
||||
</button>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
{postsQuery.isPending ? (
|
||||
<div className="flex justify-center py-24">
|
||||
<Spinner className="h-6 w-6" />
|
||||
</div>
|
||||
) : postsQuery.isError ? (
|
||||
<EmptyState
|
||||
title="Could not load the pool's posts"
|
||||
description={e621ErrorMessage(postsQuery.error)}
|
||||
/>
|
||||
) : shownPosts.length === 0 ? (
|
||||
<EmptyState
|
||||
title="No posts to show"
|
||||
description={
|
||||
hiddenCount > 0
|
||||
? "Everything here is blacklisted."
|
||||
: "This pool has no available posts."
|
||||
}
|
||||
/>
|
||||
) : (
|
||||
<div className="grid grid-cols-[repeat(auto-fill,minmax(190px,1fr))] gap-3">
|
||||
{shownPosts.map((post) => {
|
||||
const blacklisted =
|
||||
blacklistSet.size > 0 &&
|
||||
postIsBlacklisted(post, blacklistSet).length > 0;
|
||||
if (!blacklisted) {
|
||||
return (
|
||||
<PostCard
|
||||
key={post.id}
|
||||
post={post}
|
||||
inLibrary={inLibrary.has(post.file.md5)}
|
||||
from={`${location.pathname}${location.search}`}
|
||||
/>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<div
|
||||
key={post.id}
|
||||
className="min-w-0 rounded-lg ring-1 ring-ctp-red/50"
|
||||
title="Matches your blacklist"
|
||||
>
|
||||
<PostCard
|
||||
post={post}
|
||||
inLibrary={inLibrary.has(post.file.md5)}
|
||||
from={`${location.pathname}${location.search}`}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{remaining > 0 ? (
|
||||
<div className="mt-2 flex justify-center">
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={postsQuery.isFetching}
|
||||
onClick={() => setVisibleCount((count) => count + CHUNK)}
|
||||
>
|
||||
{postsQuery.isFetching
|
||||
? "Loading…"
|
||||
: `Load more posts (${remaining} remaining)`}
|
||||
</Button>
|
||||
</div>
|
||||
) : null}
|
||||
</section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,352 @@
|
||||
import { keepPreviousData, useQuery } from "@tanstack/react-query";
|
||||
import { Archive, Layers, Search } from "lucide-react";
|
||||
import { useMemo, useState } from "react";
|
||||
import { Link, useSearchParams } from "react-router-dom";
|
||||
|
||||
import { CollapsibleSidebar } from "@/components/CollapsibleSidebar";
|
||||
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
|
||||
import { cn } from "@/lib/cn";
|
||||
import {
|
||||
e621ErrorMessage,
|
||||
effectiveCredentials,
|
||||
fetchPools,
|
||||
fetchPostsByIds,
|
||||
postIsBlacklisted,
|
||||
type E621Pool,
|
||||
type E621PoolOrder,
|
||||
type E621Post,
|
||||
} from "@/lib/e621";
|
||||
import { formatDate } from "@/lib/format";
|
||||
import { useE621 } from "@/store/e621";
|
||||
|
||||
const PAGE_SIZE = 48;
|
||||
const EMPTY_POOLS: E621Pool[] = [];
|
||||
|
||||
const categoryBadge: Record<string, string> = {
|
||||
series: "border-ctp-mauve/40 bg-ctp-mauve/15 text-ctp-mauve",
|
||||
collection: "border-ctp-blue/40 bg-ctp-blue/15 text-ctp-blue",
|
||||
};
|
||||
|
||||
const orderOptions: { value: E621PoolOrder; label: string }[] = [
|
||||
{ value: "id_desc", label: "Newest first" },
|
||||
{ value: "id_asc", label: "Oldest first" },
|
||||
{ value: "name", label: "Name A–Z" },
|
||||
{ value: "post_count", label: "Most posts" },
|
||||
{ value: "created_at", label: "Recently active" },
|
||||
];
|
||||
|
||||
export default function PoolsPage() {
|
||||
const [searchParams, setSearchParams] = useSearchParams();
|
||||
const credentials = useE621((state) => state.credentials);
|
||||
const blacklist = useE621((state) => state.blacklist);
|
||||
const blacklistSet = useMemo(() => new Set(blacklist), [blacklist]);
|
||||
|
||||
const q = searchParams.get("q") ?? "";
|
||||
const category = searchParams.get("category") ?? "";
|
||||
const status = searchParams.get("status") ?? "";
|
||||
const order = (searchParams.get("order") ?? "id_desc") as E621PoolOrder;
|
||||
const page = Math.max(1, Number(searchParams.get("page") ?? "1") || 1);
|
||||
const [draft, setDraft] = useState<string | null>(null);
|
||||
const nameInput = draft ?? q;
|
||||
|
||||
const poolsQuery = useQuery({
|
||||
queryKey: [
|
||||
"e621",
|
||||
"pools",
|
||||
q,
|
||||
category,
|
||||
status,
|
||||
order,
|
||||
page,
|
||||
credentials?.username ?? "",
|
||||
],
|
||||
queryFn: ({ signal }) =>
|
||||
fetchPools(effectiveCredentials(credentials), {
|
||||
name: q,
|
||||
category,
|
||||
isActive: status,
|
||||
order,
|
||||
page,
|
||||
limit: PAGE_SIZE,
|
||||
signal,
|
||||
}),
|
||||
placeholderData: keepPreviousData,
|
||||
staleTime: 5 * 60_000,
|
||||
});
|
||||
|
||||
const pools = poolsQuery.data ?? EMPTY_POOLS;
|
||||
|
||||
// Covers come from each pool's first post, fetched in one batch.
|
||||
const coverIds = useMemo(
|
||||
() =>
|
||||
[
|
||||
...new Set(
|
||||
pools
|
||||
.map((pool) => pool.post_ids[0])
|
||||
.filter((id): id is number => typeof id === "number"),
|
||||
),
|
||||
],
|
||||
[pools],
|
||||
);
|
||||
const coversQuery = useQuery({
|
||||
queryKey: ["e621", "pool-covers", coverIds.join(","), credentials?.username ?? ""],
|
||||
queryFn: ({ signal }) =>
|
||||
fetchPostsByIds(effectiveCredentials(credentials), coverIds, signal),
|
||||
enabled: coverIds.length > 0,
|
||||
staleTime: 30 * 60_000,
|
||||
});
|
||||
const covers = useMemo(() => {
|
||||
const map = new Map<number, E621Post>();
|
||||
for (const post of coversQuery.data ?? []) map.set(post.id, post);
|
||||
return map;
|
||||
}, [coversQuery.data]);
|
||||
|
||||
function buildParams(values: {
|
||||
q: string;
|
||||
category: string;
|
||||
status: string;
|
||||
order: string;
|
||||
page: number;
|
||||
}): URLSearchParams {
|
||||
const params = new URLSearchParams();
|
||||
if (values.q.trim()) params.set("q", values.q.trim());
|
||||
if (values.category) params.set("category", values.category);
|
||||
if (values.status) params.set("status", values.status);
|
||||
if (values.order && values.order !== "id_desc")
|
||||
params.set("order", values.order);
|
||||
if (values.page > 1) params.set("page", String(values.page));
|
||||
return params;
|
||||
}
|
||||
|
||||
function update(patch: Partial<{ q: string; category: string; status: string; order: string }>) {
|
||||
setDraft(null);
|
||||
setSearchParams(
|
||||
buildParams({
|
||||
q: patch.q ?? q,
|
||||
category: patch.category ?? category,
|
||||
status: patch.status ?? status,
|
||||
order: patch.order ?? order,
|
||||
page: 1,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
const hasNext = pools.length >= PAGE_SIZE;
|
||||
|
||||
return (
|
||||
<div className="flex flex-col gap-6 lg:flex-row lg:items-start">
|
||||
<CollapsibleSidebar>
|
||||
<form
|
||||
onSubmit={(event) => {
|
||||
event.preventDefault();
|
||||
update({ q: nameInput });
|
||||
}}
|
||||
className="flex flex-col gap-2"
|
||||
>
|
||||
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||
Pool name
|
||||
</span>
|
||||
<div className="relative">
|
||||
<Search className="pointer-events-none absolute left-2.5 top-1/2 h-4 w-4 -translate-y-1/2 text-ctp-overlay0" />
|
||||
<input
|
||||
className={cn(inputClass, "pl-8 font-mono")}
|
||||
placeholder="Name contains…"
|
||||
value={nameInput}
|
||||
onChange={(event) => setDraft(event.target.value)}
|
||||
/>
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
<Button type="submit" className="flex-1">
|
||||
Search
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
variant="secondary"
|
||||
onClick={() => update({ q: "" })}
|
||||
>
|
||||
Clear
|
||||
</Button>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
<label className="flex flex-col gap-2">
|
||||
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||
Category
|
||||
</span>
|
||||
<select
|
||||
className={inputClass}
|
||||
value={category}
|
||||
onChange={(event) => update({ category: event.target.value })}
|
||||
>
|
||||
<option value="">Any category</option>
|
||||
<option value="series">Series</option>
|
||||
<option value="collection">Collection</option>
|
||||
</select>
|
||||
</label>
|
||||
|
||||
<label className="flex flex-col gap-2">
|
||||
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||
Status
|
||||
</span>
|
||||
<select
|
||||
className={inputClass}
|
||||
value={status}
|
||||
onChange={(event) => update({ status: event.target.value })}
|
||||
>
|
||||
<option value="">Any status</option>
|
||||
<option value="true">Active</option>
|
||||
<option value="false">Deleted</option>
|
||||
</select>
|
||||
</label>
|
||||
|
||||
<label className="flex flex-col gap-2">
|
||||
<span className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||
Sort
|
||||
</span>
|
||||
<select
|
||||
className={inputClass}
|
||||
value={order}
|
||||
onChange={(event) => update({ order: event.target.value })}
|
||||
>
|
||||
{orderOptions.map((option) => (
|
||||
<option key={option.value} value={option.value}>
|
||||
{option.label}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
</CollapsibleSidebar>
|
||||
|
||||
<section className="min-w-0 flex-1">
|
||||
<div className="mb-4 flex items-center justify-between gap-3">
|
||||
<p className="text-sm text-ctp-overlay0">
|
||||
{poolsQuery.data
|
||||
? `${pools.length} pool${pools.length === 1 ? "" : "s"}`
|
||||
: "…"}
|
||||
</p>
|
||||
<p className="font-mono text-xs text-ctp-overlay0">Page {page}</p>
|
||||
</div>
|
||||
|
||||
{poolsQuery.isPending ? (
|
||||
<div className="flex justify-center py-24">
|
||||
<Spinner className="h-6 w-6" />
|
||||
</div>
|
||||
) : poolsQuery.isError ? (
|
||||
<EmptyState
|
||||
title="Could not load pools from e621"
|
||||
description={e621ErrorMessage(poolsQuery.error)}
|
||||
/>
|
||||
) : pools.length === 0 ? (
|
||||
<EmptyState
|
||||
title="No pools found"
|
||||
description="Try a different name, category or status."
|
||||
/>
|
||||
) : (
|
||||
<div
|
||||
className={cn(
|
||||
"grid grid-cols-[repeat(auto-fill,minmax(220px,1fr))] gap-3 transition-opacity",
|
||||
poolsQuery.isPlaceholderData && "opacity-50",
|
||||
)}
|
||||
>
|
||||
{pools.map((pool) => {
|
||||
const cover = covers.get(pool.post_ids[0] ?? -1);
|
||||
const coverBlacklisted = Boolean(
|
||||
cover &&
|
||||
blacklistSet.size > 0 &&
|
||||
postIsBlacklisted(cover, blacklistSet).length > 0,
|
||||
);
|
||||
return (
|
||||
<Link
|
||||
key={pool.id}
|
||||
to={`/pools/${pool.id}`}
|
||||
className="group overflow-hidden rounded-lg border border-ctp-surface0 bg-ctp-base transition hover:border-ctp-surface2"
|
||||
>
|
||||
<div className="relative aspect-[4/3] overflow-hidden bg-ctp-mantle">
|
||||
{cover && cover.preview.url && !coverBlacklisted ? (
|
||||
<img
|
||||
src={cover.preview.url}
|
||||
alt={pool.name}
|
||||
loading="lazy"
|
||||
className="h-full w-full object-cover transition duration-200 group-hover:scale-[1.03]"
|
||||
/>
|
||||
) : (
|
||||
<div className="flex h-full w-full flex-col items-center justify-center gap-1 text-ctp-overlay0">
|
||||
<Layers className="h-5 w-5" />
|
||||
<span className="text-[10px]">
|
||||
{coverBlacklisted ? "Blacklisted cover" : "No cover"}
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<span
|
||||
className={cn(
|
||||
"absolute left-2 top-2 rounded-full border px-1.5 py-0.5 font-mono text-[10px] font-medium capitalize",
|
||||
categoryBadge[pool.category] ??
|
||||
"border-ctp-surface1 bg-ctp-base/80 text-ctp-subtext0",
|
||||
)}
|
||||
>
|
||||
{pool.category}
|
||||
</span>
|
||||
|
||||
<span className="absolute right-2 top-2 rounded-sm bg-ctp-crust/80 px-1.5 py-0.5 font-mono text-[10px] uppercase text-ctp-overlay1">
|
||||
{pool.post_count} posts
|
||||
</span>
|
||||
|
||||
{!pool.is_active ? (
|
||||
<span className="absolute bottom-2 left-2 flex items-center gap-1 rounded-full bg-ctp-red/90 px-1.5 py-0.5 font-mono text-[10px] font-medium text-ctp-crust">
|
||||
<Archive className="h-3 w-3" /> deleted
|
||||
</span>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
<div className="px-2.5 py-2">
|
||||
<p
|
||||
className="truncate font-mono text-xs text-ctp-subtext1"
|
||||
title={pool.name}
|
||||
>
|
||||
{pool.name}
|
||||
</p>
|
||||
<p className="mt-0.5 truncate text-[11px] text-ctp-overlay0">
|
||||
#{pool.id} · {pool.creator_name || "unknown"} · updated{" "}
|
||||
{formatDate(pool.updated_at)}
|
||||
</p>
|
||||
</div>
|
||||
</Link>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{page > 1 || hasNext ? (
|
||||
<div className="mt-6 flex items-center justify-center gap-3">
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={page <= 1 || poolsQuery.isFetching}
|
||||
onClick={() =>
|
||||
setSearchParams(
|
||||
buildParams({ q, category, status, order, page: page - 1 }),
|
||||
)
|
||||
}
|
||||
>
|
||||
Previous
|
||||
</Button>
|
||||
<span className="font-mono text-xs text-ctp-subtext0">
|
||||
Page {page}
|
||||
</span>
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={!hasNext || poolsQuery.isFetching}
|
||||
onClick={() =>
|
||||
setSearchParams(
|
||||
buildParams({ q, category, status, order, page: page + 1 }),
|
||||
)
|
||||
}
|
||||
>
|
||||
Next
|
||||
</Button>
|
||||
</div>
|
||||
) : null}
|
||||
</section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,180 @@
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { useState } from "react";
|
||||
|
||||
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
|
||||
import { api, errorMessage } from "@/lib/api";
|
||||
import { cn } from "@/lib/cn";
|
||||
import { formatDate } from "@/lib/format";
|
||||
import type { AdminUser, Paginated } from "@/lib/types";
|
||||
import { useAuth } from "@/store/auth";
|
||||
|
||||
const ROLE_OPTIONS = [
|
||||
{ value: "user", label: "User" },
|
||||
{ value: "uploader", label: "Uploader" },
|
||||
{ value: "staff", label: "Staff" },
|
||||
];
|
||||
|
||||
function UserRow({ profile }: { profile: AdminUser }) {
|
||||
const queryClient = useQueryClient();
|
||||
const [avatarJId, setAvatarJId] = useState(profile.avatar_j_id ?? "");
|
||||
const [message, setMessage] = useState<string | null>(null);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
const mutation = useMutation({
|
||||
mutationFn: (payload: Record<string, string | null>) =>
|
||||
api<AdminUser>(`/api/users/${profile.id}/`, {
|
||||
method: "PATCH",
|
||||
json: payload,
|
||||
}),
|
||||
onSuccess: (updated) => {
|
||||
setMessage("Saved.");
|
||||
setError(null);
|
||||
setAvatarJId(updated.avatar_j_id ?? "");
|
||||
void queryClient.invalidateQueries({ queryKey: ["users"] });
|
||||
},
|
||||
onError: (err) => {
|
||||
setError(errorMessage(err));
|
||||
setMessage(null);
|
||||
},
|
||||
});
|
||||
|
||||
return (
|
||||
<div className="flex flex-col gap-3 rounded-lg border border-ctp-surface0 bg-ctp-base p-3 lg:flex-row lg:items-center">
|
||||
<div className="flex min-w-0 flex-1 items-center gap-3">
|
||||
{profile.avatar_url ? (
|
||||
<img
|
||||
src={profile.avatar_url}
|
||||
alt={profile.username}
|
||||
className="h-10 w-10 shrink-0 rounded-full border border-ctp-surface1 object-cover"
|
||||
/>
|
||||
) : (
|
||||
<span className="flex h-10 w-10 shrink-0 items-center justify-center rounded-full bg-ctp-surface1 font-mono text-sm font-semibold text-ctp-lavender">
|
||||
{profile.username.slice(0, 1).toUpperCase()}
|
||||
</span>
|
||||
)}
|
||||
<div className="min-w-0">
|
||||
<p className="flex flex-wrap items-center gap-2 text-sm text-ctp-subtext1">
|
||||
<span className="font-medium">{profile.username}</span>
|
||||
{profile.is_superuser ? (
|
||||
<span className="rounded-full border border-ctp-red/40 bg-ctp-red/10 px-2 py-0.5 font-mono text-[10px] text-ctp-red">
|
||||
admin
|
||||
</span>
|
||||
) : null}
|
||||
{profile.is_staff && !profile.is_superuser ? (
|
||||
<span className="rounded-full border border-ctp-surface1 px-2 py-0.5 font-mono text-[10px] text-ctp-overlay0">
|
||||
django staff
|
||||
</span>
|
||||
) : null}
|
||||
</p>
|
||||
<p className="truncate font-mono text-[11px] text-ctp-overlay0">
|
||||
{profile.email || "no email"} · joined{" "}
|
||||
{formatDate(profile.date_joined)} · {profile.uploads_count} upload
|
||||
{profile.uploads_count === 1 ? "" : "s"}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap items-center gap-2">
|
||||
<select
|
||||
className={cn(inputClass, "w-32")}
|
||||
value={profile.role}
|
||||
onChange={(event) =>
|
||||
mutation.mutate({ role: event.target.value })
|
||||
}
|
||||
disabled={mutation.isPending}
|
||||
title="Role"
|
||||
>
|
||||
{ROLE_OPTIONS.map((option) => (
|
||||
<option key={option.value} value={option.value}>
|
||||
{option.label}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
<input
|
||||
className={cn(inputClass, "w-28 font-mono")}
|
||||
placeholder="J-ID"
|
||||
value={avatarJId}
|
||||
onChange={(event) => setAvatarJId(event.target.value)}
|
||||
title="Avatar library item"
|
||||
/>
|
||||
<Button
|
||||
variant="secondary"
|
||||
disabled={mutation.isPending}
|
||||
onClick={() => mutation.mutate({ avatar_j_id: avatarJId.trim() })}
|
||||
>
|
||||
Save avatar
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{message ? (
|
||||
<p className="text-xs text-ctp-green lg:w-20">{message}</p>
|
||||
) : null}
|
||||
{error ? (
|
||||
<p className="text-xs text-ctp-red lg:w-40">{error}</p>
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default function UsersPage() {
|
||||
const user = useAuth((state) => state.user);
|
||||
const [search, setSearch] = useState("");
|
||||
|
||||
const isStaff = Boolean(
|
||||
user &&
|
||||
(user.is_staff || user.is_superuser || user.role === "staff"),
|
||||
);
|
||||
|
||||
const query = useQuery({
|
||||
queryKey: ["users", search],
|
||||
queryFn: () =>
|
||||
api<Paginated<AdminUser>>(
|
||||
`/api/users/${search ? `?search=${encodeURIComponent(search)}` : ""}`,
|
||||
),
|
||||
enabled: isStaff,
|
||||
});
|
||||
|
||||
if (!isStaff) {
|
||||
return (
|
||||
<EmptyState
|
||||
title="Staff only"
|
||||
description="This page is limited to staff accounts."
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="mx-auto flex w-full max-w-5xl flex-col gap-4">
|
||||
<header className="flex flex-wrap items-end justify-between gap-3">
|
||||
<div>
|
||||
<h1 className="text-lg font-semibold">Users</h1>
|
||||
<p className="mt-1 text-sm text-ctp-overlay0">
|
||||
Manage roles and set profile pictures from library J-IDs.
|
||||
</p>
|
||||
</div>
|
||||
<input
|
||||
className={cn(inputClass, "w-56")}
|
||||
placeholder="Search username…"
|
||||
value={search}
|
||||
onChange={(event) => setSearch(event.target.value)}
|
||||
/>
|
||||
</header>
|
||||
|
||||
{query.isPending ? (
|
||||
<div className="flex justify-center py-16">
|
||||
<Spinner className="h-6 w-6" />
|
||||
</div>
|
||||
) : query.isError || !query.data ? (
|
||||
<EmptyState title="Could not load users" />
|
||||
) : query.data.results.length === 0 ? (
|
||||
<EmptyState title="No users found" />
|
||||
) : (
|
||||
<div className="flex flex-col gap-3">
|
||||
{query.data.results.map((profile) => (
|
||||
<UserRow key={profile.id} profile={profile} />
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
const AGE_KEY = "j621.age-verified";
|
||||
|
||||
export function isAgeVerified(): boolean {
|
||||
try {
|
||||
return window.localStorage.getItem(AGE_KEY) === "1";
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export function markAgeVerified(): void {
|
||||
try {
|
||||
window.localStorage.setItem(AGE_KEY, "1");
|
||||
} catch {
|
||||
// Private mode with storage disabled: the gate simply shows again.
|
||||
}
|
||||
}
|
||||
+10
-3
@@ -1,4 +1,4 @@
|
||||
import type { MediaItem } from "@/lib/types";
|
||||
import type { TempUpload } from "@/lib/types";
|
||||
|
||||
const TOKEN_KEY = "j621.token";
|
||||
|
||||
@@ -6,6 +6,13 @@ export function getToken(): string | null {
|
||||
return window.localStorage.getItem(TOKEN_KEY);
|
||||
}
|
||||
|
||||
export function authHeaders(extra?: HeadersInit): Headers {
|
||||
const headers = new Headers(extra);
|
||||
const token = getToken();
|
||||
if (token) headers.set("Authorization", `Token ${token}`);
|
||||
return headers;
|
||||
}
|
||||
|
||||
export function setToken(token: string | null): void {
|
||||
if (token) {
|
||||
window.localStorage.setItem(TOKEN_KEY, token);
|
||||
@@ -82,7 +89,7 @@ export function uploadFile(
|
||||
file: File,
|
||||
fields: Record<string, string> = {},
|
||||
onProgress?: (percent: number) => void,
|
||||
): Promise<MediaItem> {
|
||||
): Promise<TempUpload> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const form = new FormData();
|
||||
form.append("file", file);
|
||||
@@ -107,7 +114,7 @@ export function uploadFile(
|
||||
data = null;
|
||||
}
|
||||
if (request.status >= 200 && request.status < 300) {
|
||||
resolve(data as MediaItem);
|
||||
resolve(data as TempUpload);
|
||||
} else {
|
||||
reject(new ApiError(request.status, data));
|
||||
}
|
||||
|
||||
@@ -296,6 +296,209 @@ export async function setFavorite(
|
||||
});
|
||||
}
|
||||
|
||||
export async function fetchPostsByMd5(
|
||||
credentials: E621Credentials,
|
||||
md5s: string[],
|
||||
signal?: AbortSignal,
|
||||
): Promise<E621Post[]> {
|
||||
if (md5s.length === 0) return [];
|
||||
const data = await e621Request<{ posts: E621Post[] }>(
|
||||
credentials,
|
||||
"/posts.json",
|
||||
{
|
||||
params: {
|
||||
tags: `md5:${md5s.join(",")}`,
|
||||
limit: Math.min(md5s.length, 100),
|
||||
},
|
||||
signal,
|
||||
},
|
||||
);
|
||||
return data.posts;
|
||||
}
|
||||
|
||||
export interface E621Pool {
|
||||
id: number;
|
||||
name: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
creator_id: number;
|
||||
creator_name: string;
|
||||
description: string;
|
||||
is_active: boolean;
|
||||
category: "collection" | "series";
|
||||
post_ids: number[];
|
||||
post_count: number;
|
||||
}
|
||||
|
||||
export type E621PoolOrder =
|
||||
| "id_desc"
|
||||
| "id_asc"
|
||||
| "name"
|
||||
| "created_at"
|
||||
| "post_count";
|
||||
|
||||
export async function fetchPools(
|
||||
credentials: E621Credentials,
|
||||
options: {
|
||||
name?: string;
|
||||
category?: string;
|
||||
isActive?: string;
|
||||
order?: E621PoolOrder;
|
||||
page?: number;
|
||||
limit?: number;
|
||||
signal?: AbortSignal;
|
||||
} = {},
|
||||
): Promise<E621Pool[]> {
|
||||
const data = await e621Request<unknown>(credentials, "/pools.json", {
|
||||
params: {
|
||||
limit: options.limit ?? 48,
|
||||
page: options.page,
|
||||
"search[order]": options.order ?? "id_desc",
|
||||
"search[name_matches]": options.name || undefined,
|
||||
"search[category]": options.category || undefined,
|
||||
"search[is_active]": options.isActive || undefined,
|
||||
},
|
||||
signal: options.signal,
|
||||
});
|
||||
// The spec says a bare array; tolerate a wrapped payload too.
|
||||
if (Array.isArray(data)) return data as E621Pool[];
|
||||
const wrapped = (data as { pools?: E621Pool[] })?.pools;
|
||||
return Array.isArray(wrapped) ? wrapped : [];
|
||||
}
|
||||
|
||||
export async function fetchPool(
|
||||
credentials: E621Credentials,
|
||||
poolId: number,
|
||||
signal?: AbortSignal,
|
||||
): Promise<E621Pool> {
|
||||
const data = await e621Request<unknown>(credentials, `/pools/${poolId}.json`, {
|
||||
signal,
|
||||
});
|
||||
const wrapped = (data as { pool?: E621Pool })?.pool;
|
||||
return (wrapped ?? (data as E621Pool)) as E621Pool;
|
||||
}
|
||||
|
||||
export interface E621IqdbResult {
|
||||
post_id: number | null;
|
||||
score: number | null;
|
||||
preview_url: string | null;
|
||||
rating: string | null;
|
||||
md5: string | null;
|
||||
score_total: number | null;
|
||||
fav_count: number | null;
|
||||
width: number | null;
|
||||
height: number | null;
|
||||
tags_preview: string[];
|
||||
}
|
||||
|
||||
function flattenTagPreview(tags: unknown, limit = 8): string[] {
|
||||
if (!tags || typeof tags !== "object") return [];
|
||||
const out: string[] = [];
|
||||
for (const value of Object.values(tags as Record<string, unknown>)) {
|
||||
if (!Array.isArray(value)) continue;
|
||||
for (const tag of value) {
|
||||
if (typeof tag === "string" && !out.includes(tag)) {
|
||||
out.push(tag);
|
||||
if (out.length >= limit) return out;
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function normalizeIqdbEntry(entry: unknown): E621IqdbResult | null {
|
||||
if (!entry || typeof entry !== "object") return null;
|
||||
const record = entry as Record<string, unknown>;
|
||||
const postWrap = record.post;
|
||||
let post: Record<string, unknown> = {};
|
||||
if (postWrap && typeof postWrap === "object") {
|
||||
const wrap = postWrap as Record<string, unknown>;
|
||||
post =
|
||||
wrap.posts && typeof wrap.posts === "object"
|
||||
? (wrap.posts as Record<string, unknown>)
|
||||
: wrap;
|
||||
}
|
||||
const file = (post.file ?? {}) as Record<string, unknown>;
|
||||
const preview = (post.preview ?? {}) as Record<string, unknown>;
|
||||
const postId =
|
||||
typeof record.post_id === "number"
|
||||
? record.post_id
|
||||
: typeof post.id === "number"
|
||||
? post.id
|
||||
: null;
|
||||
const scoreTotal = post.score;
|
||||
return {
|
||||
post_id: postId,
|
||||
score: typeof record.score === "number" ? record.score : null,
|
||||
preview_url: typeof preview.url === "string" ? preview.url : null,
|
||||
rating: typeof post.rating === "string" ? post.rating : null,
|
||||
md5: typeof file.md5 === "string" ? file.md5 : null,
|
||||
score_total: typeof scoreTotal === "number" ? scoreTotal : null,
|
||||
fav_count: typeof post.fav_count === "number" ? post.fav_count : null,
|
||||
width: typeof file.width === "number" ? file.width : null,
|
||||
height: typeof file.height === "number" ? file.height : null,
|
||||
tags_preview: flattenTagPreview(post.tags),
|
||||
};
|
||||
}
|
||||
|
||||
export async function iqdbSearch(
|
||||
credentials: E621Credentials,
|
||||
file: Blob,
|
||||
filename: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<E621IqdbResult[]> {
|
||||
const form = new FormData();
|
||||
form.append("file", file, filename);
|
||||
const data = await e621Request<unknown>(credentials, "/iqdb_queries.json", {
|
||||
method: "POST",
|
||||
body: form,
|
||||
signal,
|
||||
});
|
||||
const list = Array.isArray(data)
|
||||
? data
|
||||
: Array.isArray((data as { matches?: unknown[] })?.matches)
|
||||
? (data as { matches: unknown[] }).matches
|
||||
: [];
|
||||
const candidates = list
|
||||
.map(normalizeIqdbEntry)
|
||||
.filter(
|
||||
(entry): entry is E621IqdbResult =>
|
||||
entry !== null && (entry.post_id !== null || entry.preview_url !== null),
|
||||
)
|
||||
.slice(0, 10);
|
||||
|
||||
// The IQDB payload carries no preview/file data; enrich with one batched
|
||||
// post lookup so candidates can show what they actually are.
|
||||
const ids = candidates
|
||||
.map((candidate) => candidate.post_id)
|
||||
.filter((id): id is number => typeof id === "number");
|
||||
if (ids.length > 0) {
|
||||
try {
|
||||
const posts = await fetchPostsByIds(credentials, ids, signal);
|
||||
const byId = new Map(posts.map((post) => [post.id, post]));
|
||||
return candidates.map((candidate) => {
|
||||
const post =
|
||||
candidate.post_id !== null ? byId.get(candidate.post_id) : undefined;
|
||||
if (!post) return candidate;
|
||||
return {
|
||||
...candidate,
|
||||
preview_url: post.preview.url ?? candidate.preview_url,
|
||||
rating: post.rating ?? candidate.rating,
|
||||
md5: post.file.md5 ?? candidate.md5,
|
||||
score_total: post.score.total,
|
||||
fav_count: post.fav_count,
|
||||
width: post.file.width,
|
||||
height: post.file.height,
|
||||
tags_preview: flattenTagPreview(post.tags),
|
||||
};
|
||||
});
|
||||
} catch {
|
||||
// Keep the base candidates when the enrichment request fails.
|
||||
}
|
||||
}
|
||||
return candidates;
|
||||
}
|
||||
|
||||
export function postIsBlacklisted(
|
||||
post: E621Post,
|
||||
blacklist: Set<string>,
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
import { QueryClient } from "@tanstack/react-query";
|
||||
|
||||
export const queryClient = new QueryClient({
|
||||
defaultOptions: {
|
||||
queries: {
|
||||
retry: 1,
|
||||
staleTime: 30_000,
|
||||
gcTime: 30 * 60_000,
|
||||
refetchOnWindowFocus: false,
|
||||
},
|
||||
},
|
||||
});
|
||||
@@ -7,10 +7,71 @@ export interface User {
|
||||
date_joined: string;
|
||||
role: "user" | "uploader" | "staff";
|
||||
can_upload: boolean;
|
||||
avatar_j_id: string | null;
|
||||
avatar_url: string | null;
|
||||
e621_username: string;
|
||||
e621_configured: boolean;
|
||||
}
|
||||
|
||||
export interface AdminUser {
|
||||
id: number;
|
||||
username: string;
|
||||
email: string;
|
||||
role: "user" | "uploader" | "staff";
|
||||
is_staff: boolean;
|
||||
is_superuser: boolean;
|
||||
date_joined: string;
|
||||
avatar_j_id: string | null;
|
||||
avatar_url: string | null;
|
||||
uploads_count: number;
|
||||
}
|
||||
|
||||
export interface DuplicateEntry {
|
||||
j_id: string;
|
||||
md5: string;
|
||||
filename: string;
|
||||
size: number;
|
||||
rating: string;
|
||||
location_count: number;
|
||||
uploaded_by: string | null;
|
||||
e621_post_id: number | null;
|
||||
locations?: { id: number; rel_path: string }[];
|
||||
thumbnail_url?: string;
|
||||
similarity?: number;
|
||||
}
|
||||
|
||||
export interface VisualGroupsResponse {
|
||||
count: number;
|
||||
page: number;
|
||||
per_page: number;
|
||||
has_next: boolean;
|
||||
threshold: number;
|
||||
algorithms: string[];
|
||||
groups: { size: number; members: DuplicateEntry[] }[];
|
||||
}
|
||||
|
||||
export interface VisualMatchesResponse {
|
||||
target: DuplicateEntry;
|
||||
threshold: number;
|
||||
algorithms: string[];
|
||||
count: number;
|
||||
matches: DuplicateEntry[];
|
||||
}
|
||||
|
||||
export interface StorageInfo {
|
||||
watched_folder: {
|
||||
path: string;
|
||||
total: number;
|
||||
used: number;
|
||||
free: number;
|
||||
library_size: number;
|
||||
percent_used: number;
|
||||
};
|
||||
media: { path: string; size: number };
|
||||
temp: { path: string; size: number; files: number };
|
||||
library_items: number;
|
||||
}
|
||||
|
||||
export interface MediaLocation {
|
||||
id: number;
|
||||
path: string;
|
||||
@@ -20,6 +81,8 @@ export interface MediaLocation {
|
||||
|
||||
export type Rating = "" | "s" | "q" | "e";
|
||||
|
||||
export type E621MatchStatus = "unknown" | "matched" | "not_found" | "deleted";
|
||||
|
||||
export interface E621StoredPost {
|
||||
id: number | null;
|
||||
created_at?: string | null;
|
||||
@@ -65,7 +128,11 @@ export interface MediaItem {
|
||||
uploaded_by_id: number | null;
|
||||
e621_post_id: number | null;
|
||||
e621_data: E621StoredPost | null;
|
||||
e621_match_status: E621MatchStatus;
|
||||
e621_checked_at: string | null;
|
||||
locations: MediaLocation[];
|
||||
raw_url: string;
|
||||
thumbnail_url: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
@@ -106,3 +173,145 @@ export interface FileNeighbors {
|
||||
previous: FileNeighbor | null;
|
||||
next: FileNeighbor | null;
|
||||
}
|
||||
|
||||
export interface DownloadTask {
|
||||
task_id: string;
|
||||
post_id: number | null;
|
||||
filename: string;
|
||||
status: "pending" | "downloading" | "complete" | "error" | "cancelled";
|
||||
progress: number;
|
||||
downloaded: number;
|
||||
total: number;
|
||||
speed: number | null;
|
||||
error: string;
|
||||
library_j_id: string | null;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
export interface MatchTask {
|
||||
task_id: string;
|
||||
scope: "missing" | "all";
|
||||
status: "pending" | "running" | "complete" | "error" | "cancelled";
|
||||
total: number;
|
||||
processed: number;
|
||||
matched: number;
|
||||
not_found: number;
|
||||
deleted: number;
|
||||
failed: number;
|
||||
error: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
export interface FollowCoverData {
|
||||
post_id: number | null;
|
||||
rating: string | null;
|
||||
preview_url: string | null;
|
||||
score: number | null;
|
||||
fav_count: number | null;
|
||||
}
|
||||
|
||||
export interface FollowedTag {
|
||||
id: number;
|
||||
tag: string;
|
||||
unseen_count: number;
|
||||
cover_data: FollowCoverData | null;
|
||||
last_synced_at: string | null;
|
||||
created_at: string;
|
||||
}
|
||||
|
||||
export interface FollowedPool {
|
||||
id: number;
|
||||
pool_id: number;
|
||||
name: string;
|
||||
post_count: number;
|
||||
unseen_count: number;
|
||||
cover_data: FollowCoverData | null;
|
||||
last_synced_at: string | null;
|
||||
created_at: string;
|
||||
}
|
||||
|
||||
export interface FollowFeedSource {
|
||||
kind: "tag" | "pool";
|
||||
id: number;
|
||||
label: string;
|
||||
}
|
||||
|
||||
export interface FollowFeedEntry {
|
||||
post_id: number;
|
||||
seen: boolean;
|
||||
data: {
|
||||
id?: number | null;
|
||||
rating?: string | null;
|
||||
tags?: Record<string, string[]>;
|
||||
score?: { total?: number | null } | null;
|
||||
fav_count?: number | null;
|
||||
file?: {
|
||||
width?: number | null;
|
||||
height?: number | null;
|
||||
ext?: string | null;
|
||||
} | null;
|
||||
preview?: { url?: string | null } | null;
|
||||
};
|
||||
sources: FollowFeedSource[];
|
||||
}
|
||||
|
||||
export interface FollowFeed {
|
||||
count: number;
|
||||
page: number;
|
||||
per_page: number;
|
||||
has_next: boolean;
|
||||
results: FollowFeedEntry[];
|
||||
}
|
||||
|
||||
export interface FollowCloud {
|
||||
status: "ready" | "building";
|
||||
stale: boolean;
|
||||
tags: [string, number][];
|
||||
posts: number;
|
||||
source: "user" | "default" | null;
|
||||
blacklist_count: number;
|
||||
computed_at: string | null;
|
||||
}
|
||||
|
||||
export interface E621IqdbCandidate {
|
||||
post_id: number | null;
|
||||
score: number | null;
|
||||
preview_url: string | null;
|
||||
rating: string | null;
|
||||
md5: string | null;
|
||||
score_total?: number | null;
|
||||
fav_count?: number | null;
|
||||
width?: number | null;
|
||||
height?: number | null;
|
||||
tags_preview?: string[];
|
||||
}
|
||||
|
||||
export interface TempUpload {
|
||||
temp_id: string;
|
||||
original_filename: string;
|
||||
md5: string;
|
||||
size: number;
|
||||
status: "pending" | "visual_match" | "completed" | "error";
|
||||
resolution: "" | "auto_md5" | "duplicate" | "linked" | "custom";
|
||||
e621_post_id: number | null;
|
||||
e621_data: E621StoredPost | null;
|
||||
custom_rating: Rating;
|
||||
custom_tags: string[];
|
||||
custom_notes: string;
|
||||
iqdb_data: E621IqdbCandidate[] | null;
|
||||
visual_matches:
|
||||
| {
|
||||
j_id: string;
|
||||
filename: string;
|
||||
similarity: number;
|
||||
thumbnail_url?: string;
|
||||
}[]
|
||||
| null;
|
||||
library_j_id: string | null;
|
||||
file_url: string | null;
|
||||
preview_url: string | null;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
+2
-12
@@ -1,21 +1,11 @@
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { QueryClientProvider } from "@tanstack/react-query";
|
||||
import { StrictMode } from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
import { BrowserRouter } from "react-router-dom";
|
||||
|
||||
import App from "./App";
|
||||
import "./index.css";
|
||||
|
||||
const queryClient = new QueryClient({
|
||||
defaultOptions: {
|
||||
queries: {
|
||||
retry: 1,
|
||||
staleTime: 30_000,
|
||||
gcTime: 30 * 60_000,
|
||||
refetchOnWindowFocus: false,
|
||||
},
|
||||
},
|
||||
});
|
||||
import { queryClient } from "@/lib/queryClient";
|
||||
|
||||
createRoot(document.getElementById("root")!).render(
|
||||
<StrictMode>
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { create } from "zustand";
|
||||
|
||||
import { api, getToken, setToken } from "@/lib/api";
|
||||
import { queryClient } from "@/lib/queryClient";
|
||||
import type { User } from "@/lib/types";
|
||||
import { useE621 } from "@/store/e621";
|
||||
|
||||
@@ -24,6 +25,9 @@ export const useAuth = create<AuthState>((set) => ({
|
||||
});
|
||||
setToken(data.token);
|
||||
const user = await api<User>("/api/auth/me/");
|
||||
// Never show the previous account's cached data to this one.
|
||||
queryClient.clear();
|
||||
useE621.getState().clear();
|
||||
set({ user, ready: true });
|
||||
},
|
||||
|
||||
@@ -33,6 +37,8 @@ export const useAuth = create<AuthState>((set) => ({
|
||||
json: { username, password },
|
||||
});
|
||||
setToken(data.token);
|
||||
queryClient.clear();
|
||||
useE621.getState().clear();
|
||||
set({ user: data.user, ready: true });
|
||||
},
|
||||
|
||||
@@ -43,6 +49,7 @@ export const useAuth = create<AuthState>((set) => ({
|
||||
// The token may already be invalid; clearing it locally is enough.
|
||||
}
|
||||
setToken(null);
|
||||
queryClient.clear();
|
||||
useE621.getState().clear();
|
||||
set({ user: null, ready: true });
|
||||
},
|
||||
|
||||
@@ -6,10 +6,13 @@ export interface E621RequestSample {
|
||||
at: number;
|
||||
}
|
||||
|
||||
const HISTORY_LIMIT = 50;
|
||||
|
||||
interface E621MetricsState {
|
||||
requestCount: number;
|
||||
totalMs: number;
|
||||
last: E621RequestSample | null;
|
||||
history: E621RequestSample[];
|
||||
record: (path: string, ms: number) => void;
|
||||
reset: () => void;
|
||||
}
|
||||
@@ -19,13 +22,19 @@ export const useE621Metrics = create<E621MetricsState>((set) => ({
|
||||
requestCount: 0,
|
||||
totalMs: 0,
|
||||
last: null,
|
||||
history: [],
|
||||
|
||||
record: (path, ms) =>
|
||||
set((state) => ({
|
||||
requestCount: state.requestCount + 1,
|
||||
totalMs: state.totalMs + ms,
|
||||
last: { path, ms, at: Date.now() },
|
||||
})),
|
||||
set((state) => {
|
||||
const sample: E621RequestSample = { path, ms, at: Date.now() };
|
||||
return {
|
||||
requestCount: state.requestCount + 1,
|
||||
totalMs: state.totalMs + ms,
|
||||
last: sample,
|
||||
history: [sample, ...state.history].slice(0, HISTORY_LIMIT),
|
||||
};
|
||||
}),
|
||||
|
||||
reset: () => set({ requestCount: 0, totalMs: 0, last: null }),
|
||||
reset: () =>
|
||||
set({ requestCount: 0, totalMs: 0, last: null, history: [] }),
|
||||
}));
|
||||
|
||||
@@ -7,6 +7,8 @@ interface UiState {
|
||||
toggleSidebar: () => void;
|
||||
ratings: string[];
|
||||
setRatings: (ratings: string[]) => void;
|
||||
statuses: string[];
|
||||
setStatuses: (statuses: string[]) => void;
|
||||
perPage: number;
|
||||
setPerPage: (perPage: number) => void;
|
||||
ordering: string;
|
||||
@@ -23,6 +25,9 @@ export const useUi = create<UiState>()(
|
||||
ratings: [],
|
||||
setRatings: (ratings) => set({ ratings }),
|
||||
|
||||
statuses: [],
|
||||
setStatuses: (statuses) => set({ statuses }),
|
||||
|
||||
perPage: 48,
|
||||
setPerPage: (perPage) => set({ perPage }),
|
||||
|
||||
|
||||
Reference in New Issue
Block a user