Duplicates, delete & storage, users page with J-ID avatars
Backend: - Perceptual hashes (aHash/dHash/pHash/wHash via imagehash, no imgdd) stored on items, computed on upload/download and by the new compute_visual_hashes command - Duplicates API: exact duplicates (multi-location items), visual matches for one item, union-find similarity groups with pagination - Delete API with ownership/staff checks, per-item and per-copy deletion, watched-folder path validation; storage overview and temp cleanup; file list accepts j_ids batches - Staged uploads are flagged visual_match with their library matches (threshold via VISUAL_MATCH_THRESHOLD) - Staff users API: list with upload counts, set role and avatar by J-ID; User.avatar FK with signed avatar URLs - Download threads close their DB connection and stale tasks are reaped, keeping behaviour Gunicorn-friendly Frontend: - /duplicates: exact duplicate groups with per-copy delete, visual similarity controls, search similar to a J-ID, paginated groups with selection, bulk delete and dismiss - /delete: storage cards, delete by J-ID with preview grid, temp cleanup - /users: staff directory with role selects and avatar J-ID inputs - Nav + command palette entries; top-bar avatar; upload cards and the metadata modal show library visual matches
This commit is contained in:
+14
-12
@@ -5,15 +5,16 @@ they land.
|
|||||||
|
|
||||||
## 1. Library management
|
## 1. Library management
|
||||||
|
|
||||||
- [ ] **Duplicates engine**
|
- [x] **Duplicates engine**
|
||||||
- [ ] Exact MD5 duplicate detection with grouped results
|
- [x] Exact MD5 duplicate detection with grouped results
|
||||||
- [ ] Perceptual similarity (aHash, dHash, pHash, wHash) with threshold slider
|
- [x] Perceptual similarity (aHash, dHash, pHash, wHash) with threshold slider
|
||||||
and algorithm toggles (needs `imagehash`/`imgdd` server-side)
|
and algorithm toggles (`imagehash` server-side)
|
||||||
- [ ] Visual similarity groups: pagination, "keep one / delete rest", dismiss group
|
- [x] Visual similarity groups: pagination, selection, delete and dismiss
|
||||||
- [ ] **Delete & storage page**
|
- [x] **Delete & storage page**
|
||||||
- [ ] Storage overview (watched folder, media folder)
|
- [x] Storage overview (watched folder, media folder, temp)
|
||||||
- [ ] Delete by J-ID with preview grid and bulk selection
|
- [x] Delete by J-ID with preview grid and bulk selection (plus per-copy
|
||||||
- [ ] Temp folder cleanup
|
deletion of duplicate locations)
|
||||||
|
- [x] Temp folder cleanup
|
||||||
- [ ] **Library search upgrades**
|
- [ ] **Library search upgrades**
|
||||||
- [ ] Search by tags (custom + e621 tags), not just filename
|
- [ ] Search by tags (custom + e621 tags), not just filename
|
||||||
- [ ] Tag cloud from the library
|
- [ ] Tag cloud from the library
|
||||||
@@ -58,8 +59,8 @@ Files now stage first and are resolved before entering the library.
|
|||||||
- [x] **IQDB similarity on upload** (SPA-driven)
|
- [x] **IQDB similarity on upload** (SPA-driven)
|
||||||
- [x] Automatic + manual IQDB checks with candidate posts
|
- [x] Automatic + manual IQDB checks with candidate posts
|
||||||
- [x] "Visual Similarity Detected" state with candidate picker
|
- [x] "Visual Similarity Detected" state with candidate picker
|
||||||
- [ ] Perceptual-hash comparison against the library (shared with the
|
- [x] Perceptual-hash comparison against the library (staged uploads are
|
||||||
duplicates engine, still pending)
|
flagged with their library matches as soon as they land)
|
||||||
- [x] **Upload UI**
|
- [x] **Upload UI**
|
||||||
- [x] Three-column board: Pending & Unmatched / Visual Similarity Detected /
|
- [x] Three-column board: Pending & Unmatched / Visual Similarity Detected /
|
||||||
Auto-uploaded & Indexed
|
Auto-uploaded & Indexed
|
||||||
@@ -80,7 +81,8 @@ Files now stage first and are resolved before entering the library.
|
|||||||
|
|
||||||
- [ ] Toasts instead of inline messages / confirm dialogs
|
- [ ] Toasts instead of inline messages / confirm dialogs
|
||||||
- [ ] Mobile drawer polish for metadata panels (design spec §layout)
|
- [ ] Mobile drawer polish for metadata panels (design spec §layout)
|
||||||
- [ ] Avatar picker (choose from library)
|
- [x] Profile pictures: staff Users page sets avatars from library J-IDs
|
||||||
|
(self-service picker in Account still pending)
|
||||||
- [ ] Profile extras (per-user browse preferences)
|
- [ ] Profile extras (per-user browse preferences)
|
||||||
- [ ] Command palette: tag finder, recent searches
|
- [ ] Command palette: tag finder, recent searches
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,8 @@
|
|||||||
|
from rest_framework.routers import DefaultRouter
|
||||||
|
|
||||||
|
from .views import UserViewSet
|
||||||
|
|
||||||
|
router = DefaultRouter()
|
||||||
|
router.register("users", UserViewSet, basename="user")
|
||||||
|
|
||||||
|
urlpatterns = router.urls
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# Generated by Django 6.1.1 on 2026-09-17 17:34
|
||||||
|
|
||||||
|
import django.db.models.deletion
|
||||||
|
from django.db import migrations, models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
|
||||||
|
dependencies = [
|
||||||
|
('accounts', '0003_user_role'),
|
||||||
|
('library', '0006_mediaitem_ahash_mediaitem_dhash_mediaitem_phash_and_more'),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='user',
|
||||||
|
name='avatar',
|
||||||
|
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to='library.mediaitem'),
|
||||||
|
),
|
||||||
|
]
|
||||||
@@ -15,6 +15,13 @@ class User(AbstractUser):
|
|||||||
]
|
]
|
||||||
|
|
||||||
role = models.CharField(max_length=20, choices=ROLE_CHOICES, default=ROLE_USER)
|
role = models.CharField(max_length=20, choices=ROLE_CHOICES, default=ROLE_USER)
|
||||||
|
avatar = models.ForeignKey(
|
||||||
|
"library.MediaItem",
|
||||||
|
null=True,
|
||||||
|
blank=True,
|
||||||
|
on_delete=models.SET_NULL,
|
||||||
|
related_name="+",
|
||||||
|
)
|
||||||
e621_username = models.CharField(max_length=100, blank=True, default="")
|
e621_username = models.CharField(max_length=100, blank=True, default="")
|
||||||
e621_api_key = models.CharField(max_length=100, blank=True, default="")
|
e621_api_key = models.CharField(max_length=100, blank=True, default="")
|
||||||
e621_base_url = models.CharField(max_length=200, default="https://e621.net")
|
e621_base_url = models.CharField(max_length=200, default="https://e621.net")
|
||||||
|
|||||||
@@ -1,12 +1,39 @@
|
|||||||
|
from pathlib import Path
|
||||||
|
|
||||||
from django.contrib.auth.password_validation import validate_password
|
from django.contrib.auth.password_validation import validate_password
|
||||||
|
from django.core import signing
|
||||||
from rest_framework import serializers
|
from rest_framework import serializers
|
||||||
|
|
||||||
|
from apps.library.services import MEDIA_FILE_SALT, VIDEO_EXTENSIONS
|
||||||
|
|
||||||
from .models import User
|
from .models import User
|
||||||
|
|
||||||
|
|
||||||
|
def signed_media_url(request, item):
|
||||||
|
"""A signed media URL an <img> tag can load for an authenticated user."""
|
||||||
|
if item is None:
|
||||||
|
return None
|
||||||
|
user = getattr(request, "user", None)
|
||||||
|
if user is None or not getattr(user, "is_authenticated", False):
|
||||||
|
return None
|
||||||
|
location = item.locations.first()
|
||||||
|
is_video = (
|
||||||
|
location is not None
|
||||||
|
and Path(location.rel_path).suffix.lower() in VIDEO_EXTENSIONS
|
||||||
|
)
|
||||||
|
action = "thumbnail" if is_video else "raw"
|
||||||
|
signature = signing.dumps(
|
||||||
|
{"item": item.id, "user": user.id, "action": action},
|
||||||
|
salt=MEDIA_FILE_SALT,
|
||||||
|
)
|
||||||
|
return f"/api/files/J-{item.id}/{action}/?sig={signature}"
|
||||||
|
|
||||||
|
|
||||||
class UserSerializer(serializers.ModelSerializer):
|
class UserSerializer(serializers.ModelSerializer):
|
||||||
e621_configured = serializers.BooleanField(read_only=True)
|
e621_configured = serializers.BooleanField(read_only=True)
|
||||||
can_upload = serializers.BooleanField(read_only=True)
|
can_upload = serializers.BooleanField(read_only=True)
|
||||||
|
avatar_j_id = serializers.SerializerMethodField()
|
||||||
|
avatar_url = serializers.SerializerMethodField()
|
||||||
|
|
||||||
class Meta:
|
class Meta:
|
||||||
model = User
|
model = User
|
||||||
@@ -19,11 +46,54 @@ class UserSerializer(serializers.ModelSerializer):
|
|||||||
"date_joined",
|
"date_joined",
|
||||||
"role",
|
"role",
|
||||||
"can_upload",
|
"can_upload",
|
||||||
|
"avatar_j_id",
|
||||||
|
"avatar_url",
|
||||||
"e621_username",
|
"e621_username",
|
||||||
"e621_configured",
|
"e621_configured",
|
||||||
]
|
]
|
||||||
read_only_fields = fields
|
read_only_fields = fields
|
||||||
|
|
||||||
|
def get_avatar_j_id(self, obj):
|
||||||
|
return f"J-{obj.avatar_id}" if obj.avatar_id else None
|
||||||
|
|
||||||
|
def get_avatar_url(self, obj):
|
||||||
|
return signed_media_url(self.context.get("request"), obj.avatar)
|
||||||
|
|
||||||
|
|
||||||
|
class UserListSerializer(serializers.ModelSerializer):
|
||||||
|
avatar_j_id = serializers.SerializerMethodField()
|
||||||
|
avatar_url = serializers.SerializerMethodField()
|
||||||
|
uploads_count = serializers.IntegerField(read_only=True)
|
||||||
|
|
||||||
|
class Meta:
|
||||||
|
model = User
|
||||||
|
fields = [
|
||||||
|
"id",
|
||||||
|
"username",
|
||||||
|
"email",
|
||||||
|
"role",
|
||||||
|
"is_staff",
|
||||||
|
"is_superuser",
|
||||||
|
"date_joined",
|
||||||
|
"avatar_j_id",
|
||||||
|
"avatar_url",
|
||||||
|
"uploads_count",
|
||||||
|
]
|
||||||
|
read_only_fields = fields
|
||||||
|
|
||||||
|
def get_avatar_j_id(self, obj):
|
||||||
|
return f"J-{obj.avatar_id}" if obj.avatar_id else None
|
||||||
|
|
||||||
|
def get_avatar_url(self, obj):
|
||||||
|
return signed_media_url(self.context.get("request"), obj.avatar)
|
||||||
|
|
||||||
|
|
||||||
|
class UserUpdateSerializer(serializers.Serializer):
|
||||||
|
avatar_j_id = serializers.CharField(
|
||||||
|
required=False, allow_blank=True, allow_null=True
|
||||||
|
)
|
||||||
|
role = serializers.ChoiceField(choices=User.ROLE_CHOICES, required=False)
|
||||||
|
|
||||||
|
|
||||||
class RegisterSerializer(serializers.ModelSerializer):
|
class RegisterSerializer(serializers.ModelSerializer):
|
||||||
password = serializers.CharField(write_only=True, validators=[validate_password])
|
password = serializers.CharField(write_only=True, validators=[validate_password])
|
||||||
|
|||||||
@@ -1,13 +1,19 @@
|
|||||||
from rest_framework import status
|
from rest_framework import mixins, permissions, status, viewsets
|
||||||
from rest_framework.authtoken.models import Token
|
from rest_framework.authtoken.models import Token
|
||||||
from rest_framework.permissions import AllowAny, IsAuthenticated
|
from rest_framework.permissions import AllowAny, IsAuthenticated
|
||||||
from rest_framework.response import Response
|
from rest_framework.response import Response
|
||||||
from rest_framework.views import APIView
|
from rest_framework.views import APIView
|
||||||
|
from django.db.models import Count, Q
|
||||||
|
|
||||||
|
from apps.library.models import MediaItem
|
||||||
|
|
||||||
|
from .models import User
|
||||||
from .serializers import (
|
from .serializers import (
|
||||||
E621CredentialsSerializer,
|
E621CredentialsSerializer,
|
||||||
RegisterSerializer,
|
RegisterSerializer,
|
||||||
|
UserListSerializer,
|
||||||
UserSerializer,
|
UserSerializer,
|
||||||
|
UserUpdateSerializer,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -20,7 +26,10 @@ class RegisterView(APIView):
|
|||||||
user = serializer.save()
|
user = serializer.save()
|
||||||
token, _ = Token.objects.get_or_create(user=user)
|
token, _ = Token.objects.get_or_create(user=user)
|
||||||
return Response(
|
return Response(
|
||||||
{"token": token.key, "user": UserSerializer(user).data},
|
{
|
||||||
|
"token": token.key,
|
||||||
|
"user": UserSerializer(user, context={"request": request}).data,
|
||||||
|
},
|
||||||
status=status.HTTP_201_CREATED,
|
status=status.HTTP_201_CREATED,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -29,7 +38,7 @@ class MeView(APIView):
|
|||||||
permission_classes = [IsAuthenticated]
|
permission_classes = [IsAuthenticated]
|
||||||
|
|
||||||
def get(self, request):
|
def get(self, request):
|
||||||
return Response(UserSerializer(request.user).data)
|
return Response(UserSerializer(request.user, context={"request": request}).data)
|
||||||
|
|
||||||
|
|
||||||
class LogoutView(APIView):
|
class LogoutView(APIView):
|
||||||
@@ -71,3 +80,81 @@ class E621CredentialsView(APIView):
|
|||||||
update_fields=["e621_username", "e621_api_key", "e621_base_url"]
|
update_fields=["e621_username", "e621_api_key", "e621_base_url"]
|
||||||
)
|
)
|
||||||
return Response(self._payload(user))
|
return Response(self._payload(user))
|
||||||
|
|
||||||
|
|
||||||
|
class IsStaffUser(permissions.BasePermission):
|
||||||
|
message = "Staff only."
|
||||||
|
|
||||||
|
def has_permission(self, request, view):
|
||||||
|
user = request.user
|
||||||
|
return bool(
|
||||||
|
user
|
||||||
|
and user.is_authenticated
|
||||||
|
and (
|
||||||
|
user.is_superuser
|
||||||
|
or user.is_staff
|
||||||
|
or user.role == user.ROLE_STAFF
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class UserViewSet(
|
||||||
|
mixins.ListModelMixin,
|
||||||
|
mixins.RetrieveModelMixin,
|
||||||
|
viewsets.GenericViewSet,
|
||||||
|
):
|
||||||
|
"""Staff user directory: roles and J-ID avatars."""
|
||||||
|
|
||||||
|
permission_classes = [IsStaffUser]
|
||||||
|
http_method_names = ["get", "patch", "head", "options"]
|
||||||
|
|
||||||
|
def get_queryset(self):
|
||||||
|
queryset = User.objects.annotate(uploads_count=Count("uploads")).order_by("id")
|
||||||
|
search = self.request.query_params.get("search", "").strip()
|
||||||
|
if search:
|
||||||
|
queryset = queryset.filter(
|
||||||
|
Q(username__icontains=search) | Q(email__icontains=search)
|
||||||
|
)
|
||||||
|
return queryset
|
||||||
|
|
||||||
|
def get_serializer_class(self):
|
||||||
|
if self.request.method in {"PATCH", "PUT"}:
|
||||||
|
return UserUpdateSerializer
|
||||||
|
return UserListSerializer
|
||||||
|
|
||||||
|
def update(self, request, *args, **kwargs):
|
||||||
|
user = self.get_object()
|
||||||
|
serializer = UserUpdateSerializer(data=request.data)
|
||||||
|
serializer.is_valid(raise_exception=True)
|
||||||
|
data = serializer.validated_data
|
||||||
|
|
||||||
|
update_fields = []
|
||||||
|
if "role" in data:
|
||||||
|
user.role = data["role"]
|
||||||
|
update_fields.append("role")
|
||||||
|
if "avatar_j_id" in data:
|
||||||
|
value = str(data.get("avatar_j_id") or "").strip()
|
||||||
|
if not value:
|
||||||
|
user.avatar = None
|
||||||
|
else:
|
||||||
|
numeric = value[2:] if value.upper().startswith("J-") else value
|
||||||
|
item = (
|
||||||
|
MediaItem.objects.filter(pk=int(numeric)).first()
|
||||||
|
if numeric.isdigit()
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
if item is None:
|
||||||
|
return Response(
|
||||||
|
{"detail": f"No library item {value}."},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
user.avatar = item
|
||||||
|
update_fields.append("avatar")
|
||||||
|
if update_fields:
|
||||||
|
user.save(update_fields=update_fields)
|
||||||
|
|
||||||
|
return Response(
|
||||||
|
UserListSerializer(user, context=self.get_serializer_context()).data
|
||||||
|
)
|
||||||
|
|
||||||
|
partial_update = update
|
||||||
|
|||||||
@@ -3,10 +3,12 @@
|
|||||||
import logging
|
import logging
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
|
from datetime import timedelta
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from urllib.parse import urlparse
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
from django.conf import settings
|
from django.conf import settings
|
||||||
|
from django.db import connection
|
||||||
from django.utils import timezone
|
from django.utils import timezone
|
||||||
|
|
||||||
from . import services
|
from . import services
|
||||||
@@ -14,6 +16,27 @@ from .models import DownloadTask
|
|||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
STALE_AFTER = timedelta(minutes=30)
|
||||||
|
|
||||||
|
|
||||||
|
def reap_stale_downloads():
|
||||||
|
"""Mark tasks left hanging by a recycled worker as failed.
|
||||||
|
|
||||||
|
Gunicorn recycles workers (--max-requests, timeouts); a download thread
|
||||||
|
dies with its worker, so long-stuck tasks are surfaced as errors instead
|
||||||
|
of pretending to run forever.
|
||||||
|
"""
|
||||||
|
cutoff = timezone.now() - STALE_AFTER
|
||||||
|
return DownloadTask.objects.filter(
|
||||||
|
status__in=[DownloadTask.STATUS_PENDING, DownloadTask.STATUS_DOWNLOADING],
|
||||||
|
updated_at__lt=cutoff,
|
||||||
|
).update(
|
||||||
|
status=DownloadTask.STATUS_ERROR,
|
||||||
|
error="The worker restarted before this download finished.",
|
||||||
|
speed=None,
|
||||||
|
updated_at=timezone.now(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def start_download_task(task_id):
|
def start_download_task(task_id):
|
||||||
thread = threading.Thread(target=run_download_task, args=(task_id,), daemon=True)
|
thread = threading.Thread(target=run_download_task, args=(task_id,), daemon=True)
|
||||||
@@ -79,6 +102,7 @@ def run_download_task(task_id):
|
|||||||
)
|
)
|
||||||
item, _, location, _ = services.index_file(destination, folder)
|
item, _, location, _ = services.index_file(destination, folder)
|
||||||
services.rename_location_to_j_id(item, location)
|
services.rename_location_to_j_id(item, location)
|
||||||
|
services.ensure_visual_hashes(item)
|
||||||
|
|
||||||
update_fields = []
|
update_fields = []
|
||||||
if item.uploaded_by_id is None and task.user_id is not None:
|
if item.uploaded_by_id is None and task.user_id is not None:
|
||||||
@@ -120,3 +144,7 @@ def run_download_task(task_id):
|
|||||||
speed=None,
|
speed=None,
|
||||||
updated_at=timezone.now(),
|
updated_at=timezone.now(),
|
||||||
)
|
)
|
||||||
|
finally:
|
||||||
|
# Background threads hold their own DB connection; release it so
|
||||||
|
# Gunicorn workers do not leak connections when threads finish.
|
||||||
|
connection.close()
|
||||||
|
|||||||
@@ -0,0 +1,43 @@
|
|||||||
|
from django.core.management.base import BaseCommand
|
||||||
|
from django.db.models import Q
|
||||||
|
|
||||||
|
from apps.library.models import MediaItem
|
||||||
|
from apps.library.services import HASH_FIELDS, ensure_visual_hashes
|
||||||
|
|
||||||
|
|
||||||
|
class Command(BaseCommand):
|
||||||
|
help = "Compute perceptual hashes for library items missing them."
|
||||||
|
|
||||||
|
def add_arguments(self, parser):
|
||||||
|
parser.add_argument(
|
||||||
|
"--force",
|
||||||
|
action="store_true",
|
||||||
|
help="Recompute hashes even when they already exist.",
|
||||||
|
)
|
||||||
|
|
||||||
|
def handle(self, *args, **options):
|
||||||
|
queryset = MediaItem.objects.prefetch_related("locations").order_by("id")
|
||||||
|
if not options["force"]:
|
||||||
|
missing = Q()
|
||||||
|
for field in HASH_FIELDS:
|
||||||
|
missing |= Q(**{field: ""})
|
||||||
|
queryset = queryset.filter(missing)
|
||||||
|
|
||||||
|
total = queryset.count()
|
||||||
|
if total == 0:
|
||||||
|
self.stdout.write(self.style.SUCCESS("All items already have hashes."))
|
||||||
|
return
|
||||||
|
|
||||||
|
self.stdout.write(f"Computing hashes for {total} item(s) ...")
|
||||||
|
done = 0
|
||||||
|
for item in queryset.iterator(chunk_size=100):
|
||||||
|
if options["force"]:
|
||||||
|
item.ahash = item.dhash = item.phash = item.whash = ""
|
||||||
|
ensure_visual_hashes(item)
|
||||||
|
done += 1
|
||||||
|
if done % 100 == 0:
|
||||||
|
self.stdout.write(f"Processed {done} ...")
|
||||||
|
|
||||||
|
self.stdout.write(
|
||||||
|
self.style.SUCCESS(f"Done. Processed {done} item(s).")
|
||||||
|
)
|
||||||
+33
@@ -0,0 +1,33 @@
|
|||||||
|
# Generated by Django 6.1.1 on 2026-09-17 17:34
|
||||||
|
|
||||||
|
from django.db import migrations, models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
|
||||||
|
dependencies = [
|
||||||
|
('library', '0005_downloadtask'),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='mediaitem',
|
||||||
|
name='ahash',
|
||||||
|
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='mediaitem',
|
||||||
|
name='dhash',
|
||||||
|
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='mediaitem',
|
||||||
|
name='phash',
|
||||||
|
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='mediaitem',
|
||||||
|
name='whash',
|
||||||
|
field=models.CharField(blank=True, db_index=True, default='', max_length=32),
|
||||||
|
),
|
||||||
|
]
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
# Generated by Django 6.1.1 on 2026-09-17 17:42
|
||||||
|
|
||||||
|
from django.db import migrations, models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
|
||||||
|
dependencies = [
|
||||||
|
('library', '0006_mediaitem_ahash_mediaitem_dhash_mediaitem_phash_and_more'),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='tempupload',
|
||||||
|
name='visual_matches',
|
||||||
|
field=models.JSONField(blank=True, null=True),
|
||||||
|
),
|
||||||
|
]
|
||||||
@@ -24,6 +24,11 @@ class MediaItem(models.Model):
|
|||||||
hidden_from_guests = models.BooleanField(default=False, db_index=True)
|
hidden_from_guests = models.BooleanField(default=False, db_index=True)
|
||||||
e621_post_id = models.IntegerField(null=True, blank=True, db_index=True)
|
e621_post_id = models.IntegerField(null=True, blank=True, db_index=True)
|
||||||
e621_data = models.JSONField(null=True, blank=True)
|
e621_data = models.JSONField(null=True, blank=True)
|
||||||
|
# Perceptual hashes (hex strings) used by the duplicates engine.
|
||||||
|
ahash = models.CharField(max_length=32, blank=True, default="", db_index=True)
|
||||||
|
dhash = models.CharField(max_length=32, blank=True, default="", db_index=True)
|
||||||
|
phash = models.CharField(max_length=32, blank=True, default="", db_index=True)
|
||||||
|
whash = models.CharField(max_length=32, blank=True, default="", db_index=True)
|
||||||
created_at = models.DateTimeField(auto_now_add=True)
|
created_at = models.DateTimeField(auto_now_add=True)
|
||||||
updated_at = models.DateTimeField(auto_now=True)
|
updated_at = models.DateTimeField(auto_now=True)
|
||||||
|
|
||||||
@@ -100,6 +105,7 @@ class TempUpload(models.Model):
|
|||||||
custom_tags = models.JSONField(default=list, blank=True)
|
custom_tags = models.JSONField(default=list, blank=True)
|
||||||
custom_notes = models.TextField(blank=True, default="")
|
custom_notes = models.TextField(blank=True, default="")
|
||||||
iqdb_data = models.JSONField(null=True, blank=True)
|
iqdb_data = models.JSONField(null=True, blank=True)
|
||||||
|
visual_matches = models.JSONField(null=True, blank=True)
|
||||||
library_item = models.ForeignKey(
|
library_item = models.ForeignKey(
|
||||||
MediaItem,
|
MediaItem,
|
||||||
null=True,
|
null=True,
|
||||||
|
|||||||
@@ -117,6 +117,7 @@ class TempUploadSerializer(serializers.ModelSerializer):
|
|||||||
"custom_tags",
|
"custom_tags",
|
||||||
"custom_notes",
|
"custom_notes",
|
||||||
"iqdb_data",
|
"iqdb_data",
|
||||||
|
"visual_matches",
|
||||||
"library_j_id",
|
"library_j_id",
|
||||||
"file_url",
|
"file_url",
|
||||||
"preview_url",
|
"preview_url",
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import hashlib
|
import hashlib
|
||||||
import json
|
import json
|
||||||
|
import logging
|
||||||
import mimetypes
|
import mimetypes
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
@@ -7,12 +8,19 @@ import shutil
|
|||||||
import subprocess
|
import subprocess
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import imagehash
|
||||||
from django.conf import settings
|
from django.conf import settings
|
||||||
from django.http import FileResponse, Http404, HttpResponse
|
from django.http import FileResponse, Http404, HttpResponse
|
||||||
from django.utils.text import get_valid_filename
|
from django.utils.text import get_valid_filename
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
from .models import MediaItem, MediaLocation
|
from .models import MediaItem, MediaLocation
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HASH_FIELDS = ("ahash", "dhash", "phash", "whash")
|
||||||
|
IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".gif", ".apng", ".webp"}
|
||||||
|
|
||||||
ALLOWED_EXTENSIONS = {
|
ALLOWED_EXTENSIONS = {
|
||||||
".jpg",
|
".jpg",
|
||||||
".jpeg",
|
".jpeg",
|
||||||
@@ -82,6 +90,41 @@ def rename_location_to_j_id(item, location):
|
|||||||
return location
|
return location
|
||||||
|
|
||||||
|
|
||||||
|
def compute_visual_hashes(path):
|
||||||
|
"""Perceptual hashes for an image file (empty dict for other files)."""
|
||||||
|
path = Path(path)
|
||||||
|
if path.suffix.lower() not in IMAGE_EXTENSIONS:
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
with Image.open(path) as image:
|
||||||
|
converted = image.convert("RGB")
|
||||||
|
return {
|
||||||
|
"ahash": str(imagehash.average_hash(converted, hash_size=8)),
|
||||||
|
"dhash": str(imagehash.dhash(converted, hash_size=8)),
|
||||||
|
"phash": str(imagehash.phash(converted, hash_size=8)),
|
||||||
|
"whash": str(imagehash.whash(converted, hash_size=8)),
|
||||||
|
}
|
||||||
|
except Exception: # noqa: BLE001 - hashing must never break indexing
|
||||||
|
logger.exception("Could not compute visual hashes for %s", path)
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_visual_hashes(item):
|
||||||
|
"""Fill in missing perceptual hashes for a media item."""
|
||||||
|
if all(getattr(item, field) for field in HASH_FIELDS):
|
||||||
|
return item
|
||||||
|
location = item.locations.first()
|
||||||
|
if location is None:
|
||||||
|
return item
|
||||||
|
hashes = compute_visual_hashes(location.path)
|
||||||
|
if not hashes:
|
||||||
|
return item
|
||||||
|
for field, value in hashes.items():
|
||||||
|
setattr(item, field, value)
|
||||||
|
item.save(update_fields=[*hashes.keys(), "updated_at"])
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
def parse_tags(raw):
|
def parse_tags(raw):
|
||||||
"""Normalize a comma-separated string or JSON list into a list of tags."""
|
"""Normalize a comma-separated string or JSON list into a list of tags."""
|
||||||
if raw is None:
|
if raw is None:
|
||||||
|
|||||||
@@ -0,0 +1,394 @@
|
|||||||
|
"""Library tooling: duplicates, deletion and storage overview."""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import shutil
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from django.conf import settings
|
||||||
|
from django.core.cache import cache
|
||||||
|
from django.db.models import Count, Q
|
||||||
|
from rest_framework import status
|
||||||
|
from rest_framework.permissions import IsAuthenticated
|
||||||
|
from rest_framework.response import Response
|
||||||
|
from rest_framework.views import APIView
|
||||||
|
|
||||||
|
from .models import MediaItem, MediaLocation
|
||||||
|
from .permissions import CanUpload
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HASH_FIELDS = ("ahash", "dhash", "phash", "whash")
|
||||||
|
STORAGE_CACHE_KEY = "j621.library.storage"
|
||||||
|
STORAGE_CACHE_TTL = 60
|
||||||
|
GROUPS_PER_PAGE = 20
|
||||||
|
|
||||||
|
|
||||||
|
def parse_threshold(value, default=0.8):
|
||||||
|
try:
|
||||||
|
parsed = float(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return default
|
||||||
|
if parsed > 1:
|
||||||
|
parsed = parsed / 100.0
|
||||||
|
return min(max(parsed, 0.0), 1.0)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_algorithms(value):
|
||||||
|
if not isinstance(value, list):
|
||||||
|
return list(HASH_FIELDS)
|
||||||
|
selected = [field for field in value if field in HASH_FIELDS]
|
||||||
|
return selected or list(HASH_FIELDS)
|
||||||
|
|
||||||
|
|
||||||
|
def hashes_similarity(first_hashes, second_hashes, algorithms, threshold):
|
||||||
|
"""Best similarity between two hash mappings, or None below threshold."""
|
||||||
|
best = None
|
||||||
|
for field in algorithms:
|
||||||
|
first = first_hashes.get(field) or ""
|
||||||
|
second = second_hashes.get(field) or ""
|
||||||
|
if not first or not second:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
distance = bin(int(first, 16) ^ int(second, 16)).count("1")
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
value = 1.0 - distance / 64.0
|
||||||
|
if best is None or value > best:
|
||||||
|
best = value
|
||||||
|
if best is None or best < threshold:
|
||||||
|
return None
|
||||||
|
return best
|
||||||
|
|
||||||
|
|
||||||
|
def similarity_between(item_a, item_b, algorithms, threshold):
|
||||||
|
"""Best similarity across the selected algorithms, or None below threshold."""
|
||||||
|
return hashes_similarity(
|
||||||
|
{field: getattr(item_a, field, "") for field in algorithms},
|
||||||
|
{field: getattr(item_b, field, "") for field in algorithms},
|
||||||
|
algorithms,
|
||||||
|
threshold,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def hashed_items(algorithms):
|
||||||
|
query = Q()
|
||||||
|
for field in algorithms:
|
||||||
|
query |= ~Q(**{field: ""})
|
||||||
|
return list(MediaItem.objects.filter(query).prefetch_related("locations"))
|
||||||
|
|
||||||
|
|
||||||
|
def display_rating(item):
|
||||||
|
if item.rating:
|
||||||
|
return item.rating
|
||||||
|
data = item.e621_data or {}
|
||||||
|
rating = data.get("rating") if isinstance(data, dict) else None
|
||||||
|
return rating if rating in {"s", "q", "e"} else ""
|
||||||
|
|
||||||
|
|
||||||
|
def item_brief(item):
|
||||||
|
locations = list(item.locations.all())
|
||||||
|
location = locations[0] if locations else None
|
||||||
|
return {
|
||||||
|
"j_id": f"J-{item.id}",
|
||||||
|
"md5": item.md5,
|
||||||
|
"filename": Path(location.rel_path).name if location else item.md5,
|
||||||
|
"size": item.size,
|
||||||
|
"rating": display_rating(item),
|
||||||
|
"location_count": len(locations),
|
||||||
|
"uploaded_by": item.uploaded_by.username if item.uploaded_by else None,
|
||||||
|
"e621_post_id": item.e621_post_id,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_item(data):
|
||||||
|
j_id = str(data.get("j_id") or "").strip()
|
||||||
|
md5 = str(data.get("md5") or "").strip().lower()
|
||||||
|
if j_id:
|
||||||
|
numeric = j_id[2:] if j_id.upper().startswith("J-") else j_id
|
||||||
|
if numeric.isdigit():
|
||||||
|
item = MediaItem.objects.filter(pk=int(numeric)).first()
|
||||||
|
if item is not None:
|
||||||
|
return item
|
||||||
|
if md5:
|
||||||
|
return MediaItem.objects.filter(md5=md5).first()
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class ExactDuplicatesView(APIView):
|
||||||
|
"""Items whose content exists at more than one path."""
|
||||||
|
|
||||||
|
permission_classes = [IsAuthenticated]
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
items = (
|
||||||
|
MediaItem.objects.annotate(location_count=Count("locations"))
|
||||||
|
.filter(location_count__gt=1)
|
||||||
|
.prefetch_related("locations")
|
||||||
|
.order_by("-location_count", "id")
|
||||||
|
)
|
||||||
|
groups = []
|
||||||
|
for item in items:
|
||||||
|
brief = item_brief(item)
|
||||||
|
brief["locations"] = [
|
||||||
|
{"id": location.id, "rel_path": location.rel_path}
|
||||||
|
for location in item.locations.all()
|
||||||
|
]
|
||||||
|
groups.append(brief)
|
||||||
|
return Response({"count": len(groups), "groups": groups})
|
||||||
|
|
||||||
|
|
||||||
|
class VisualMatchesView(APIView):
|
||||||
|
"""Items visually similar to one library item."""
|
||||||
|
|
||||||
|
permission_classes = [IsAuthenticated]
|
||||||
|
|
||||||
|
def post(self, request):
|
||||||
|
threshold = parse_threshold(request.data.get("threshold"))
|
||||||
|
algorithms = parse_algorithms(request.data.get("algorithms"))
|
||||||
|
target = resolve_item(request.data)
|
||||||
|
if target is None:
|
||||||
|
return Response(
|
||||||
|
{"detail": "A j_id or md5 is required."},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
matches = []
|
||||||
|
for item in hashed_items(algorithms):
|
||||||
|
if item.pk == target.pk:
|
||||||
|
continue
|
||||||
|
similarity = similarity_between(target, item, algorithms, threshold)
|
||||||
|
if similarity is None:
|
||||||
|
continue
|
||||||
|
brief = item_brief(item)
|
||||||
|
brief["similarity"] = round(similarity * 100, 1)
|
||||||
|
matches.append(brief)
|
||||||
|
matches.sort(key=lambda entry: entry["similarity"], reverse=True)
|
||||||
|
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"target": item_brief(target),
|
||||||
|
"threshold": round(threshold * 100, 1),
|
||||||
|
"algorithms": algorithms,
|
||||||
|
"count": len(matches),
|
||||||
|
"matches": matches[:200],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class VisualGroupsView(APIView):
|
||||||
|
"""Groups of visually similar items (union-find over perceptual hashes).
|
||||||
|
|
||||||
|
Pairwise comparison is O(n^2) with fast bit operations, which is fine for
|
||||||
|
a personal library. Revisit with a bucketed index if libraries grow huge.
|
||||||
|
"""
|
||||||
|
|
||||||
|
permission_classes = [IsAuthenticated]
|
||||||
|
|
||||||
|
def post(self, request):
|
||||||
|
threshold = parse_threshold(request.data.get("threshold"))
|
||||||
|
algorithms = parse_algorithms(request.data.get("algorithms"))
|
||||||
|
page = max(1, int(request.data.get("page") or 1))
|
||||||
|
|
||||||
|
items = hashed_items(algorithms)
|
||||||
|
parent = list(range(len(items)))
|
||||||
|
|
||||||
|
def find(index):
|
||||||
|
while parent[index] != index:
|
||||||
|
parent[index] = parent[parent[index]]
|
||||||
|
index = parent[index]
|
||||||
|
return index
|
||||||
|
|
||||||
|
def union(first, second):
|
||||||
|
root_a = find(first)
|
||||||
|
root_b = find(second)
|
||||||
|
if root_a != root_b:
|
||||||
|
parent[root_b] = root_a
|
||||||
|
|
||||||
|
for first in range(len(items)):
|
||||||
|
for second in range(first + 1, len(items)):
|
||||||
|
if find(first) == find(second):
|
||||||
|
continue
|
||||||
|
if (
|
||||||
|
similarity_between(
|
||||||
|
items[first], items[second], algorithms, threshold
|
||||||
|
)
|
||||||
|
is not None
|
||||||
|
):
|
||||||
|
union(first, second)
|
||||||
|
|
||||||
|
grouped = {}
|
||||||
|
for index, item in enumerate(items):
|
||||||
|
grouped.setdefault(find(index), []).append(item)
|
||||||
|
|
||||||
|
groups = [members for members in grouped.values() if len(members) >= 2]
|
||||||
|
groups.sort(key=len, reverse=True)
|
||||||
|
|
||||||
|
total = len(groups)
|
||||||
|
start = (page - 1) * GROUPS_PER_PAGE
|
||||||
|
page_groups = groups[start : start + GROUPS_PER_PAGE]
|
||||||
|
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"count": total,
|
||||||
|
"page": page,
|
||||||
|
"per_page": GROUPS_PER_PAGE,
|
||||||
|
"has_next": start + GROUPS_PER_PAGE < total,
|
||||||
|
"threshold": round(threshold * 100, 1),
|
||||||
|
"algorithms": algorithms,
|
||||||
|
"groups": [
|
||||||
|
{"size": len(members), "members": [item_brief(item) for item in members]}
|
||||||
|
for members in page_groups
|
||||||
|
],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def can_delete(user, item):
|
||||||
|
if user.is_superuser or user.role == user.ROLE_STAFF:
|
||||||
|
return True
|
||||||
|
return item.uploaded_by_id == user.id
|
||||||
|
|
||||||
|
|
||||||
|
def remove_watched_file(path):
|
||||||
|
"""Delete a file only when it lives inside the watched folder."""
|
||||||
|
watched = Path(settings.WATCHED_FOLDER).resolve()
|
||||||
|
try:
|
||||||
|
resolved = Path(path).resolve()
|
||||||
|
resolved.relative_to(watched)
|
||||||
|
except (ValueError, OSError):
|
||||||
|
return False
|
||||||
|
if resolved.is_file():
|
||||||
|
resolved.unlink()
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
class DeleteFilesView(APIView):
|
||||||
|
"""Delete items (with every copy) or individual duplicate locations."""
|
||||||
|
|
||||||
|
permission_classes = [CanUpload]
|
||||||
|
|
||||||
|
def post(self, request):
|
||||||
|
j_ids = request.data.get("j_ids") or []
|
||||||
|
location_ids = request.data.get("location_ids") or []
|
||||||
|
if not isinstance(j_ids, list) or not isinstance(location_ids, list):
|
||||||
|
return Response(
|
||||||
|
{"detail": "j_ids and location_ids must be lists."},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
deleted = []
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
numeric_ids = []
|
||||||
|
for value in j_ids:
|
||||||
|
text = str(value).strip()
|
||||||
|
numeric = text[2:] if text.upper().startswith("J-") else text
|
||||||
|
if numeric.isdigit():
|
||||||
|
numeric_ids.append(int(numeric))
|
||||||
|
|
||||||
|
for item in MediaItem.objects.filter(pk__in=numeric_ids).prefetch_related(
|
||||||
|
"locations"
|
||||||
|
):
|
||||||
|
if not can_delete(request.user, item):
|
||||||
|
errors.append({"j_id": f"J-{item.id}", "error": "permission denied"})
|
||||||
|
continue
|
||||||
|
for location in item.locations.all():
|
||||||
|
remove_watched_file(location.path)
|
||||||
|
deleted.append(f"J-{item.id}")
|
||||||
|
item.delete()
|
||||||
|
|
||||||
|
cache.delete(STORAGE_CACHE_KEY)
|
||||||
|
|
||||||
|
for location in MediaLocation.objects.filter(
|
||||||
|
id__in=location_ids
|
||||||
|
).select_related("item"):
|
||||||
|
if not can_delete(request.user, location.item):
|
||||||
|
errors.append(
|
||||||
|
{"location": location.id, "error": "permission denied"}
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
location_id = location.id
|
||||||
|
item = location.item
|
||||||
|
remove_watched_file(location.path)
|
||||||
|
location.delete()
|
||||||
|
if item.locations.exists():
|
||||||
|
deleted.append(f"location {location_id}")
|
||||||
|
else:
|
||||||
|
deleted.append(f"J-{item.id}")
|
||||||
|
item.delete()
|
||||||
|
|
||||||
|
return Response({"deleted": deleted, "errors": errors})
|
||||||
|
|
||||||
|
|
||||||
|
class ClearTempView(APIView):
|
||||||
|
"""Remove staged files from the temp upload folder."""
|
||||||
|
|
||||||
|
permission_classes = [CanUpload]
|
||||||
|
|
||||||
|
def post(self, request):
|
||||||
|
temp_dir = Path(settings.MEDIA_ROOT) / "uploads" / "temp"
|
||||||
|
removed = 0
|
||||||
|
if temp_dir.exists():
|
||||||
|
for entry in temp_dir.iterdir():
|
||||||
|
if entry.is_file():
|
||||||
|
entry.unlink()
|
||||||
|
removed += 1
|
||||||
|
cache.delete(STORAGE_CACHE_KEY)
|
||||||
|
return Response({"removed": removed})
|
||||||
|
|
||||||
|
|
||||||
|
class StorageView(APIView):
|
||||||
|
"""Disk usage for the watched folder, media root and temp uploads."""
|
||||||
|
|
||||||
|
permission_classes = [IsAuthenticated]
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
cached = cache.get(STORAGE_CACHE_KEY)
|
||||||
|
if cached is not None:
|
||||||
|
return Response(cached)
|
||||||
|
|
||||||
|
watched = Path(settings.WATCHED_FOLDER)
|
||||||
|
media_root = Path(settings.MEDIA_ROOT)
|
||||||
|
temp_dir = media_root / "uploads" / "temp"
|
||||||
|
|
||||||
|
usage = shutil.disk_usage(
|
||||||
|
str(watched) if watched.exists() else str(Path(settings.BASE_DIR))
|
||||||
|
)
|
||||||
|
library_size = (
|
||||||
|
sum(file.stat().st_size for file in watched.rglob("*") if file.is_file())
|
||||||
|
if watched.exists()
|
||||||
|
else 0
|
||||||
|
)
|
||||||
|
media_size = (
|
||||||
|
sum(file.stat().st_size for file in media_root.rglob("*") if file.is_file())
|
||||||
|
if media_root.exists()
|
||||||
|
else 0
|
||||||
|
)
|
||||||
|
temp_files = (
|
||||||
|
[file for file in temp_dir.rglob("*") if file.is_file()]
|
||||||
|
if temp_dir.exists()
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"watched_folder": {
|
||||||
|
"path": str(watched),
|
||||||
|
"total": usage.total,
|
||||||
|
"used": usage.used,
|
||||||
|
"free": usage.free,
|
||||||
|
"library_size": library_size,
|
||||||
|
"percent_used": (
|
||||||
|
round(usage.used / usage.total * 100, 1) if usage.total else 0
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"media": {"path": str(media_root), "size": media_size},
|
||||||
|
"temp": {
|
||||||
|
"path": str(temp_dir),
|
||||||
|
"size": sum(file.stat().st_size for file in temp_files),
|
||||||
|
"files": len(temp_files),
|
||||||
|
},
|
||||||
|
"library_items": MediaItem.objects.count(),
|
||||||
|
}
|
||||||
|
cache.set(STORAGE_CACHE_KEY, payload, STORAGE_CACHE_TTL)
|
||||||
|
return Response(payload)
|
||||||
@@ -28,10 +28,40 @@ from . import services
|
|||||||
from .models import MediaItem, TempUpload
|
from .models import MediaItem, TempUpload
|
||||||
from .permissions import CanUpload
|
from .permissions import CanUpload
|
||||||
from .serializers import TempUploadSerializer
|
from .serializers import TempUploadSerializer
|
||||||
|
from .tools import HASH_FIELDS, hashes_similarity
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def find_library_matches(path, limit=10):
|
||||||
|
"""Library items visually similar to a staged file."""
|
||||||
|
hashes = services.compute_visual_hashes(path)
|
||||||
|
if not hashes:
|
||||||
|
return []
|
||||||
|
algorithms = list(HASH_FIELDS)
|
||||||
|
threshold = settings.VISUAL_MATCH_THRESHOLD
|
||||||
|
matches = []
|
||||||
|
for item in MediaItem.objects.prefetch_related("locations"):
|
||||||
|
similarity = hashes_similarity(
|
||||||
|
hashes,
|
||||||
|
{field: getattr(item, field, "") for field in algorithms},
|
||||||
|
algorithms,
|
||||||
|
threshold,
|
||||||
|
)
|
||||||
|
if similarity is None:
|
||||||
|
continue
|
||||||
|
location = item.locations.first()
|
||||||
|
matches.append(
|
||||||
|
{
|
||||||
|
"j_id": f"J-{item.id}",
|
||||||
|
"filename": Path(location.rel_path).name if location else item.md5,
|
||||||
|
"similarity": round(similarity * 100, 1),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
matches.sort(key=lambda entry: entry["similarity"], reverse=True)
|
||||||
|
return matches[:limit]
|
||||||
|
|
||||||
|
|
||||||
def complete_temp_upload(temp, download_url=None):
|
def complete_temp_upload(temp, download_url=None):
|
||||||
"""Index the upload into the library.
|
"""Index the upload into the library.
|
||||||
|
|
||||||
@@ -57,6 +87,7 @@ def complete_temp_upload(temp, download_url=None):
|
|||||||
raise
|
raise
|
||||||
item, _, location, _ = services.index_file(destination, folder)
|
item, _, location, _ = services.index_file(destination, folder)
|
||||||
services.rename_location_to_j_id(item, location)
|
services.rename_location_to_j_id(item, location)
|
||||||
|
services.ensure_visual_hashes(item)
|
||||||
if temp.file:
|
if temp.file:
|
||||||
temp.file.delete(save=False)
|
temp.file.delete(save=False)
|
||||||
else:
|
else:
|
||||||
@@ -69,6 +100,7 @@ def complete_temp_upload(temp, download_url=None):
|
|||||||
shutil.copyfileobj(source, target)
|
shutil.copyfileobj(source, target)
|
||||||
item, _, location, _ = services.index_file(destination, folder)
|
item, _, location, _ = services.index_file(destination, folder)
|
||||||
services.rename_location_to_j_id(item, location)
|
services.rename_location_to_j_id(item, location)
|
||||||
|
services.ensure_visual_hashes(item)
|
||||||
temp.file.delete(save=False)
|
temp.file.delete(save=False)
|
||||||
|
|
||||||
temp.library_item = item
|
temp.library_item = item
|
||||||
@@ -143,6 +175,11 @@ class TempUploadViewSet(
|
|||||||
temp.resolution = TempUpload.RESOLUTION_DUPLICATE
|
temp.resolution = TempUpload.RESOLUTION_DUPLICATE
|
||||||
temp.library_item = existing
|
temp.library_item = existing
|
||||||
temp.file.delete(save=False)
|
temp.file.delete(save=False)
|
||||||
|
else:
|
||||||
|
matches = find_library_matches(temp.file.path)
|
||||||
|
if matches:
|
||||||
|
temp.visual_matches = matches
|
||||||
|
temp.status = TempUpload.STATUS_VISUAL_MATCH
|
||||||
temp.save()
|
temp.save()
|
||||||
return Response(
|
return Response(
|
||||||
self.get_serializer(temp).data, status=status.HTTP_201_CREATED
|
self.get_serializer(temp).data, status=status.HTTP_201_CREATED
|
||||||
|
|||||||
@@ -1,6 +1,14 @@
|
|||||||
from django.urls import include, path
|
from django.urls import include, path
|
||||||
from rest_framework.routers import DefaultRouter
|
from rest_framework.routers import DefaultRouter
|
||||||
|
|
||||||
|
from .tools import (
|
||||||
|
ClearTempView,
|
||||||
|
DeleteFilesView,
|
||||||
|
ExactDuplicatesView,
|
||||||
|
StorageView,
|
||||||
|
VisualGroupsView,
|
||||||
|
VisualMatchesView,
|
||||||
|
)
|
||||||
from .uploads import TempUploadViewSet
|
from .uploads import TempUploadViewSet
|
||||||
from .views import ClientDownloadView, DownloadTaskViewSet, MediaItemViewSet
|
from .views import ClientDownloadView, DownloadTaskViewSet, MediaItemViewSet
|
||||||
|
|
||||||
@@ -12,4 +20,22 @@ router.register("online/downloads", DownloadTaskViewSet, basename="download")
|
|||||||
urlpatterns = [
|
urlpatterns = [
|
||||||
path("", include(router.urls)),
|
path("", include(router.urls)),
|
||||||
path("online/file/", ClientDownloadView.as_view(), name="client_download"),
|
path("online/file/", ClientDownloadView.as_view(), name="client_download"),
|
||||||
|
path(
|
||||||
|
"duplicates/md5/",
|
||||||
|
ExactDuplicatesView.as_view(),
|
||||||
|
name="duplicates_md5",
|
||||||
|
),
|
||||||
|
path(
|
||||||
|
"duplicates/visual/",
|
||||||
|
VisualMatchesView.as_view(),
|
||||||
|
name="duplicates_visual",
|
||||||
|
),
|
||||||
|
path(
|
||||||
|
"duplicates/visual_groups/",
|
||||||
|
VisualGroupsView.as_view(),
|
||||||
|
name="duplicates_groups",
|
||||||
|
),
|
||||||
|
path("delete/", DeleteFilesView.as_view(), name="delete_files"),
|
||||||
|
path("temp/clear/", ClearTempView.as_view(), name="clear_temp"),
|
||||||
|
path("storage/", StorageView.as_view(), name="storage_info"),
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ from rest_framework.response import Response
|
|||||||
from rest_framework.views import APIView
|
from rest_framework.views import APIView
|
||||||
|
|
||||||
from . import services
|
from . import services
|
||||||
from .downloads import start_download_task
|
from .downloads import reap_stale_downloads, start_download_task
|
||||||
from .models import DownloadTask, MediaItem
|
from .models import DownloadTask, MediaItem
|
||||||
from .permissions import CanUpload, IsUploaderOrStaffOrReadOnly
|
from .permissions import CanUpload, IsUploaderOrStaffOrReadOnly
|
||||||
from .serializers import DownloadTaskSerializer, MediaItemSerializer
|
from .serializers import DownloadTaskSerializer, MediaItemSerializer
|
||||||
@@ -41,6 +41,15 @@ class MediaItemViewSet(
|
|||||||
)
|
)
|
||||||
if not self.request.user.is_authenticated:
|
if not self.request.user.is_authenticated:
|
||||||
queryset = queryset.filter(hidden_from_guests=False)
|
queryset = queryset.filter(hidden_from_guests=False)
|
||||||
|
j_ids = self.request.query_params.get("j_ids", "").strip()
|
||||||
|
if j_ids:
|
||||||
|
numeric_ids = []
|
||||||
|
for value in j_ids.split(","):
|
||||||
|
text = value.strip()
|
||||||
|
number = text[2:] if text.upper().startswith("J-") else text
|
||||||
|
if number.isdigit():
|
||||||
|
numeric_ids.append(int(number))
|
||||||
|
queryset = queryset.filter(pk__in=numeric_ids)
|
||||||
search = self.request.query_params.get("search", "").strip()
|
search = self.request.query_params.get("search", "").strip()
|
||||||
if search:
|
if search:
|
||||||
queryset = queryset.filter(locations__rel_path__icontains=search)
|
queryset = queryset.filter(locations__rel_path__icontains=search)
|
||||||
@@ -203,6 +212,7 @@ class DownloadTaskViewSet(
|
|||||||
http_method_names = ["get", "post", "head", "options"]
|
http_method_names = ["get", "post", "head", "options"]
|
||||||
|
|
||||||
def get_queryset(self):
|
def get_queryset(self):
|
||||||
|
reap_stale_downloads()
|
||||||
queryset = DownloadTask.objects.select_related("library_item")
|
queryset = DownloadTask.objects.select_related("library_item")
|
||||||
user = self.request.user
|
user = self.request.user
|
||||||
if not (user.is_staff or user.is_superuser):
|
if not (user.is_staff or user.is_superuser):
|
||||||
|
|||||||
@@ -166,6 +166,9 @@ GUEST_BLACKLIST_FALLBACK = [
|
|||||||
]
|
]
|
||||||
GUEST_BLACKLIST_TTL = int(os.getenv("GUEST_BLACKLIST_TTL", "3600"))
|
GUEST_BLACKLIST_TTL = int(os.getenv("GUEST_BLACKLIST_TTL", "3600"))
|
||||||
|
|
||||||
|
# Similarity threshold for flagging staged uploads that match library items.
|
||||||
|
VISUAL_MATCH_THRESHOLD = float(os.getenv("VISUAL_MATCH_THRESHOLD", "0.9"))
|
||||||
|
|
||||||
# Redis cache (run via docker compose at the repo root), shared by web
|
# Redis cache (run via docker compose at the repo root), shared by web
|
||||||
# workers and management commands (e.g. the mirrored guest blacklist).
|
# workers and management commands (e.g. the mirrored guest blacklist).
|
||||||
CACHES = {
|
CACHES = {
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ from django.urls import include, path
|
|||||||
urlpatterns = [
|
urlpatterns = [
|
||||||
path("admin/", admin.site.urls),
|
path("admin/", admin.site.urls),
|
||||||
path("api/auth/", include("apps.accounts.urls")),
|
path("api/auth/", include("apps.accounts.urls")),
|
||||||
|
path("api/", include("apps.accounts.api_urls")),
|
||||||
path("api/", include("apps.core.urls")),
|
path("api/", include("apps.core.urls")),
|
||||||
path("api/", include("apps.library.urls")),
|
path("api/", include("apps.library.urls")),
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -4,5 +4,6 @@ django-filter>=25.1
|
|||||||
Pillow>=11.0
|
Pillow>=11.0
|
||||||
python-dotenv>=1.0
|
python-dotenv>=1.0
|
||||||
requests>=2.32
|
requests>=2.32
|
||||||
|
imagehash>=4.3
|
||||||
PyMySQL>=1.1
|
PyMySQL>=1.1
|
||||||
redis>=5.0
|
redis>=5.0
|
||||||
|
|||||||
@@ -13,10 +13,13 @@ import AccountPage from "@/features/account/AccountPage";
|
|||||||
import LoginPage from "@/features/auth/LoginPage";
|
import LoginPage from "@/features/auth/LoginPage";
|
||||||
import RegisterPage from "@/features/auth/RegisterPage";
|
import RegisterPage from "@/features/auth/RegisterPage";
|
||||||
import DetailPage from "@/features/detail/DetailPage";
|
import DetailPage from "@/features/detail/DetailPage";
|
||||||
|
import DeletePage from "@/features/library/DeletePage";
|
||||||
|
import DuplicatesPage from "@/features/library/DuplicatesPage";
|
||||||
import LibraryPage from "@/features/library/LibraryPage";
|
import LibraryPage from "@/features/library/LibraryPage";
|
||||||
import Md5Redirect from "@/features/library/Md5Redirect";
|
import Md5Redirect from "@/features/library/Md5Redirect";
|
||||||
import OnlinePage from "@/features/online/OnlinePage";
|
import OnlinePage from "@/features/online/OnlinePage";
|
||||||
import UploadPage from "@/features/upload/UploadPage";
|
import UploadPage from "@/features/upload/UploadPage";
|
||||||
|
import UsersPage from "@/features/users/UsersPage";
|
||||||
import { useAuth } from "@/store/auth";
|
import { useAuth } from "@/store/auth";
|
||||||
|
|
||||||
function RequireAuth({ children }: { children: ReactNode }) {
|
function RequireAuth({ children }: { children: ReactNode }) {
|
||||||
@@ -53,6 +56,30 @@ export default function App() {
|
|||||||
<Route path="/view/:md5" element={<Md5Redirect />} />
|
<Route path="/view/:md5" element={<Md5Redirect />} />
|
||||||
<Route path="/online" element={<OnlinePage />} />
|
<Route path="/online" element={<OnlinePage />} />
|
||||||
<Route path="/online/view/:postId" element={<PostRedirect />} />
|
<Route path="/online/view/:postId" element={<PostRedirect />} />
|
||||||
|
<Route
|
||||||
|
path="/duplicates"
|
||||||
|
element={
|
||||||
|
<RequireAuth>
|
||||||
|
<DuplicatesPage />
|
||||||
|
</RequireAuth>
|
||||||
|
}
|
||||||
|
/>
|
||||||
|
<Route
|
||||||
|
path="/delete"
|
||||||
|
element={
|
||||||
|
<RequireAuth>
|
||||||
|
<DeletePage />
|
||||||
|
</RequireAuth>
|
||||||
|
}
|
||||||
|
/>
|
||||||
|
<Route
|
||||||
|
path="/users"
|
||||||
|
element={
|
||||||
|
<RequireAuth>
|
||||||
|
<UsersPage />
|
||||||
|
</RequireAuth>
|
||||||
|
}
|
||||||
|
/>
|
||||||
<Route
|
<Route
|
||||||
path="/upload"
|
path="/upload"
|
||||||
element={
|
element={
|
||||||
|
|||||||
@@ -60,7 +60,16 @@ export function AppShell() {
|
|||||||
const navItems = [
|
const navItems = [
|
||||||
{ to: "/", label: "Library" },
|
{ to: "/", label: "Library" },
|
||||||
{ to: "/online", label: "Online" },
|
{ to: "/online", label: "Online" },
|
||||||
...(user?.can_upload ? [{ to: "/upload", label: "Upload" }] : []),
|
...(user?.can_upload
|
||||||
|
? [
|
||||||
|
{ to: "/upload", label: "Upload" },
|
||||||
|
{ to: "/duplicates", label: "Duplicates" },
|
||||||
|
{ to: "/delete", label: "Delete" },
|
||||||
|
]
|
||||||
|
: []),
|
||||||
|
...(user && (user.is_staff || user.is_superuser || user.role === "staff")
|
||||||
|
? [{ to: "/users", label: "Users" }]
|
||||||
|
: []),
|
||||||
];
|
];
|
||||||
|
|
||||||
const navigationType = useNavigationType();
|
const navigationType = useNavigationType();
|
||||||
@@ -120,9 +129,17 @@ export function AppShell() {
|
|||||||
title="Account settings"
|
title="Account settings"
|
||||||
className="flex items-center gap-2 transition hover:opacity-80"
|
className="flex items-center gap-2 transition hover:opacity-80"
|
||||||
>
|
>
|
||||||
<span className="flex h-7 w-7 items-center justify-center rounded-full bg-ctp-surface1 font-mono text-xs font-semibold text-ctp-lavender">
|
{user.avatar_url ? (
|
||||||
{user.username.slice(0, 1).toUpperCase()}
|
<img
|
||||||
</span>
|
src={user.avatar_url}
|
||||||
|
alt={user.username}
|
||||||
|
className="h-7 w-7 rounded-full border border-ctp-surface1 object-cover"
|
||||||
|
/>
|
||||||
|
) : (
|
||||||
|
<span className="flex h-7 w-7 items-center justify-center rounded-full bg-ctp-surface1 font-mono text-xs font-semibold text-ctp-lavender">
|
||||||
|
{user.username.slice(0, 1).toUpperCase()}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
<span className="hidden text-sm text-ctp-subtext1 sm:block">
|
<span className="hidden text-sm text-ctp-subtext1 sm:block">
|
||||||
{user.username}
|
{user.username}
|
||||||
</span>
|
</span>
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import {
|
import {
|
||||||
|
Copy,
|
||||||
FolderOpen,
|
FolderOpen,
|
||||||
Globe,
|
Globe,
|
||||||
LogIn,
|
LogIn,
|
||||||
@@ -6,7 +7,9 @@ import {
|
|||||||
Menu,
|
Menu,
|
||||||
Search,
|
Search,
|
||||||
Settings,
|
Settings,
|
||||||
|
Trash2,
|
||||||
Upload,
|
Upload,
|
||||||
|
Users,
|
||||||
type LucideIcon,
|
type LucideIcon,
|
||||||
} from "lucide-react";
|
} from "lucide-react";
|
||||||
import { useMemo, useState } from "react";
|
import { useMemo, useState } from "react";
|
||||||
@@ -59,6 +62,18 @@ function CommandPaletteDialog({ onClose }: { onClose: () => void }) {
|
|||||||
icon: Upload,
|
icon: Upload,
|
||||||
run: () => navigate("/upload"),
|
run: () => navigate("/upload"),
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
id: "duplicates",
|
||||||
|
label: "Find duplicates",
|
||||||
|
icon: Copy,
|
||||||
|
run: () => navigate("/duplicates"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "delete",
|
||||||
|
label: "Delete files & storage",
|
||||||
|
icon: Trash2,
|
||||||
|
run: () => navigate("/delete"),
|
||||||
|
},
|
||||||
]
|
]
|
||||||
: []),
|
: []),
|
||||||
{
|
{
|
||||||
@@ -94,6 +109,14 @@ function CommandPaletteDialog({ onClose }: { onClose: () => void }) {
|
|||||||
icon: Settings,
|
icon: Settings,
|
||||||
run: () => navigate("/account"),
|
run: () => navigate("/account"),
|
||||||
});
|
});
|
||||||
|
if (user.is_staff || user.is_superuser || user.role === "staff") {
|
||||||
|
list.push({
|
||||||
|
id: "users",
|
||||||
|
label: "Manage users",
|
||||||
|
icon: Users,
|
||||||
|
run: () => navigate("/users"),
|
||||||
|
});
|
||||||
|
}
|
||||||
list.push({
|
list.push({
|
||||||
id: "logout",
|
id: "logout",
|
||||||
label: `Log out (${user.username})`,
|
label: `Log out (${user.username})`,
|
||||||
|
|||||||
@@ -0,0 +1,271 @@
|
|||||||
|
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||||
|
import { useState } from "react";
|
||||||
|
import { Link } from "react-router-dom";
|
||||||
|
|
||||||
|
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
|
||||||
|
import { api, errorMessage } from "@/lib/api";
|
||||||
|
import { cn } from "@/lib/cn";
|
||||||
|
import { formatBytes } from "@/lib/format";
|
||||||
|
import type { MediaItem, Paginated, StorageInfo } from "@/lib/types";
|
||||||
|
|
||||||
|
function parseJIds(text: string): string[] {
|
||||||
|
return text
|
||||||
|
.split(/[\s,]+/)
|
||||||
|
.map((value) => value.trim().toUpperCase())
|
||||||
|
.filter((value) => /^J-\d+$/.test(value));
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function DeletePage() {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
const [jIdsInput, setJIdsInput] = useState("");
|
||||||
|
const [loadedIds, setLoadedIds] = useState<string[]>([]);
|
||||||
|
const [selected, setSelected] = useState<string[]>([]);
|
||||||
|
const [status, setStatus] = useState<string | null>(null);
|
||||||
|
|
||||||
|
const storageQuery = useQuery({
|
||||||
|
queryKey: ["storage"],
|
||||||
|
queryFn: () => api<StorageInfo>("/api/storage/"),
|
||||||
|
});
|
||||||
|
|
||||||
|
const filesQuery = useQuery({
|
||||||
|
queryKey: ["delete-files", loadedIds],
|
||||||
|
queryFn: () =>
|
||||||
|
api<Paginated<MediaItem>>(`/api/files/?j_ids=${loadedIds.join(",")}`),
|
||||||
|
enabled: loadedIds.length > 0,
|
||||||
|
});
|
||||||
|
|
||||||
|
const deleteMutation = useMutation({
|
||||||
|
mutationFn: (jIds: string[]) =>
|
||||||
|
api<{ deleted: string[]; errors: { j_id?: string; error: string }[] }>(
|
||||||
|
"/api/delete/",
|
||||||
|
{ method: "POST", json: { j_ids: jIds } },
|
||||||
|
),
|
||||||
|
onSuccess: (result) => {
|
||||||
|
setSelected([]);
|
||||||
|
setLoadedIds([]);
|
||||||
|
setStatus(
|
||||||
|
`Deleted ${result.deleted.length} item(s)${
|
||||||
|
result.errors.length > 0 ? ` · ${result.errors.length} failed` : ""
|
||||||
|
}.`,
|
||||||
|
);
|
||||||
|
void queryClient.invalidateQueries({ queryKey: ["storage"] });
|
||||||
|
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||||
|
void queryClient.invalidateQueries({ queryKey: ["duplicates-md5"] });
|
||||||
|
},
|
||||||
|
onError: (error) => setStatus(errorMessage(error)),
|
||||||
|
});
|
||||||
|
|
||||||
|
const tempMutation = useMutation({
|
||||||
|
mutationFn: () =>
|
||||||
|
api<{ removed: number }>("/api/temp/clear/", { method: "POST" }),
|
||||||
|
onSuccess: (result) => {
|
||||||
|
setStatus(`Removed ${result.removed} temp file(s).`);
|
||||||
|
void queryClient.invalidateQueries({ queryKey: ["storage"] });
|
||||||
|
},
|
||||||
|
onError: (error) => setStatus(errorMessage(error)),
|
||||||
|
});
|
||||||
|
|
||||||
|
function toggleSelected(jId: string) {
|
||||||
|
setSelected((current) =>
|
||||||
|
current.includes(jId)
|
||||||
|
? current.filter((value) => value !== jId)
|
||||||
|
: [...current, jId],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const storage = storageQuery.data;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="mx-auto flex w-full max-w-6xl flex-col gap-6">
|
||||||
|
<header>
|
||||||
|
<h1 className="text-lg font-semibold">Delete & storage</h1>
|
||||||
|
<p className="mt-1 text-sm text-ctp-overlay0">
|
||||||
|
Inspect disk usage and remove files from the library.
|
||||||
|
</p>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
{status ? (
|
||||||
|
<p className="rounded-md border border-ctp-surface1 bg-ctp-base px-3 py-2 text-xs text-ctp-subtext0">
|
||||||
|
{status}
|
||||||
|
</p>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
<section className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
|
||||||
|
{storageQuery.isPending ? (
|
||||||
|
<Spinner className="h-5 w-5" />
|
||||||
|
) : storage ? (
|
||||||
|
<>
|
||||||
|
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||||
|
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
|
||||||
|
Library size
|
||||||
|
</p>
|
||||||
|
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
|
||||||
|
{formatBytes(storage.watched_folder.library_size)}
|
||||||
|
</p>
|
||||||
|
<p className="mt-1 truncate font-mono text-[10px] text-ctp-overlay0" title={storage.watched_folder.path}>
|
||||||
|
{storage.watched_folder.path}
|
||||||
|
</p>
|
||||||
|
<p className="font-mono text-[10px] text-ctp-overlay0">
|
||||||
|
{storage.library_items} items
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||||
|
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
|
||||||
|
Disk
|
||||||
|
</p>
|
||||||
|
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
|
||||||
|
{formatBytes(storage.watched_folder.used)} /{" "}
|
||||||
|
{formatBytes(storage.watched_folder.total)}
|
||||||
|
</p>
|
||||||
|
<div className="mt-2 h-1.5 w-full overflow-hidden rounded-full bg-ctp-surface0">
|
||||||
|
<div
|
||||||
|
className={cn(
|
||||||
|
"h-full",
|
||||||
|
storage.watched_folder.percent_used >= 95
|
||||||
|
? "bg-ctp-red"
|
||||||
|
: storage.watched_folder.percent_used >= 80
|
||||||
|
? "bg-ctp-peach"
|
||||||
|
: "bg-ctp-blue",
|
||||||
|
)}
|
||||||
|
style={{ width: `${storage.watched_folder.percent_used}%` }}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<p className="mt-1 font-mono text-[10px] text-ctp-overlay0">
|
||||||
|
{storage.watched_folder.percent_used}% used ·{" "}
|
||||||
|
{formatBytes(storage.watched_folder.free)} free
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||||
|
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
|
||||||
|
Media folder
|
||||||
|
</p>
|
||||||
|
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
|
||||||
|
{formatBytes(storage.media.size)}
|
||||||
|
</p>
|
||||||
|
<p className="mt-1 truncate font-mono text-[10px] text-ctp-overlay0" title={storage.media.path}>
|
||||||
|
{storage.media.path}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<div className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||||
|
<p className="text-[11px] uppercase tracking-wide text-ctp-overlay0">
|
||||||
|
Temp uploads
|
||||||
|
</p>
|
||||||
|
<p className="mt-1 font-mono text-sm text-ctp-subtext1">
|
||||||
|
{formatBytes(storage.temp.size)} · {storage.temp.files} file(s)
|
||||||
|
</p>
|
||||||
|
<Button
|
||||||
|
variant="secondary"
|
||||||
|
className="mt-2 px-2 py-1 text-xs"
|
||||||
|
disabled={tempMutation.isPending || storage.temp.files === 0}
|
||||||
|
onClick={() => tempMutation.mutate()}
|
||||||
|
>
|
||||||
|
{tempMutation.isPending ? "Clearing…" : "Clear temp folder"}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<EmptyState title="Could not load storage info" />
|
||||||
|
)}
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section className="flex flex-col gap-3">
|
||||||
|
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||||
|
Delete files by J-ID
|
||||||
|
</h2>
|
||||||
|
<textarea
|
||||||
|
className={cn(inputClass, "min-h-20 resize-y font-mono")}
|
||||||
|
placeholder="J-12, J-34 J-56 comma, space or newline separated"
|
||||||
|
value={jIdsInput}
|
||||||
|
onChange={(event) => setJIdsInput(event.target.value)}
|
||||||
|
/>
|
||||||
|
<div className="flex items-center gap-3">
|
||||||
|
<Button
|
||||||
|
variant="secondary"
|
||||||
|
onClick={() => {
|
||||||
|
setStatus(null);
|
||||||
|
setSelected([]);
|
||||||
|
setLoadedIds(parseJIds(jIdsInput));
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Load files
|
||||||
|
</Button>
|
||||||
|
{selected.length > 0 ? (
|
||||||
|
<Button
|
||||||
|
variant="danger"
|
||||||
|
disabled={deleteMutation.isPending}
|
||||||
|
onClick={() => deleteMutation.mutate(selected)}
|
||||||
|
>
|
||||||
|
{deleteMutation.isPending
|
||||||
|
? "Deleting…"
|
||||||
|
: `Delete ${selected.length} selected`}
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{loadedIds.length === 0 ? (
|
||||||
|
<p className="text-xs text-ctp-overlay0">
|
||||||
|
Enter one or more J-IDs to preview them here.
|
||||||
|
</p>
|
||||||
|
) : filesQuery.isPending ? (
|
||||||
|
<Spinner className="h-5 w-5" />
|
||||||
|
) : (filesQuery.data?.results.length ?? 0) === 0 ? (
|
||||||
|
<EmptyState title="No matching library items" />
|
||||||
|
) : (
|
||||||
|
<div className="grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
|
||||||
|
{filesQuery.data?.results.map((item) => {
|
||||||
|
const isSelected = selected.includes(item.j_id);
|
||||||
|
return (
|
||||||
|
<div
|
||||||
|
key={item.j_id}
|
||||||
|
className={cn(
|
||||||
|
"overflow-hidden rounded-lg border bg-ctp-base",
|
||||||
|
isSelected ? "border-ctp-mauve" : "border-ctp-surface0",
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => toggleSelected(item.j_id)}
|
||||||
|
className="block w-full text-left"
|
||||||
|
>
|
||||||
|
<div className="relative aspect-square overflow-hidden bg-ctp-mantle">
|
||||||
|
<img
|
||||||
|
src={`/api/files/${item.j_id}/thumbnail/`}
|
||||||
|
alt={item.filename}
|
||||||
|
loading="lazy"
|
||||||
|
className="h-full w-full object-cover"
|
||||||
|
/>
|
||||||
|
<span
|
||||||
|
className={cn(
|
||||||
|
"absolute left-2 top-2 flex h-5 w-5 items-center justify-center rounded border text-[11px]",
|
||||||
|
isSelected
|
||||||
|
? "border-ctp-mauve bg-ctp-mauve text-ctp-crust"
|
||||||
|
: "border-ctp-surface1 bg-ctp-crust/70 text-transparent",
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
✓
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
<div className="px-2 py-1.5">
|
||||||
|
<Link
|
||||||
|
to={`/detail/${item.j_id}`}
|
||||||
|
className="font-mono text-[10px] text-ctp-blue hover:underline"
|
||||||
|
>
|
||||||
|
{item.j_id}
|
||||||
|
</Link>
|
||||||
|
<p className="truncate font-mono text-[11px] text-ctp-subtext1" title={item.filename}>
|
||||||
|
{item.filename}
|
||||||
|
</p>
|
||||||
|
<p className="font-mono text-[10px] text-ctp-overlay0">
|
||||||
|
{formatBytes(item.size)}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</section>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,468 @@
|
|||||||
|
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||||
|
import { useMemo, useState } from "react";
|
||||||
|
import { Link } from "react-router-dom";
|
||||||
|
|
||||||
|
import { Button, Spinner, inputClass } from "@/components/ui";
|
||||||
|
import { api, errorMessage } from "@/lib/api";
|
||||||
|
import { cn } from "@/lib/cn";
|
||||||
|
import { formatBytes } from "@/lib/format";
|
||||||
|
import type {
|
||||||
|
DuplicateEntry,
|
||||||
|
VisualGroupsResponse,
|
||||||
|
VisualMatchesResponse,
|
||||||
|
} from "@/lib/types";
|
||||||
|
|
||||||
|
const ALGORITHMS = [
|
||||||
|
{ value: "ahash", label: "aHash" },
|
||||||
|
{ value: "dhash", label: "dHash" },
|
||||||
|
{ value: "phash", label: "pHash" },
|
||||||
|
{ value: "whash", label: "wHash" },
|
||||||
|
];
|
||||||
|
|
||||||
|
interface DeleteResult {
|
||||||
|
deleted: string[];
|
||||||
|
errors: { j_id?: string; location?: number; error: string }[];
|
||||||
|
}
|
||||||
|
|
||||||
|
function MemberCard({
|
||||||
|
entry,
|
||||||
|
selected,
|
||||||
|
onToggle,
|
||||||
|
}: {
|
||||||
|
entry: DuplicateEntry;
|
||||||
|
selected: boolean;
|
||||||
|
onToggle: () => void;
|
||||||
|
}) {
|
||||||
|
return (
|
||||||
|
<div
|
||||||
|
className={cn(
|
||||||
|
"overflow-hidden rounded-lg border bg-ctp-base",
|
||||||
|
selected ? "border-ctp-mauve" : "border-ctp-surface0",
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
<button type="button" onClick={onToggle} className="block w-full text-left">
|
||||||
|
<div className="relative aspect-square overflow-hidden bg-ctp-mantle">
|
||||||
|
<img
|
||||||
|
src={`/api/files/${entry.j_id}/thumbnail/`}
|
||||||
|
alt={entry.j_id}
|
||||||
|
loading="lazy"
|
||||||
|
className="h-full w-full object-cover"
|
||||||
|
/>
|
||||||
|
<span
|
||||||
|
className={cn(
|
||||||
|
"absolute left-2 top-2 flex h-5 w-5 items-center justify-center rounded border text-[11px]",
|
||||||
|
selected
|
||||||
|
? "border-ctp-mauve bg-ctp-mauve text-ctp-crust"
|
||||||
|
: "border-ctp-surface1 bg-ctp-crust/70 text-transparent",
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
✓
|
||||||
|
</span>
|
||||||
|
{entry.similarity !== undefined ? (
|
||||||
|
<span className="absolute right-2 top-2 rounded-sm bg-ctp-crust/80 px-1.5 py-0.5 font-mono text-[10px] text-ctp-peach">
|
||||||
|
{entry.similarity}%
|
||||||
|
</span>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
<div className="px-2 py-1.5">
|
||||||
|
<Link
|
||||||
|
to={`/detail/${entry.j_id}`}
|
||||||
|
className="font-mono text-[10px] text-ctp-blue hover:underline"
|
||||||
|
>
|
||||||
|
{entry.j_id}
|
||||||
|
</Link>
|
||||||
|
<p
|
||||||
|
className="truncate font-mono text-[11px] text-ctp-subtext1"
|
||||||
|
title={entry.filename}
|
||||||
|
>
|
||||||
|
{entry.filename}
|
||||||
|
</p>
|
||||||
|
<p className="font-mono text-[10px] text-ctp-overlay0">
|
||||||
|
{formatBytes(entry.size)}
|
||||||
|
{entry.location_count > 1 ? ` · ${entry.location_count} copies` : ""}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function DuplicatesPage() {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
const [threshold, setThreshold] = useState(80);
|
||||||
|
const [algorithms, setAlgorithms] = useState<string[]>(
|
||||||
|
ALGORITHMS.map((algorithm) => algorithm.value),
|
||||||
|
);
|
||||||
|
const [groupsPage, setGroupsPage] = useState(1);
|
||||||
|
const [groupsEnabled, setGroupsEnabled] = useState(false);
|
||||||
|
const [selected, setSelected] = useState<string[]>([]);
|
||||||
|
const [dismissed, setDismissed] = useState<string[]>([]);
|
||||||
|
const [similarTarget, setSimilarTarget] = useState("");
|
||||||
|
const [status, setStatus] = useState<string | null>(null);
|
||||||
|
|
||||||
|
const exactQuery = useQuery({
|
||||||
|
queryKey: ["duplicates-md5"],
|
||||||
|
queryFn: () =>
|
||||||
|
api<{ count: number; groups: DuplicateEntry[] }>("/api/duplicates/md5/"),
|
||||||
|
});
|
||||||
|
|
||||||
|
const groupsQuery = useQuery({
|
||||||
|
queryKey: [
|
||||||
|
"duplicate-groups",
|
||||||
|
groupsEnabled,
|
||||||
|
threshold,
|
||||||
|
algorithms,
|
||||||
|
groupsPage,
|
||||||
|
],
|
||||||
|
queryFn: () =>
|
||||||
|
api<VisualGroupsResponse>("/api/duplicates/visual_groups/", {
|
||||||
|
method: "POST",
|
||||||
|
json: { threshold, algorithms, page: groupsPage },
|
||||||
|
}),
|
||||||
|
enabled: groupsEnabled && algorithms.length > 0,
|
||||||
|
});
|
||||||
|
|
||||||
|
const similarMutation = useMutation({
|
||||||
|
mutationFn: (jId: string) =>
|
||||||
|
api<VisualMatchesResponse>("/api/duplicates/visual/", {
|
||||||
|
method: "POST",
|
||||||
|
json: { j_id: jId, threshold, algorithms },
|
||||||
|
}),
|
||||||
|
onError: (error) => setStatus(errorMessage(error)),
|
||||||
|
});
|
||||||
|
|
||||||
|
const deleteMutation = useMutation({
|
||||||
|
mutationFn: (payload: { j_ids?: string[]; location_ids?: number[] }) =>
|
||||||
|
api<DeleteResult>("/api/delete/", { method: "POST", json: payload }),
|
||||||
|
onSuccess: (result) => {
|
||||||
|
setSelected([]);
|
||||||
|
setStatus(
|
||||||
|
`Deleted ${result.deleted.length} entr${
|
||||||
|
result.deleted.length === 1 ? "y" : "ies"
|
||||||
|
}${result.errors.length > 0 ? ` · ${result.errors.length} failed` : ""}.`,
|
||||||
|
);
|
||||||
|
void queryClient.invalidateQueries({ queryKey: ["duplicates-md5"] });
|
||||||
|
void queryClient.invalidateQueries({ queryKey: ["duplicate-groups"] });
|
||||||
|
void queryClient.invalidateQueries({ queryKey: ["files"] });
|
||||||
|
},
|
||||||
|
onError: (error) => setStatus(errorMessage(error)),
|
||||||
|
});
|
||||||
|
|
||||||
|
function toggleSelected(jId: string) {
|
||||||
|
setSelected((current) =>
|
||||||
|
current.includes(jId)
|
||||||
|
? current.filter((value) => value !== jId)
|
||||||
|
: [...current, jId],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function toggleAlgorithm(value: string) {
|
||||||
|
setGroupsPage(1);
|
||||||
|
setAlgorithms((current) =>
|
||||||
|
current.includes(value)
|
||||||
|
? current.filter((algorithm) => algorithm !== value)
|
||||||
|
: [...current, value],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const groups = useMemo(
|
||||||
|
() =>
|
||||||
|
(groupsQuery.data?.groups ?? []).filter(
|
||||||
|
(group) =>
|
||||||
|
!dismissed.includes(
|
||||||
|
group.members
|
||||||
|
.map((member) => member.j_id)
|
||||||
|
.sort()
|
||||||
|
.join("|"),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
[groupsQuery.data, dismissed],
|
||||||
|
);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="mx-auto flex w-full max-w-6xl flex-col gap-8">
|
||||||
|
<header>
|
||||||
|
<h1 className="text-lg font-semibold">Duplicates</h1>
|
||||||
|
<p className="mt-1 text-sm text-ctp-overlay0">
|
||||||
|
Clean up exact copies and visually similar files.
|
||||||
|
</p>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
{status ? (
|
||||||
|
<p className="rounded-md border border-ctp-surface1 bg-ctp-base px-3 py-2 text-xs text-ctp-subtext0">
|
||||||
|
{status}
|
||||||
|
</p>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
<section className="flex flex-col gap-3">
|
||||||
|
<header className="flex items-center justify-between gap-2">
|
||||||
|
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||||
|
Exact duplicates
|
||||||
|
</h2>
|
||||||
|
<span className="font-mono text-[11px] text-ctp-overlay0">
|
||||||
|
{exactQuery.data?.count ?? 0} item(s)
|
||||||
|
</span>
|
||||||
|
</header>
|
||||||
|
{exactQuery.isPending ? (
|
||||||
|
<Spinner className="h-5 w-5" />
|
||||||
|
) : (exactQuery.data?.count ?? 0) === 0 ? (
|
||||||
|
<p className="rounded-lg border border-dashed border-ctp-surface1 px-3 py-6 text-center text-xs text-ctp-overlay0">
|
||||||
|
No items exist at multiple paths.
|
||||||
|
</p>
|
||||||
|
) : (
|
||||||
|
<div className="flex flex-col gap-3">
|
||||||
|
{exactQuery.data?.groups.map((group) => (
|
||||||
|
<div
|
||||||
|
key={group.j_id}
|
||||||
|
className="flex flex-col gap-3 rounded-lg border border-ctp-surface0 bg-ctp-base p-3 lg:flex-row"
|
||||||
|
>
|
||||||
|
<div className="w-32 shrink-0">
|
||||||
|
<div className="aspect-square overflow-hidden rounded-md bg-ctp-mantle">
|
||||||
|
<img
|
||||||
|
src={`/api/files/${group.j_id}/thumbnail/`}
|
||||||
|
alt={group.j_id}
|
||||||
|
loading="lazy"
|
||||||
|
className="h-full w-full object-cover"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<Link
|
||||||
|
to={`/detail/${group.j_id}`}
|
||||||
|
className="font-mono text-[10px] text-ctp-blue hover:underline"
|
||||||
|
>
|
||||||
|
{group.j_id}
|
||||||
|
</Link>
|
||||||
|
</div>
|
||||||
|
<div className="min-w-0 flex-1">
|
||||||
|
<p className="truncate font-mono text-xs text-ctp-subtext1">
|
||||||
|
{group.filename}
|
||||||
|
</p>
|
||||||
|
<ul className="mt-2 flex flex-col gap-1.5">
|
||||||
|
{(group.locations ?? []).map((location) => (
|
||||||
|
<li
|
||||||
|
key={location.id}
|
||||||
|
className="flex items-center justify-between gap-3 rounded-md border border-ctp-surface0 px-2 py-1"
|
||||||
|
>
|
||||||
|
<span className="truncate font-mono text-[11px] text-ctp-overlay1" title={location.rel_path}>
|
||||||
|
{location.rel_path}
|
||||||
|
</span>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() =>
|
||||||
|
deleteMutation.mutate({ location_ids: [location.id] })
|
||||||
|
}
|
||||||
|
className="shrink-0 rounded-md px-1.5 py-0.5 font-mono text-[10px] text-ctp-red transition hover:bg-ctp-red/15"
|
||||||
|
>
|
||||||
|
delete copy
|
||||||
|
</button>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
<div className="shrink-0">
|
||||||
|
<Button
|
||||||
|
variant="danger"
|
||||||
|
className="px-2 py-1 text-xs"
|
||||||
|
onClick={() => deleteMutation.mutate({ j_ids: [group.j_id] })}
|
||||||
|
>
|
||||||
|
Delete item
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section className="flex flex-col gap-3">
|
||||||
|
<h2 className="text-sm font-semibold text-ctp-subtext1">
|
||||||
|
Visual similarity
|
||||||
|
</h2>
|
||||||
|
<div className="flex flex-col gap-3 rounded-lg border border-ctp-surface0 bg-ctp-base p-3">
|
||||||
|
<div className="flex flex-wrap items-center gap-4">
|
||||||
|
<label className="flex flex-1 items-center gap-3 text-xs text-ctp-subtext0">
|
||||||
|
Similarity {threshold}%
|
||||||
|
<input
|
||||||
|
type="range"
|
||||||
|
min={50}
|
||||||
|
max={100}
|
||||||
|
value={threshold}
|
||||||
|
onChange={(event) => {
|
||||||
|
setGroupsPage(1);
|
||||||
|
setThreshold(Number(event.target.value));
|
||||||
|
}}
|
||||||
|
className="flex-1"
|
||||||
|
/>
|
||||||
|
</label>
|
||||||
|
<div className="flex flex-wrap items-center gap-2">
|
||||||
|
{ALGORITHMS.map((algorithm) => (
|
||||||
|
<label
|
||||||
|
key={algorithm.value}
|
||||||
|
className={cn(
|
||||||
|
"cursor-pointer rounded-full border px-2.5 py-1 font-mono text-[11px] transition",
|
||||||
|
algorithms.includes(algorithm.value)
|
||||||
|
? "border-ctp-mauve/40 bg-ctp-mauve/15 text-ctp-mauve"
|
||||||
|
: "border-ctp-surface1 text-ctp-subtext0 hover:border-ctp-surface2",
|
||||||
|
)}
|
||||||
|
>
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
className="hidden"
|
||||||
|
checked={algorithms.includes(algorithm.value)}
|
||||||
|
onChange={() => toggleAlgorithm(algorithm.value)}
|
||||||
|
/>
|
||||||
|
{algorithm.label}
|
||||||
|
</label>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
<Button
|
||||||
|
onClick={() => {
|
||||||
|
setGroupsEnabled(true);
|
||||||
|
void groupsQuery.refetch();
|
||||||
|
}}
|
||||||
|
disabled={algorithms.length === 0 || groupsQuery.isFetching}
|
||||||
|
>
|
||||||
|
{groupsQuery.isFetching ? "Comparing…" : "Find groups"}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex flex-wrap items-end gap-2 border-t border-ctp-surface0 pt-3">
|
||||||
|
<label className="flex flex-col gap-1 text-xs text-ctp-subtext0">
|
||||||
|
Find files similar to
|
||||||
|
<input
|
||||||
|
className={cn(inputClass, "w-32 font-mono")}
|
||||||
|
placeholder="J-123"
|
||||||
|
value={similarTarget}
|
||||||
|
onChange={(event) => setSimilarTarget(event.target.value)}
|
||||||
|
/>
|
||||||
|
</label>
|
||||||
|
<Button
|
||||||
|
variant="secondary"
|
||||||
|
onClick={() => similarMutation.mutate(similarTarget.trim())}
|
||||||
|
disabled={!similarTarget.trim() || similarMutation.isPending}
|
||||||
|
>
|
||||||
|
{similarMutation.isPending ? "Searching…" : "Search"}
|
||||||
|
</Button>
|
||||||
|
{similarMutation.isError ? (
|
||||||
|
<p className="text-xs text-ctp-red">
|
||||||
|
{errorMessage(similarMutation.error)}
|
||||||
|
</p>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{similarMutation.data ? (
|
||||||
|
similarMutation.data.count === 0 ? (
|
||||||
|
<p className="text-xs text-ctp-overlay0">
|
||||||
|
No matches at {similarMutation.data.threshold}% similarity.
|
||||||
|
</p>
|
||||||
|
) : (
|
||||||
|
<div className="grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
|
||||||
|
{similarMutation.data.matches.map((entry) => (
|
||||||
|
<MemberCard
|
||||||
|
key={entry.j_id}
|
||||||
|
entry={entry}
|
||||||
|
selected={selected.includes(entry.j_id)}
|
||||||
|
onToggle={() => toggleSelected(entry.j_id)}
|
||||||
|
/>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{groupsEnabled ? (
|
||||||
|
groupsQuery.isPending ? (
|
||||||
|
<Spinner className="h-5 w-5" />
|
||||||
|
) : groups.length === 0 ? (
|
||||||
|
<p className="rounded-lg border border-dashed border-ctp-surface1 px-3 py-6 text-center text-xs text-ctp-overlay0">
|
||||||
|
No groups at {threshold}% similarity.
|
||||||
|
</p>
|
||||||
|
) : (
|
||||||
|
<div className="flex flex-col gap-4">
|
||||||
|
{groups.map((group) => {
|
||||||
|
const groupKey = group.members
|
||||||
|
.map((member) => member.j_id)
|
||||||
|
.sort()
|
||||||
|
.join("|");
|
||||||
|
return (
|
||||||
|
<div
|
||||||
|
key={groupKey}
|
||||||
|
className="rounded-lg border border-ctp-surface0 bg-ctp-base p-3"
|
||||||
|
>
|
||||||
|
<header className="flex items-center justify-between gap-2">
|
||||||
|
<span className="font-mono text-[11px] text-ctp-overlay0">
|
||||||
|
{group.size} similar items
|
||||||
|
</span>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => setDismissed((current) => [...current, groupKey])}
|
||||||
|
className="rounded-md px-1.5 py-0.5 font-mono text-[10px] text-ctp-overlay0 transition hover:bg-ctp-surface0 hover:text-ctp-text"
|
||||||
|
>
|
||||||
|
dismiss
|
||||||
|
</button>
|
||||||
|
</header>
|
||||||
|
<div className="mt-3 grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
|
||||||
|
{group.members.map((member) => (
|
||||||
|
<MemberCard
|
||||||
|
key={member.j_id}
|
||||||
|
entry={member}
|
||||||
|
selected={selected.includes(member.j_id)}
|
||||||
|
onToggle={() => toggleSelected(member.j_id)}
|
||||||
|
/>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
|
||||||
|
<div className="flex items-center justify-between gap-3">
|
||||||
|
<Button
|
||||||
|
variant="secondary"
|
||||||
|
disabled={groupsPage <= 1 || groupsQuery.isFetching}
|
||||||
|
onClick={() => setGroupsPage((page) => Math.max(1, page - 1))}
|
||||||
|
>
|
||||||
|
Previous
|
||||||
|
</Button>
|
||||||
|
<span className="font-mono text-xs text-ctp-overlay0">
|
||||||
|
Page {groupsQuery.data?.page ?? groupsPage} ·{" "}
|
||||||
|
{groupsQuery.data?.count ?? 0} group(s)
|
||||||
|
</span>
|
||||||
|
<Button
|
||||||
|
variant="secondary"
|
||||||
|
disabled={!groupsQuery.data?.has_next || groupsQuery.isFetching}
|
||||||
|
onClick={() => setGroupsPage((page) => page + 1)}
|
||||||
|
>
|
||||||
|
Next
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{selected.length > 0 ? (
|
||||||
|
<div className="sticky bottom-16 z-20 flex items-center justify-between gap-3 rounded-lg border border-ctp-mauve/40 bg-ctp-surface0 px-3 py-2 shadow-[0_4px_12px_rgba(17,17,27,0.6)]">
|
||||||
|
<span className="text-xs text-ctp-subtext1">
|
||||||
|
{selected.length} selected
|
||||||
|
</span>
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<Button
|
||||||
|
variant="ghost"
|
||||||
|
className="px-2 py-1 text-xs"
|
||||||
|
onClick={() => setSelected([])}
|
||||||
|
>
|
||||||
|
Clear
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
variant="danger"
|
||||||
|
className="px-2 py-1 text-xs"
|
||||||
|
disabled={deleteMutation.isPending}
|
||||||
|
onClick={() => deleteMutation.mutate({ j_ids: selected })}
|
||||||
|
>
|
||||||
|
{deleteMutation.isPending ? "Deleting…" : "Delete selected"}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
</section>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -137,9 +137,11 @@ function TempCard({
|
|||||||
No preview
|
No preview
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
{temp.status === "visual_match" && temp.iqdb_data?.length ? (
|
{temp.status === "visual_match" &&
|
||||||
|
(temp.iqdb_data?.length ?? 0) + (temp.visual_matches?.length ?? 0) > 0 ? (
|
||||||
<span className="absolute left-2 top-2 rounded-full bg-ctp-peach/90 px-1.5 py-0.5 font-mono text-[10px] font-medium text-ctp-crust">
|
<span className="absolute left-2 top-2 rounded-full bg-ctp-peach/90 px-1.5 py-0.5 font-mono text-[10px] font-medium text-ctp-crust">
|
||||||
{temp.iqdb_data.length} similar
|
{(temp.iqdb_data?.length ?? 0) + (temp.visual_matches?.length ?? 0)}{" "}
|
||||||
|
similar
|
||||||
</span>
|
</span>
|
||||||
) : null}
|
) : null}
|
||||||
</div>
|
</div>
|
||||||
@@ -282,6 +284,13 @@ function MetadataModal({
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function discardUpload() {
|
||||||
|
void api(`/api/uploads/${temp.temp_id}/`, { method: "DELETE" }).then(() => {
|
||||||
|
void queryClient.invalidateQueries({ queryKey: ["uploads"] });
|
||||||
|
onClose();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div
|
<div
|
||||||
className="fixed inset-0 z-[70] flex items-start justify-center overflow-y-auto bg-ctp-crust/70 px-4 py-10 backdrop-blur-sm"
|
className="fixed inset-0 z-[70] flex items-start justify-center overflow-y-auto bg-ctp-crust/70 px-4 py-10 backdrop-blur-sm"
|
||||||
@@ -364,6 +373,54 @@ function MetadataModal({
|
|||||||
|
|
||||||
{tab === "link" ? (
|
{tab === "link" ? (
|
||||||
<div className="flex flex-col gap-3">
|
<div className="flex flex-col gap-3">
|
||||||
|
{temp.visual_matches && temp.visual_matches.length > 0 ? (
|
||||||
|
<div>
|
||||||
|
<p className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||||
|
Already in your library
|
||||||
|
</p>
|
||||||
|
<div className="mt-2 flex flex-col gap-2">
|
||||||
|
{temp.visual_matches.map((match) => (
|
||||||
|
<div
|
||||||
|
key={match.j_id}
|
||||||
|
className="flex items-center gap-3 rounded-md border border-ctp-surface0 bg-ctp-surface0/40 p-2"
|
||||||
|
>
|
||||||
|
<img
|
||||||
|
src={`/api/files/${match.j_id}/thumbnail/`}
|
||||||
|
alt={match.j_id}
|
||||||
|
loading="lazy"
|
||||||
|
className="h-12 w-12 shrink-0 rounded object-cover"
|
||||||
|
/>
|
||||||
|
<span className="min-w-0 flex-1">
|
||||||
|
<Link
|
||||||
|
to={`/detail/${match.j_id}`}
|
||||||
|
className="font-mono text-[11px] text-ctp-blue hover:underline"
|
||||||
|
>
|
||||||
|
{match.j_id}
|
||||||
|
</Link>
|
||||||
|
<span className="block truncate font-mono text-[11px] text-ctp-subtext0" title={match.filename}>
|
||||||
|
{match.filename}
|
||||||
|
</span>
|
||||||
|
</span>
|
||||||
|
<span className="shrink-0 font-mono text-[11px] text-ctp-peach">
|
||||||
|
{match.similarity}%
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
<p className="mt-2 text-[11px] text-ctp-overlay0">
|
||||||
|
This upload looks like it is already in the library —
|
||||||
|
you can discard it below.
|
||||||
|
</p>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={discardUpload}
|
||||||
|
className="mt-2 rounded-md px-1.5 py-0.5 font-mono text-[10px] text-ctp-red transition hover:bg-ctp-red/15"
|
||||||
|
>
|
||||||
|
discard this upload
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
{temp.iqdb_data && temp.iqdb_data.length > 0 ? (
|
{temp.iqdb_data && temp.iqdb_data.length > 0 ? (
|
||||||
<div>
|
<div>
|
||||||
<p className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
<p className="text-xs font-medium uppercase tracking-wide text-ctp-overlay1">
|
||||||
|
|||||||
@@ -0,0 +1,180 @@
|
|||||||
|
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||||
|
import { useState } from "react";
|
||||||
|
|
||||||
|
import { Button, EmptyState, Spinner, inputClass } from "@/components/ui";
|
||||||
|
import { api, errorMessage } from "@/lib/api";
|
||||||
|
import { cn } from "@/lib/cn";
|
||||||
|
import { formatDate } from "@/lib/format";
|
||||||
|
import type { AdminUser, Paginated } from "@/lib/types";
|
||||||
|
import { useAuth } from "@/store/auth";
|
||||||
|
|
||||||
|
const ROLE_OPTIONS = [
|
||||||
|
{ value: "user", label: "User" },
|
||||||
|
{ value: "uploader", label: "Uploader" },
|
||||||
|
{ value: "staff", label: "Staff" },
|
||||||
|
];
|
||||||
|
|
||||||
|
function UserRow({ profile }: { profile: AdminUser }) {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
const [avatarJId, setAvatarJId] = useState(profile.avatar_j_id ?? "");
|
||||||
|
const [message, setMessage] = useState<string | null>(null);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
|
||||||
|
const mutation = useMutation({
|
||||||
|
mutationFn: (payload: Record<string, string | null>) =>
|
||||||
|
api<AdminUser>(`/api/users/${profile.id}/`, {
|
||||||
|
method: "PATCH",
|
||||||
|
json: payload,
|
||||||
|
}),
|
||||||
|
onSuccess: (updated) => {
|
||||||
|
setMessage("Saved.");
|
||||||
|
setError(null);
|
||||||
|
setAvatarJId(updated.avatar_j_id ?? "");
|
||||||
|
void queryClient.invalidateQueries({ queryKey: ["users"] });
|
||||||
|
},
|
||||||
|
onError: (err) => {
|
||||||
|
setError(errorMessage(err));
|
||||||
|
setMessage(null);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="flex flex-col gap-3 rounded-lg border border-ctp-surface0 bg-ctp-base p-3 lg:flex-row lg:items-center">
|
||||||
|
<div className="flex min-w-0 flex-1 items-center gap-3">
|
||||||
|
{profile.avatar_url ? (
|
||||||
|
<img
|
||||||
|
src={profile.avatar_url}
|
||||||
|
alt={profile.username}
|
||||||
|
className="h-10 w-10 shrink-0 rounded-full border border-ctp-surface1 object-cover"
|
||||||
|
/>
|
||||||
|
) : (
|
||||||
|
<span className="flex h-10 w-10 shrink-0 items-center justify-center rounded-full bg-ctp-surface1 font-mono text-sm font-semibold text-ctp-lavender">
|
||||||
|
{profile.username.slice(0, 1).toUpperCase()}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
<div className="min-w-0">
|
||||||
|
<p className="flex flex-wrap items-center gap-2 text-sm text-ctp-subtext1">
|
||||||
|
<span className="font-medium">{profile.username}</span>
|
||||||
|
{profile.is_superuser ? (
|
||||||
|
<span className="rounded-full border border-ctp-red/40 bg-ctp-red/10 px-2 py-0.5 font-mono text-[10px] text-ctp-red">
|
||||||
|
admin
|
||||||
|
</span>
|
||||||
|
) : null}
|
||||||
|
{profile.is_staff && !profile.is_superuser ? (
|
||||||
|
<span className="rounded-full border border-ctp-surface1 px-2 py-0.5 font-mono text-[10px] text-ctp-overlay0">
|
||||||
|
django staff
|
||||||
|
</span>
|
||||||
|
) : null}
|
||||||
|
</p>
|
||||||
|
<p className="truncate font-mono text-[11px] text-ctp-overlay0">
|
||||||
|
{profile.email || "no email"} · joined{" "}
|
||||||
|
{formatDate(profile.date_joined)} · {profile.uploads_count} upload
|
||||||
|
{profile.uploads_count === 1 ? "" : "s"}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex flex-wrap items-center gap-2">
|
||||||
|
<select
|
||||||
|
className={cn(inputClass, "w-32")}
|
||||||
|
value={profile.role}
|
||||||
|
onChange={(event) =>
|
||||||
|
mutation.mutate({ role: event.target.value })
|
||||||
|
}
|
||||||
|
disabled={mutation.isPending}
|
||||||
|
title="Role"
|
||||||
|
>
|
||||||
|
{ROLE_OPTIONS.map((option) => (
|
||||||
|
<option key={option.value} value={option.value}>
|
||||||
|
{option.label}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
<input
|
||||||
|
className={cn(inputClass, "w-28 font-mono")}
|
||||||
|
placeholder="J-ID"
|
||||||
|
value={avatarJId}
|
||||||
|
onChange={(event) => setAvatarJId(event.target.value)}
|
||||||
|
title="Avatar library item"
|
||||||
|
/>
|
||||||
|
<Button
|
||||||
|
variant="secondary"
|
||||||
|
disabled={mutation.isPending}
|
||||||
|
onClick={() => mutation.mutate({ avatar_j_id: avatarJId.trim() })}
|
||||||
|
>
|
||||||
|
Save avatar
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{message ? (
|
||||||
|
<p className="text-xs text-ctp-green lg:w-20">{message}</p>
|
||||||
|
) : null}
|
||||||
|
{error ? (
|
||||||
|
<p className="text-xs text-ctp-red lg:w-40">{error}</p>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function UsersPage() {
|
||||||
|
const user = useAuth((state) => state.user);
|
||||||
|
const [search, setSearch] = useState("");
|
||||||
|
|
||||||
|
const isStaff = Boolean(
|
||||||
|
user &&
|
||||||
|
(user.is_staff || user.is_superuser || user.role === "staff"),
|
||||||
|
);
|
||||||
|
|
||||||
|
const query = useQuery({
|
||||||
|
queryKey: ["users", search],
|
||||||
|
queryFn: () =>
|
||||||
|
api<Paginated<AdminUser>>(
|
||||||
|
`/api/users/${search ? `?search=${encodeURIComponent(search)}` : ""}`,
|
||||||
|
),
|
||||||
|
enabled: isStaff,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!isStaff) {
|
||||||
|
return (
|
||||||
|
<EmptyState
|
||||||
|
title="Staff only"
|
||||||
|
description="This page is limited to staff accounts."
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="mx-auto flex w-full max-w-5xl flex-col gap-4">
|
||||||
|
<header className="flex flex-wrap items-end justify-between gap-3">
|
||||||
|
<div>
|
||||||
|
<h1 className="text-lg font-semibold">Users</h1>
|
||||||
|
<p className="mt-1 text-sm text-ctp-overlay0">
|
||||||
|
Manage roles and set profile pictures from library J-IDs.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<input
|
||||||
|
className={cn(inputClass, "w-56")}
|
||||||
|
placeholder="Search username…"
|
||||||
|
value={search}
|
||||||
|
onChange={(event) => setSearch(event.target.value)}
|
||||||
|
/>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
{query.isPending ? (
|
||||||
|
<div className="flex justify-center py-16">
|
||||||
|
<Spinner className="h-6 w-6" />
|
||||||
|
</div>
|
||||||
|
) : query.isError || !query.data ? (
|
||||||
|
<EmptyState title="Could not load users" />
|
||||||
|
) : query.data.results.length === 0 ? (
|
||||||
|
<EmptyState title="No users found" />
|
||||||
|
) : (
|
||||||
|
<div className="flex flex-col gap-3">
|
||||||
|
{query.data.results.map((profile) => (
|
||||||
|
<UserRow key={profile.id} profile={profile} />
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -7,10 +7,70 @@ export interface User {
|
|||||||
date_joined: string;
|
date_joined: string;
|
||||||
role: "user" | "uploader" | "staff";
|
role: "user" | "uploader" | "staff";
|
||||||
can_upload: boolean;
|
can_upload: boolean;
|
||||||
|
avatar_j_id: string | null;
|
||||||
|
avatar_url: string | null;
|
||||||
e621_username: string;
|
e621_username: string;
|
||||||
e621_configured: boolean;
|
e621_configured: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface AdminUser {
|
||||||
|
id: number;
|
||||||
|
username: string;
|
||||||
|
email: string;
|
||||||
|
role: "user" | "uploader" | "staff";
|
||||||
|
is_staff: boolean;
|
||||||
|
is_superuser: boolean;
|
||||||
|
date_joined: string;
|
||||||
|
avatar_j_id: string | null;
|
||||||
|
avatar_url: string | null;
|
||||||
|
uploads_count: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DuplicateEntry {
|
||||||
|
j_id: string;
|
||||||
|
md5: string;
|
||||||
|
filename: string;
|
||||||
|
size: number;
|
||||||
|
rating: string;
|
||||||
|
location_count: number;
|
||||||
|
uploaded_by: string | null;
|
||||||
|
e621_post_id: number | null;
|
||||||
|
locations?: { id: number; rel_path: string }[];
|
||||||
|
similarity?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface VisualGroupsResponse {
|
||||||
|
count: number;
|
||||||
|
page: number;
|
||||||
|
per_page: number;
|
||||||
|
has_next: boolean;
|
||||||
|
threshold: number;
|
||||||
|
algorithms: string[];
|
||||||
|
groups: { size: number; members: DuplicateEntry[] }[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface VisualMatchesResponse {
|
||||||
|
target: DuplicateEntry;
|
||||||
|
threshold: number;
|
||||||
|
algorithms: string[];
|
||||||
|
count: number;
|
||||||
|
matches: DuplicateEntry[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface StorageInfo {
|
||||||
|
watched_folder: {
|
||||||
|
path: string;
|
||||||
|
total: number;
|
||||||
|
used: number;
|
||||||
|
free: number;
|
||||||
|
library_size: number;
|
||||||
|
percent_used: number;
|
||||||
|
};
|
||||||
|
media: { path: string; size: number };
|
||||||
|
temp: { path: string; size: number; files: number };
|
||||||
|
library_items: number;
|
||||||
|
}
|
||||||
|
|
||||||
export interface MediaLocation {
|
export interface MediaLocation {
|
||||||
id: number;
|
id: number;
|
||||||
path: string;
|
path: string;
|
||||||
@@ -148,6 +208,9 @@ export interface TempUpload {
|
|||||||
custom_tags: string[];
|
custom_tags: string[];
|
||||||
custom_notes: string;
|
custom_notes: string;
|
||||||
iqdb_data: E621IqdbCandidate[] | null;
|
iqdb_data: E621IqdbCandidate[] | null;
|
||||||
|
visual_matches:
|
||||||
|
| { j_id: string; filename: string; similarity: number }[]
|
||||||
|
| null;
|
||||||
library_j_id: string | null;
|
library_j_id: string | null;
|
||||||
file_url: string | null;
|
file_url: string | null;
|
||||||
preview_url: string | null;
|
preview_url: string | null;
|
||||||
|
|||||||
Reference in New Issue
Block a user