Duplicates, delete & storage, users page with J-ID avatars
Backend: - Perceptual hashes (aHash/dHash/pHash/wHash via imagehash, no imgdd) stored on items, computed on upload/download and by the new compute_visual_hashes command - Duplicates API: exact duplicates (multi-location items), visual matches for one item, union-find similarity groups with pagination - Delete API with ownership/staff checks, per-item and per-copy deletion, watched-folder path validation; storage overview and temp cleanup; file list accepts j_ids batches - Staged uploads are flagged visual_match with their library matches (threshold via VISUAL_MATCH_THRESHOLD) - Staff users API: list with upload counts, set role and avatar by J-ID; User.avatar FK with signed avatar URLs - Download threads close their DB connection and stale tasks are reaped, keeping behaviour Gunicorn-friendly Frontend: - /duplicates: exact duplicate groups with per-copy delete, visual similarity controls, search similar to a J-ID, paginated groups with selection, bulk delete and dismiss - /delete: storage cards, delete by J-ID with preview grid, temp cleanup - /users: staff directory with role selects and avatar J-ID inputs - Nav + command palette entries; top-bar avatar; upload cards and the metadata modal show library visual matches
This commit is contained in:
@@ -3,10 +3,12 @@
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from django.conf import settings
|
||||
from django.db import connection
|
||||
from django.utils import timezone
|
||||
|
||||
from . import services
|
||||
@@ -14,6 +16,27 @@ from .models import DownloadTask
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
STALE_AFTER = timedelta(minutes=30)
|
||||
|
||||
|
||||
def reap_stale_downloads():
|
||||
"""Mark tasks left hanging by a recycled worker as failed.
|
||||
|
||||
Gunicorn recycles workers (--max-requests, timeouts); a download thread
|
||||
dies with its worker, so long-stuck tasks are surfaced as errors instead
|
||||
of pretending to run forever.
|
||||
"""
|
||||
cutoff = timezone.now() - STALE_AFTER
|
||||
return DownloadTask.objects.filter(
|
||||
status__in=[DownloadTask.STATUS_PENDING, DownloadTask.STATUS_DOWNLOADING],
|
||||
updated_at__lt=cutoff,
|
||||
).update(
|
||||
status=DownloadTask.STATUS_ERROR,
|
||||
error="The worker restarted before this download finished.",
|
||||
speed=None,
|
||||
updated_at=timezone.now(),
|
||||
)
|
||||
|
||||
|
||||
def start_download_task(task_id):
|
||||
thread = threading.Thread(target=run_download_task, args=(task_id,), daemon=True)
|
||||
@@ -79,6 +102,7 @@ def run_download_task(task_id):
|
||||
)
|
||||
item, _, location, _ = services.index_file(destination, folder)
|
||||
services.rename_location_to_j_id(item, location)
|
||||
services.ensure_visual_hashes(item)
|
||||
|
||||
update_fields = []
|
||||
if item.uploaded_by_id is None and task.user_id is not None:
|
||||
@@ -120,3 +144,7 @@ def run_download_task(task_id):
|
||||
speed=None,
|
||||
updated_at=timezone.now(),
|
||||
)
|
||||
finally:
|
||||
# Background threads hold their own DB connection; release it so
|
||||
# Gunicorn workers do not leak connections when threads finish.
|
||||
connection.close()
|
||||
|
||||
Reference in New Issue
Block a user