JakeBreath
f86eccf9a3
Findings from the audit (50-check harness across guest/user/uploader/staff/ admin) and their fixes: - SSRF: 'Download to Library' and the staged-upload resolve path fetched any http(s) URL. services.validate_remote_url now enforces the e621 media allowlist and open_remote re-validates every redirect hop; the download-task create endpoint and the guest proxy use them, so internal addresses (127.0.0.1, LAN, metadata) are rejected with 400. - Privilege escalation: staff could promote users to staff and demote other staff. Role changes across the staff boundary now require an admin, matching the account-deletion rules; the Users page hides what the backend would refuse. - SPA-only gating: /api/storage/ and /api/duplicates/* were readable by any authenticated account (absolute paths, duplicate groups) while the SPA only shows them to uploaders. They now require CanUpload. - Throttling (REST_FRAMEWORK, env-overridable, counted in Redis): anon 120/min, user 600/min, login 5/min, register 20/hour, guest e621 proxy 60/hour. Login now goes through a throttled view. - e621 API keys are encrypted at rest with a Fernet key derived from SECRET_KEY (apps/accounts/crypto.py); a data migration encrypts existing rows and the column widens first. Reads decrypt transparently, legacy plaintext still works, and a changed SECRET_KEY reads as 'not configured' instead of leaking. Rotating SECRET_KEY now invalidates stored keys as well as signed media URLs. - Hardening: the server refuses to start with DEBUG=False while SECRET_KEY is still the development default. Verified: corrected harness 50/50 (guest visibility, IDOR, signed-URL tamper/expiry, staged-upload/similarity privacy, role matrix, SSRF), login throttles at the 6th attempt with 429, anon polling unaffected, the guest proxy still reaches allowlisted hosts, live e621 auth works with the decrypted key, and DB rows hold only ciphertext.
98 lines
3.2 KiB
Python
98 lines
3.2 KiB
Python
"""Minimal e621 API client for server-side matching and metadata refresh.
|
|
|
|
The SPA talks to e621 directly for browsing; this client exists for work the
|
|
browser cannot do reliably: long batch scans, and requests tied to a library
|
|
item rather than an open page. It uses the requesting user's stored
|
|
credentials and a global throttle (e621 asks for at most two requests per
|
|
second).
|
|
"""
|
|
|
|
import threading
|
|
import time
|
|
|
|
import requests
|
|
from django.conf import settings
|
|
|
|
REQUEST_INTERVAL = 0.5 # seconds between requests, per process
|
|
|
|
|
|
class E621Error(Exception):
|
|
"""A non-404 failure while talking to e621."""
|
|
|
|
|
|
class E621NotFound(E621Error):
|
|
"""The requested post does not exist (HTTP 404)."""
|
|
|
|
|
|
_throttle_lock = threading.Lock()
|
|
_last_request_at = 0.0
|
|
|
|
|
|
def credentials_configured(user):
|
|
return bool(user is not None and getattr(user, "e621_configured", False))
|
|
|
|
|
|
def _wait_for_slot():
|
|
global _last_request_at
|
|
with _throttle_lock:
|
|
delay = _last_request_at + REQUEST_INTERVAL - time.monotonic()
|
|
if delay > 0:
|
|
time.sleep(delay)
|
|
_last_request_at = time.monotonic()
|
|
|
|
|
|
def get(user, path, params=None, timeout=30, require_auth=True):
|
|
"""GET an e621 API path using the user's credentials.
|
|
|
|
Reads that e621 serves anonymously (searches, pools, tags) can pass
|
|
require_auth=False; matching endpoints keep requiring credentials.
|
|
|
|
Raises E621NotFound for 404s and E621Error for everything else that isn't
|
|
a 2xx, so callers never see requests exceptions.
|
|
"""
|
|
configured = credentials_configured(user)
|
|
if require_auth and not configured:
|
|
raise E621Error("Configure your e621 credentials in Account first.")
|
|
base = (getattr(user, "e621_base_url", "") or "https://e621.net").rstrip("/")
|
|
_wait_for_slot()
|
|
try:
|
|
response = requests.get(
|
|
f"{base}{path}",
|
|
params=params,
|
|
auth=(
|
|
(user.e621_username, user.e621_api_key_plain)
|
|
if configured
|
|
else None
|
|
),
|
|
headers={"User-Agent": settings.USER_AGENT},
|
|
timeout=timeout,
|
|
)
|
|
except requests.RequestException as exc:
|
|
raise E621Error(f"Could not reach e621: {exc}") from exc
|
|
if response.status_code == 404:
|
|
raise E621NotFound(f"e621 returned 404 for {path}")
|
|
if response.status_code >= 400:
|
|
raise E621Error(f"e621 replied {response.status_code} for {path}")
|
|
try:
|
|
return response.json()
|
|
except ValueError as exc:
|
|
raise E621Error("e621 returned an unexpected response.") from exc
|
|
|
|
|
|
def find_post_by_md5(user, md5):
|
|
"""The e621 post with this exact MD5, or None."""
|
|
payload = get(user, "/posts.json", params={"tags": f"md5:{md5}", "limit": 1})
|
|
posts = payload.get("posts") if isinstance(payload, dict) else None
|
|
if not posts:
|
|
return None
|
|
return posts[0]
|
|
|
|
|
|
def fetch_post(user, post_id):
|
|
"""One post by id. Raises E621NotFound when the post is gone."""
|
|
payload = get(user, f"/posts/{int(post_id)}.json")
|
|
post = payload.get("post") if isinstance(payload, dict) else None
|
|
if not isinstance(post, dict):
|
|
raise E621Error("e621 returned an unexpected post payload.")
|
|
return post
|