JakeBreath
e62d7af42f
- MD5 auto-match now sends 75 md5: metatags per posts.json query, the same batch size the original J621-Django app used (was 20); the limit cap no longer truncates batches. - every settled upload is upserted into the staged list right away, so the Pending / Visual Similarity / Auto-uploaded columns move as files land instead of waiting for the whole batch (auto-matched resolves and IQDB results use the same path). - finished upload tiles fade out and remove themselves ~2s after completing; failures stay until cleared. Batch counters are tracked separately from the visible tiles so the header and progress bar stay accurate as tiles vanish. - AGENTS.md now points at the original Django app for reference behavior. Verified live with a headless upload run: the columns showed the new files immediately and the 8 tiles were gone ~3s after finishing.
99 lines
5.5 KiB
Markdown
99 lines
5.5 KiB
Markdown
Use the venv on backend/venv/
|
|
|
|
DO NOT mess with the system's python
|
|
|
|
The original J621 (Django) app is kept for reference at
|
|
/mnt/Disco/Proyects/Python3.13/J621-Django/
|
|
Consult it when matching original behavior (batch sizes for e621 lookups,
|
|
per-page sets, upload flow) instead of guessing.
|
|
|
|
For Frontend work:
|
|
|
|
read both Frontend Design related Markdown files on the frontend/ folder
|
|
|
|
For anything touching e621 endpoints or payloads (posts, pools, tags, users,
|
|
favorites, IQDB, blacklists, uploads), consult the e621 OpenAPI spec first:
|
|
https://e621.wiki/openapi.yaml
|
|
|
|
Download it once per session (it is ~450 KB YAML), then grep it for the exact
|
|
path/parameter/schema names instead of guessing:
|
|
|
|
curl -sL https://e621.wiki/openapi.yaml -o /tmp/opencode/e621-openapi.yaml
|
|
grep -n "^ /pools.json" /tmp/opencode/e621-openapi.yaml
|
|
|
|
Notes learned from the spec (verify before trusting):
|
|
- Index endpoints often return a bare array (e.g. GET /pools.json) while the
|
|
show endpoint returns the object directly (e.g. GET /pools/{id}.json) — not
|
|
always wrapped in a named key.
|
|
- Pools index supports search[order] (id_asc, id_desc, name, created_at,
|
|
post_count), search[name_matches], search[category], search[is_active].
|
|
- Own settings (including user[blacklisted_tags]) are updated with
|
|
PATCH /users/{id}.json as form-encoded data; /users/me.json returns the user
|
|
object at the top level.
|
|
|
|
Project constraints (do not regress):
|
|
- Backend is Django 6 + DRF with token auth (no session auth on the API);
|
|
MariaDB/Redis come from docker-compose.yml.
|
|
- Deployment will be Docker-based (compose); do not add systemd/cron unit
|
|
files for scheduling — use the container setup for timers/workers.
|
|
- deploy/ is the deployment source of truth: J621-Frontend (SPA on static
|
|
nginx) and J621-Backend (gunicorn + whitenoise, ffmpeg, migrations on
|
|
start), three compose variants (both / frontend-only / backend-only) behind
|
|
a shared nginx proxy service, and a Tailscale sidecar per compose.
|
|
serve.*.json are the public Funnel variants (only ports 443 / 8443 / 10000);
|
|
serve.*.tailnet.json + compose.tailnet*.yml are the same stacks without
|
|
AllowFunnel (tailnet-only, no public exposure). Images are pushed to the
|
|
Gitea registry with deploy/push_*.sh (multi-arch, :latest + :sha, GIT_HASH
|
|
baked in for the version pill); deploy/gen_env.sh generates .env with
|
|
openssl secrets (--update keeps SECRET_KEY, --force rotates it).
|
|
- Periodic commands (follow syncs, similarity cleanup, guest blacklist
|
|
refresh) run in the composes' `scheduler` service — the backend image with
|
|
the j621-scheduler entrypoint, intervals via J621_*_EVERY. No host cron.
|
|
- Security/permission tests live in backend/apps/core/tests and need a
|
|
one-time grant: GRANT ALL ON `test_j621`.* TO 'j621'@'%';
|
|
|
|
Dev environment (this machine):
|
|
- MariaDB/Redis run from the repo root: `docker compose up -d` (host ports
|
|
3307 and 6380 — other local projects own 3305/6379/3406).
|
|
- Backend dev server: `backend/venv/bin/python manage.py runserver 0.0.0.0:8000`.
|
|
Django's system checks touch the database, so start MariaDB first; run it
|
|
from backend/ with the venv python.
|
|
- Frontend dev server: `npm run dev` in frontend/ — Vite listens on
|
|
[::1]:5173 (http://localhost:5173) and proxies /api to 127.0.0.1:8000.
|
|
- Restarting: kill dev servers by PID from `pgrep -af`; never
|
|
`pkill -f "manage.py runserver"` (the pattern matches its own shell).
|
|
- Suite: `backend/venv/bin/python manage.py test apps.core.tests` (~80 s).
|
|
- Same-origin and cross-origin frontends both work, and the SPA can also run
|
|
with no backend at all: a production build asks on first start (/setup) and
|
|
stores the answer in localStorage — a URL, "" for same origin, or "none"
|
|
for local mode. Local mode shows only the e621-facing pages (Online, Pools)
|
|
with credentials kept in this browser (j621.e621) and a "Setup Backend"
|
|
button in the shell. DEFAULT_BACKEND_URL in frontend/src/lib/backend.ts is
|
|
the prefilled default; the backend allows extra origins via
|
|
CORS_ALLOWED_ORIGINS (plus CSRF_TRUSTED_ORIGINS for the admin), and API
|
|
media URLs are absolute (built from the request host), so signed files load
|
|
cross-origin too. TRUST_PROXY_HEADERS=true is required behind a
|
|
TLS-terminating proxy.
|
|
- No server-side media processing: the home server cannot handle it.
|
|
Compression/optimization runs client-side (WebCodecs + WASM in a worker)
|
|
and the server only applies the result via POST /api/files/J-x/optimize/.
|
|
- Do not use imgdd; perceptual hashing is imagehash server-side.
|
|
- Chat/messaging features are out of scope.
|
|
|
|
Security hardening (do not weaken):
|
|
- e621 API keys are stored Fernet-encrypted with a key derived from
|
|
SECRET_KEY (apps/accounts/crypto.py); rotating SECRET_KEY invalidates them
|
|
(and all signed media URLs), so users must re-enter the key.
|
|
- API throttles live in REST_FRAMEWORK (env-overridable): anon 120/min,
|
|
user 600/min, login 5/min, register 20/hour, e621_proxy 60/hour.
|
|
- Only admins (superusers) may grant/revoke the staff role or delete
|
|
staff/admin accounts; staff manage regular/uploader accounts only.
|
|
- Storage, duplicates, delete, temp-clear, uploads and downloads require
|
|
upload rights (CanUpload), not merely authentication.
|
|
- Server-side fetching only happens for allowlisted e621 media hosts via
|
|
services.validate_remote_url / open_remote (every redirect hop is
|
|
re-validated); do not call requests.get on user-supplied URLs elsewhere.
|
|
- A repeatable audit harness (permission matrix, IDOR, guest visibility,
|
|
signed URLs, SSRF, throttles) was used to verify this; formalising it as
|
|
a test suite is still open in the roadmap.
|