JakeBreath
8ebda6ab20
Test suite (the manual audit harness, now a real test):
- backend/apps/core/tests/test_security.py: 20 transactional tests across
guest visibility, object ownership, staged-upload/similarity privacy,
staff role boundaries, deletion rules, encrypted credentials, throttling
and the download allowlist. Uses temp media folders and clears cache.
Needs a one-time GRANT on test_j621 (documented in the module + README).
Images (deploy/J621-Frontend, deploy/J621-Backend, repo root as context):
- Frontend: node build -> static nginx with SPA fallback, asset caching and
an internal health endpoint.
- Backend: gunicorn + whitenoise (admin static collected at build), ffmpeg
for video thumbnails, migrations applied on start, GIT_HASH build arg so
the shell's version pill shows the commit.
Composes (distinct project names so they coexist with the dev stack):
- compose.yml (both), compose.frontend.yml, compose.backend.yml.
- A shared nginx proxy service is the only entry point (no host nginx, no
published host ports): /api,/admin,/static,/health -> backend, everything
else -> SPA; both upstreams resolve at request time so one config serves
all variants.
- A Tailscale sidecar per compose shares the nginx network namespace;
serve.default/frontend/backend.json use funnel ports 443 and 8443 only
(10000 is the remaining allowance) with ${TS_CERT_DOMAIN} substitution.
Registry: push_frontend.sh / push_backend.sh / push_all.sh build multi-arch
images and push :latest + :<sha> to the Gitea registry, following the
existing Packs-site pattern.
Docs: deploy/README.md + .env.example, ROADMAP section 6 updated,
AGENTS.md deployment and test notes.
Verified: all three composes validate, both nginx configs pass nginx -t,
both images build, the combined stack boots against real MariaDB/Redis
(migrations applied, /health ok, whitenoise serving admin static, env=prod,
git hash baked in), the proxy serves the SPA and routes /api, and
manage.py test apps.core.tests passes 20/20.
4.2 KiB
4.2 KiB
Use the venv on backend/venv/
DO NOT mess with the system's python
For Frontend work:
read both Frontend Design related Markdown files on the frontend/ folder
For anything touching e621 endpoints or payloads (posts, pools, tags, users, favorites, IQDB, blacklists, uploads), consult the e621 OpenAPI spec first: https://e621.wiki/openapi.yaml
Download it once per session (it is ~450 KB YAML), then grep it for the exact path/parameter/schema names instead of guessing:
curl -sL https://e621.wiki/openapi.yaml -o /tmp/opencode/e621-openapi.yaml
grep -n "^ /pools.json" /tmp/opencode/e621-openapi.yaml
Notes learned from the spec (verify before trusting):
- Index endpoints often return a bare array (e.g. GET /pools.json) while the show endpoint returns the object directly (e.g. GET /pools/{id}.json) — not always wrapped in a named key.
- Pools index supports search[order] (id_asc, id_desc, name, created_at, post_count), search[name_matches], search[category], search[is_active].
- Own settings (including user[blacklisted_tags]) are updated with PATCH /users/{id}.json as form-encoded data; /users/me.json returns the user object at the top level.
Project constraints (do not regress):
- Backend is Django 6 + DRF with token auth (no session auth on the API); MariaDB/Redis come from docker-compose.yml.
- Deployment will be Docker-based (compose); do not add systemd/cron unit files for scheduling — use the container setup for timers/workers.
- deploy/ is the deployment source of truth: J621-Frontend (SPA on static nginx) and J621-Backend (gunicorn + whitenoise, ffmpeg, migrations on start), three compose variants (both / frontend-only / backend-only) behind a shared nginx proxy service, and a Tailscale sidecar per compose whose serve configs only use funnel ports 443 / 8443 / 10000. Images are pushed to the Gitea registry with deploy/push_*.sh (multi-arch, :latest + :sha, GIT_HASH baked in for the version pill).
- Security/permission tests live in backend/apps/core/tests and need a
one-time grant: GRANT ALL ON
test_j621.* TO 'j621'@'%'; - Same-origin and cross-origin frontends both work, and the SPA can also run with no backend at all: a production build asks on first start (/setup) and stores the answer in localStorage — a URL, "" for same origin, or "none" for local mode. Local mode shows only the e621-facing pages (Online, Pools) with credentials kept in this browser (j621.e621) and a "Setup Backend" button in the shell. DEFAULT_BACKEND_URL in frontend/src/lib/backend.ts is the prefilled default; the backend allows extra origins via CORS_ALLOWED_ORIGINS (plus CSRF_TRUSTED_ORIGINS for the admin), and API media URLs are absolute (built from the request host), so signed files load cross-origin too. TRUST_PROXY_HEADERS=true is required behind a TLS-terminating proxy.
- No server-side media processing: the home server cannot handle it. Compression/optimization runs client-side (WebCodecs + WASM in a worker) and the server only applies the result via POST /api/files/J-x/optimize/.
- Do not use imgdd; perceptual hashing is imagehash server-side.
- Chat/messaging features are out of scope.
Security hardening (do not weaken):
- e621 API keys are stored Fernet-encrypted with a key derived from SECRET_KEY (apps/accounts/crypto.py); rotating SECRET_KEY invalidates them (and all signed media URLs), so users must re-enter the key.
- API throttles live in REST_FRAMEWORK (env-overridable): anon 120/min, user 600/min, login 5/min, register 20/hour, e621_proxy 60/hour.
- Only admins (superusers) may grant/revoke the staff role or delete staff/admin accounts; staff manage regular/uploader accounts only.
- Storage, duplicates, delete, temp-clear, uploads and downloads require upload rights (CanUpload), not merely authentication.
- Server-side fetching only happens for allowlisted e621 media hosts via services.validate_remote_url / open_remote (every redirect hop is re-validated); do not call requests.get on user-supplied URLs elsewhere.
- A repeatable audit harness (permission matrix, IDOR, guest visibility, signed URLs, SSRF, throttles) was used to verify this; formalising it as a test suite is still open in the roadmap.