JakeBreath
8ebda6ab20
Test suite (the manual audit harness, now a real test):
- backend/apps/core/tests/test_security.py: 20 transactional tests across
guest visibility, object ownership, staged-upload/similarity privacy,
staff role boundaries, deletion rules, encrypted credentials, throttling
and the download allowlist. Uses temp media folders and clears cache.
Needs a one-time GRANT on test_j621 (documented in the module + README).
Images (deploy/J621-Frontend, deploy/J621-Backend, repo root as context):
- Frontend: node build -> static nginx with SPA fallback, asset caching and
an internal health endpoint.
- Backend: gunicorn + whitenoise (admin static collected at build), ffmpeg
for video thumbnails, migrations applied on start, GIT_HASH build arg so
the shell's version pill shows the commit.
Composes (distinct project names so they coexist with the dev stack):
- compose.yml (both), compose.frontend.yml, compose.backend.yml.
- A shared nginx proxy service is the only entry point (no host nginx, no
published host ports): /api,/admin,/static,/health -> backend, everything
else -> SPA; both upstreams resolve at request time so one config serves
all variants.
- A Tailscale sidecar per compose shares the nginx network namespace;
serve.default/frontend/backend.json use funnel ports 443 and 8443 only
(10000 is the remaining allowance) with ${TS_CERT_DOMAIN} substitution.
Registry: push_frontend.sh / push_backend.sh / push_all.sh build multi-arch
images and push :latest + :<sha> to the Gitea registry, following the
existing Packs-site pattern.
Docs: deploy/README.md + .env.example, ROADMAP section 6 updated,
AGENTS.md deployment and test notes.
Verified: all three composes validate, both nginx configs pass nginx -t,
both images build, the combined stack boots against real MariaDB/Redis
(migrations applied, /health ok, whitenoise serving admin static, env=prod,
git hash baked in), the proxy serves the SPA and routes /api, and
manage.py test apps.core.tests passes 20/20.
76 lines
4.2 KiB
Markdown
76 lines
4.2 KiB
Markdown
Use the venv on backend/venv/
|
|
|
|
DO NOT mess with the system's python
|
|
|
|
For Frontend work:
|
|
|
|
read both Frontend Design related Markdown files on the frontend/ folder
|
|
|
|
For anything touching e621 endpoints or payloads (posts, pools, tags, users,
|
|
favorites, IQDB, blacklists, uploads), consult the e621 OpenAPI spec first:
|
|
https://e621.wiki/openapi.yaml
|
|
|
|
Download it once per session (it is ~450 KB YAML), then grep it for the exact
|
|
path/parameter/schema names instead of guessing:
|
|
|
|
curl -sL https://e621.wiki/openapi.yaml -o /tmp/opencode/e621-openapi.yaml
|
|
grep -n "^ /pools.json" /tmp/opencode/e621-openapi.yaml
|
|
|
|
Notes learned from the spec (verify before trusting):
|
|
- Index endpoints often return a bare array (e.g. GET /pools.json) while the
|
|
show endpoint returns the object directly (e.g. GET /pools/{id}.json) — not
|
|
always wrapped in a named key.
|
|
- Pools index supports search[order] (id_asc, id_desc, name, created_at,
|
|
post_count), search[name_matches], search[category], search[is_active].
|
|
- Own settings (including user[blacklisted_tags]) are updated with
|
|
PATCH /users/{id}.json as form-encoded data; /users/me.json returns the user
|
|
object at the top level.
|
|
|
|
Project constraints (do not regress):
|
|
- Backend is Django 6 + DRF with token auth (no session auth on the API);
|
|
MariaDB/Redis come from docker-compose.yml.
|
|
- Deployment will be Docker-based (compose); do not add systemd/cron unit
|
|
files for scheduling — use the container setup for timers/workers.
|
|
- deploy/ is the deployment source of truth: J621-Frontend (SPA on static
|
|
nginx) and J621-Backend (gunicorn + whitenoise, ffmpeg, migrations on
|
|
start), three compose variants (both / frontend-only / backend-only) behind
|
|
a shared nginx proxy service, and a Tailscale sidecar per compose whose
|
|
serve configs only use funnel ports 443 / 8443 / 10000. Images are pushed
|
|
to the Gitea registry with deploy/push_*.sh (multi-arch, :latest + :sha,
|
|
GIT_HASH baked in for the version pill).
|
|
- Security/permission tests live in backend/apps/core/tests and need a
|
|
one-time grant: GRANT ALL ON `test_j621`.* TO 'j621'@'%';
|
|
- Same-origin and cross-origin frontends both work, and the SPA can also run
|
|
with no backend at all: a production build asks on first start (/setup) and
|
|
stores the answer in localStorage — a URL, "" for same origin, or "none"
|
|
for local mode. Local mode shows only the e621-facing pages (Online, Pools)
|
|
with credentials kept in this browser (j621.e621) and a "Setup Backend"
|
|
button in the shell. DEFAULT_BACKEND_URL in frontend/src/lib/backend.ts is
|
|
the prefilled default; the backend allows extra origins via
|
|
CORS_ALLOWED_ORIGINS (plus CSRF_TRUSTED_ORIGINS for the admin), and API
|
|
media URLs are absolute (built from the request host), so signed files load
|
|
cross-origin too. TRUST_PROXY_HEADERS=true is required behind a
|
|
TLS-terminating proxy.
|
|
- No server-side media processing: the home server cannot handle it.
|
|
Compression/optimization runs client-side (WebCodecs + WASM in a worker)
|
|
and the server only applies the result via POST /api/files/J-x/optimize/.
|
|
- Do not use imgdd; perceptual hashing is imagehash server-side.
|
|
- Chat/messaging features are out of scope.
|
|
|
|
Security hardening (do not weaken):
|
|
- e621 API keys are stored Fernet-encrypted with a key derived from
|
|
SECRET_KEY (apps/accounts/crypto.py); rotating SECRET_KEY invalidates them
|
|
(and all signed media URLs), so users must re-enter the key.
|
|
- API throttles live in REST_FRAMEWORK (env-overridable): anon 120/min,
|
|
user 600/min, login 5/min, register 20/hour, e621_proxy 60/hour.
|
|
- Only admins (superusers) may grant/revoke the staff role or delete
|
|
staff/admin accounts; staff manage regular/uploader accounts only.
|
|
- Storage, duplicates, delete, temp-clear, uploads and downloads require
|
|
upload rights (CanUpload), not merely authentication.
|
|
- Server-side fetching only happens for allowlisted e621 media hosts via
|
|
services.validate_remote_url / open_remote (every redirect hop is
|
|
re-validated); do not call requests.get on user-supplied URLs elsewhere.
|
|
- A repeatable audit harness (permission matrix, IDOR, guest visibility,
|
|
signed URLs, SSRF, throttles) was used to verify this; formalising it as
|
|
a test suite is still open in the roadmap.
|