Docker deployment (2 images, 3 composes, Tailscale funnels) + committed security suite
Test suite (the manual audit harness, now a real test):
- backend/apps/core/tests/test_security.py: 20 transactional tests across
guest visibility, object ownership, staged-upload/similarity privacy,
staff role boundaries, deletion rules, encrypted credentials, throttling
and the download allowlist. Uses temp media folders and clears cache.
Needs a one-time GRANT on test_j621 (documented in the module + README).
Images (deploy/J621-Frontend, deploy/J621-Backend, repo root as context):
- Frontend: node build -> static nginx with SPA fallback, asset caching and
an internal health endpoint.
- Backend: gunicorn + whitenoise (admin static collected at build), ffmpeg
for video thumbnails, migrations applied on start, GIT_HASH build arg so
the shell's version pill shows the commit.
Composes (distinct project names so they coexist with the dev stack):
- compose.yml (both), compose.frontend.yml, compose.backend.yml.
- A shared nginx proxy service is the only entry point (no host nginx, no
published host ports): /api,/admin,/static,/health -> backend, everything
else -> SPA; both upstreams resolve at request time so one config serves
all variants.
- A Tailscale sidecar per compose shares the nginx network namespace;
serve.default/frontend/backend.json use funnel ports 443 and 8443 only
(10000 is the remaining allowance) with ${TS_CERT_DOMAIN} substitution.
Registry: push_frontend.sh / push_backend.sh / push_all.sh build multi-arch
images and push :latest + :<sha> to the Gitea registry, following the
existing Packs-site pattern.
Docs: deploy/README.md + .env.example, ROADMAP section 6 updated,
AGENTS.md deployment and test notes.
Verified: all three composes validate, both nginx configs pass nginx -t,
both images build, the combined stack boots against real MariaDB/Redis
(migrations applied, /health ok, whitenoise serving admin static, env=prod,
git hash baked in), the proxy serves the SPA and routes /api, and
manage.py test apps.core.tests passes 20/20.
This commit is contained in:
@@ -31,6 +31,15 @@ Project constraints (do not regress):
|
||||
MariaDB/Redis come from docker-compose.yml.
|
||||
- Deployment will be Docker-based (compose); do not add systemd/cron unit
|
||||
files for scheduling — use the container setup for timers/workers.
|
||||
- deploy/ is the deployment source of truth: J621-Frontend (SPA on static
|
||||
nginx) and J621-Backend (gunicorn + whitenoise, ffmpeg, migrations on
|
||||
start), three compose variants (both / frontend-only / backend-only) behind
|
||||
a shared nginx proxy service, and a Tailscale sidecar per compose whose
|
||||
serve configs only use funnel ports 443 / 8443 / 10000. Images are pushed
|
||||
to the Gitea registry with deploy/push_*.sh (multi-arch, :latest + :sha,
|
||||
GIT_HASH baked in for the version pill).
|
||||
- Security/permission tests live in backend/apps/core/tests and need a
|
||||
one-time grant: GRANT ALL ON `test_j621`.* TO 'j621'@'%';
|
||||
- Same-origin and cross-origin frontends both work, and the SPA can also run
|
||||
with no backend at all: a production build asks on first start (/setup) and
|
||||
stores the answer in localStorage — a URL, "" for same origin, or "none"
|
||||
|
||||
+16
-8
@@ -145,14 +145,22 @@ Files now stage first and are resolved before entering the library.
|
||||
|
||||
## 6. Infrastructure
|
||||
|
||||
- [ ] Guest blacklist refresh on a timer (`refresh_guest_blacklist` via cron/systemd)
|
||||
- [ ] Follow sync on a timer (`sync_followed_tags` + `sync_followed_pools` via
|
||||
cron/systemd, e.g. every 30 minutes)
|
||||
- [ ] Similarity temp cleanup on a timer (`cleanup_similarity` via cron; the
|
||||
TTL also cleans lazily when new checks are created)
|
||||
- [ ] Production setup: build the SPA, serve via Nginx (static + `/media` +
|
||||
`/library`), systemd unit for Waitress
|
||||
- [ ] Automated tests (backend API + frontend components)
|
||||
- [ ] Guest blacklist refresh on a timer (in-container scheduler)
|
||||
- [ ] Follow sync on a timer (in-container scheduler, e.g. every 30 minutes)
|
||||
- [ ] Similarity temp cleanup on a timer (in-container scheduler; the TTL
|
||||
also cleans lazily when new checks are created)
|
||||
- [x] Production setup: two Docker images (SPA on static nginx, API on
|
||||
gunicorn + whitenoise with ffmpeg) and three compose variants (both /
|
||||
frontend-only / backend-only) behind a shared nginx proxy service, each
|
||||
with a Tailscale sidecar and a funnel serve config (ports 443 / 8443 /
|
||||
10000 only). Multi-arch images are pushed to the Gitea registry by
|
||||
`deploy/push_*.sh`
|
||||
- [x] Security/permission test suite (`manage.py test apps.core.tests`):
|
||||
guest visibility, object ownership, upload/similarity privacy, role
|
||||
boundaries, deletion rules, encrypted credentials, throttling, SSRF
|
||||
allowlist
|
||||
- [ ] Broader automated tests (more backend API coverage + frontend
|
||||
components)
|
||||
- [x] Backfill e621 metadata for items downloaded before metadata was stored —
|
||||
covered by the match scan (`/api/matches/` scope `all`, or per-item
|
||||
Recheck) and `manage.py match_e621 --scope all`
|
||||
|
||||
@@ -0,0 +1,449 @@
|
||||
"""Security regression suite.
|
||||
|
||||
Covers the checks that must never live on the SPA side alone: guest
|
||||
visibility, object ownership, staged-upload/similarity privacy, staff role
|
||||
boundaries, account deletion rules, encrypted credentials, throttling and
|
||||
the SSRF allowlist for server-side downloads.
|
||||
|
||||
Run: backend/venv/bin/python manage.py test apps.core.tests
|
||||
|
||||
The database user needs rights to create the test database once, e.g.:
|
||||
GRANT ALL ON `test_j621`.* TO 'j621'@'%';
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.core import signing
|
||||
from django.core.cache import cache
|
||||
from django.core.files.uploadedfile import SimpleUploadedFile
|
||||
from django.test import Client, TestCase, override_settings
|
||||
|
||||
from rest_framework.authtoken.models import Token
|
||||
|
||||
from apps.library import services
|
||||
from apps.library.models import (
|
||||
DownloadTask,
|
||||
MediaItem,
|
||||
MediaLocation,
|
||||
SimilarityCheck,
|
||||
TempUpload,
|
||||
)
|
||||
from apps.library.services import MEDIA_FILE_SALT
|
||||
|
||||
User = get_user_model()
|
||||
|
||||
|
||||
def jpost(client, path, body=None):
|
||||
return client.post(path, data=json.dumps(body or {}), content_type="application/json")
|
||||
|
||||
|
||||
def jpatch(client, path, body=None):
|
||||
return client.patch(path, data=json.dumps(body or {}), content_type="application/json")
|
||||
|
||||
|
||||
class SecurityTestCase(TestCase):
|
||||
"""Shared fixtures: throwaway accounts for every role and temp folders."""
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
super().setUpClass()
|
||||
cls._tmp = tempfile.mkdtemp(prefix="j621-security-")
|
||||
cls._media = Path(cls._tmp) / "media"
|
||||
cls._watched = cls._media / "library"
|
||||
cls._watched.mkdir(parents=True, exist_ok=True)
|
||||
cls._settings = override_settings(
|
||||
MEDIA_ROOT=str(cls._media),
|
||||
WATCHED_FOLDER=str(cls._watched),
|
||||
)
|
||||
cls._settings.enable()
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
cls._settings.disable()
|
||||
shutil.rmtree(cls._tmp, ignore_errors=True)
|
||||
super().tearDownClass()
|
||||
|
||||
def setUp(self):
|
||||
cache.clear() # throttle counters and cached payloads
|
||||
self.users = {}
|
||||
for name, role in (
|
||||
("sec-user", "user"),
|
||||
("sec-uploader", "uploader"),
|
||||
("sec-uploader2", "uploader"),
|
||||
("sec-staff", "staff"),
|
||||
("sec-staff2", "staff"),
|
||||
):
|
||||
user = User.objects.create_user(
|
||||
username=name, password="audit-pass-123456"
|
||||
)
|
||||
user.role = role
|
||||
user.save(update_fields=["role"])
|
||||
self.users[name] = user
|
||||
self.admin = User.objects.create_superuser(
|
||||
username="sec-admin", password="audit-pass-123456"
|
||||
)
|
||||
self.tokens = {
|
||||
name: Token.objects.create(user=user).key
|
||||
for name, user in self.users.items()
|
||||
}
|
||||
self.tokens["sec-admin"] = Token.objects.create(user=self.admin).key
|
||||
|
||||
def client_for(self, name):
|
||||
client = Client()
|
||||
client.defaults["HTTP_AUTHORIZATION"] = f"Token {self.tokens[name]}"
|
||||
return client
|
||||
|
||||
@property
|
||||
def guest(self):
|
||||
return Client()
|
||||
|
||||
def make_item(self, label, *, hidden=False, owner=None, tags=None):
|
||||
path = self._watched / f"{label}.bin"
|
||||
path.write_bytes(b"audit-" + label.encode())
|
||||
item = MediaItem.objects.create(
|
||||
md5=hashlib.md5(label.encode()).hexdigest(),
|
||||
size=path.stat().st_size,
|
||||
uploaded_by=owner,
|
||||
tags=tags or [],
|
||||
)
|
||||
MediaLocation.objects.create(
|
||||
item=item, path=str(path), rel_path=path.name, mtime=time.time()
|
||||
)
|
||||
if hidden:
|
||||
MediaItem.objects.filter(pk=item.pk).update(hidden_from_guests=True)
|
||||
item.refresh_from_db()
|
||||
return item
|
||||
|
||||
def old_signature(self, item, action="raw", age=3 * 86400):
|
||||
"""A valid signature minted `age` seconds ago."""
|
||||
real_time = signing.time
|
||||
|
||||
class Backdated:
|
||||
def time(self):
|
||||
return real_time.time() - age
|
||||
|
||||
try:
|
||||
signing.time = Backdated()
|
||||
return signing.dumps(
|
||||
{"item": item.id, "user": self.users["sec-uploader"].id, "action": action},
|
||||
salt=MEDIA_FILE_SALT,
|
||||
)
|
||||
finally:
|
||||
signing.time = real_time
|
||||
|
||||
|
||||
class GuestVisibilityTests(SecurityTestCase):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
self.visible = self.make_item("visible", owner=self.users["sec-uploader"])
|
||||
self.hidden = self.make_item(
|
||||
"hidden",
|
||||
hidden=True,
|
||||
owner=self.users["sec-uploader"],
|
||||
tags=["sec_audit_hidden_tag"],
|
||||
)
|
||||
|
||||
def test_guest_list_and_retrieve_hide_protected_items(self):
|
||||
response = self.guest.get(
|
||||
f"/api/files/?j_ids=J-{self.visible.id},J-{self.hidden.id}"
|
||||
)
|
||||
self.assertEqual(
|
||||
{row["j_id"] for row in response.json()["results"]},
|
||||
{f"J-{self.visible.id}"},
|
||||
)
|
||||
self.assertEqual(self.guest.get(f"/api/files/J-{self.hidden.id}/").status_code, 404)
|
||||
self.assertEqual(
|
||||
self.guest.get(f"/api/files/J-{self.hidden.id}/raw/").status_code, 404
|
||||
)
|
||||
self.assertEqual(
|
||||
self.guest.get(f"/api/files/J-{self.hidden.id}/thumbnail/").status_code, 404
|
||||
)
|
||||
self.assertEqual(self.guest.get(f"/api/files/{self.hidden.md5}/").status_code, 404)
|
||||
self.assertEqual(
|
||||
self.guest.get(f"/api/files/J-{self.hidden.id}/neighbors/").status_code, 404
|
||||
)
|
||||
|
||||
def test_guest_lookup_and_tag_cloud_hide_protected_items(self):
|
||||
response = jpost(
|
||||
self.guest,
|
||||
"/api/files/lookup/",
|
||||
{"md5s": [self.visible.md5, self.hidden.md5]},
|
||||
)
|
||||
self.assertEqual(response.json()["found"], [self.visible.md5])
|
||||
cloud = self.guest.get("/api/tags/cloud/").json()["tags"]
|
||||
self.assertNotIn("sec_audit_hidden_tag", [entry["tag"] for entry in cloud])
|
||||
|
||||
def test_authenticated_users_and_signed_urls_see_protected_items(self):
|
||||
uploader = self.client_for("sec-uploader")
|
||||
self.assertEqual(uploader.get(f"/api/files/J-{self.hidden.id}/").status_code, 200)
|
||||
signed = signing.dumps(
|
||||
{"item": self.hidden.id, "user": self.users["sec-uploader"].id, "action": "raw"},
|
||||
salt=MEDIA_FILE_SALT,
|
||||
)
|
||||
self.assertEqual(
|
||||
self.guest.get(f"/api/files/J-{self.hidden.id}/raw/?sig={signed}").status_code,
|
||||
200,
|
||||
)
|
||||
|
||||
def test_signature_integrity(self):
|
||||
signed = signing.dumps(
|
||||
{"item": self.hidden.id, "user": self.users["sec-uploader"].id, "action": "raw"},
|
||||
salt=MEDIA_FILE_SALT,
|
||||
)
|
||||
raw = f"/api/files/J-{self.hidden.id}/raw/"
|
||||
thumbnail = f"/api/files/J-{self.hidden.id}/thumbnail/"
|
||||
# Tampered signature (same action).
|
||||
self.assertEqual(self.guest.get(f"{raw}?sig={signed[:-4]}AAAA").status_code, 404)
|
||||
# Valid signature, wrong action.
|
||||
self.assertEqual(self.guest.get(f"{thumbnail}?sig={signed}").status_code, 404)
|
||||
# Expired signature (minted three days ago).
|
||||
expired = self.old_signature(self.hidden)
|
||||
self.assertEqual(self.guest.get(f"{raw}?sig={expired}").status_code, 404)
|
||||
|
||||
|
||||
class RoleBoundaryTests(SecurityTestCase):
|
||||
def test_non_uploader_is_read_only(self):
|
||||
user = self.client_for("sec-user")
|
||||
self.assertEqual(user.get("/api/storage/").status_code, 403)
|
||||
self.assertEqual(user.get("/api/duplicates/md5/").status_code, 403)
|
||||
self.assertEqual(
|
||||
jpost(user, "/api/duplicates/visual/", {"j_id": "J-1"}).status_code, 403
|
||||
)
|
||||
self.assertEqual(user.get("/api/uploads/").status_code, 403)
|
||||
self.assertEqual(user.get("/api/online/downloads/").status_code, 403)
|
||||
self.assertEqual(user.get("/api/matches/").status_code, 403)
|
||||
self.assertEqual(user.get("/api/stats/").status_code, 403)
|
||||
self.assertEqual(user.get("/api/users/").status_code, 403)
|
||||
self.assertEqual(jpost(user, "/api/delete/", {"j_ids": []}).status_code, 403)
|
||||
self.assertEqual(jpost(user, "/api/temp/clear/").status_code, 403)
|
||||
|
||||
def test_uploader_and_staff_reach_their_tools(self):
|
||||
uploader = self.client_for("sec-uploader")
|
||||
staff = self.client_for("sec-staff")
|
||||
self.assertEqual(uploader.get("/api/storage/").status_code, 200)
|
||||
self.assertEqual(jpost(uploader, "/api/delete/", {"j_ids": []}).status_code, 200)
|
||||
self.assertEqual(uploader.get("/api/duplicates/md5/").status_code, 200)
|
||||
self.assertEqual(staff.get("/api/stats/").status_code, 200)
|
||||
self.assertEqual(staff.get("/api/users/").status_code, 200)
|
||||
|
||||
def test_only_admins_change_staff_roles(self):
|
||||
staff = self.client_for("sec-staff")
|
||||
user_id = self.users["sec-user"].id
|
||||
self.assertEqual(
|
||||
jpatch(staff, f"/api/users/{user_id}/", {"role": "staff"}).status_code,
|
||||
403,
|
||||
)
|
||||
self.assertEqual(
|
||||
jpatch(
|
||||
staff,
|
||||
f"/api/users/{self.users['sec-staff2'].id}/",
|
||||
{"role": "user"},
|
||||
).status_code,
|
||||
403,
|
||||
)
|
||||
admin = self.client_for("sec-admin")
|
||||
self.assertEqual(
|
||||
jpatch(admin, f"/api/users/{user_id}/", {"role": "uploader"}).status_code,
|
||||
200,
|
||||
)
|
||||
|
||||
def test_deletion_rules(self):
|
||||
staff = self.client_for("sec-staff")
|
||||
admin = self.client_for("sec-admin")
|
||||
self.assertEqual(
|
||||
staff.delete(f"/api/users/{self.users['sec-user'].id}/").status_code, 204
|
||||
)
|
||||
self.assertEqual(
|
||||
staff.delete(f"/api/users/{self.users['sec-staff2'].id}/").status_code, 403
|
||||
)
|
||||
self.assertEqual(
|
||||
staff.delete(f"/api/users/{self.users['sec-staff'].id}/").status_code, 400
|
||||
)
|
||||
self.assertEqual(
|
||||
admin.delete(f"/api/users/{self.users['sec-staff2'].id}/").status_code, 204
|
||||
)
|
||||
|
||||
def test_deleting_a_user_keeps_library_items(self):
|
||||
owner = self.users["sec-uploader2"]
|
||||
item = self.make_item("orphan", owner=owner)
|
||||
self.client_for("sec-admin").delete(f"/api/users/{owner.id}/")
|
||||
item.refresh_from_db()
|
||||
self.assertIsNone(item.uploaded_by_id)
|
||||
|
||||
|
||||
class ItemOwnershipTests(SecurityTestCase):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
self.item = self.make_item("owned", owner=self.users["sec-uploader"])
|
||||
|
||||
def test_other_uploaders_cannot_touch_items(self):
|
||||
other = self.client_for("sec-uploader2")
|
||||
self.assertEqual(
|
||||
jpost(other, f"/api/files/J-{self.item.id}/match/", {"post_id": "x"}).status_code,
|
||||
403,
|
||||
)
|
||||
self.assertEqual(
|
||||
jpost(other, f"/api/files/J-{self.item.id}/unlink/").status_code, 403
|
||||
)
|
||||
self.assertEqual(
|
||||
other.post(f"/api/files/J-{self.item.id}/optimize/").status_code, 403
|
||||
)
|
||||
self.assertEqual(
|
||||
jpatch(other, f"/api/files/J-{self.item.id}/", {"rating": "s"}).status_code,
|
||||
403,
|
||||
)
|
||||
|
||||
def test_owner_and_staff_pass_the_permission_check(self):
|
||||
for name in ("sec-uploader", "sec-staff"):
|
||||
client = self.client_for(name)
|
||||
# An invalid post id proves authorization happened: validation answers.
|
||||
self.assertEqual(
|
||||
jpost(client, f"/api/files/J-{self.item.id}/match/", {"post_id": "x"}).status_code,
|
||||
400,
|
||||
)
|
||||
self.assertEqual(
|
||||
client.post(f"/api/files/J-{self.item.id}/optimize/").status_code, 400
|
||||
)
|
||||
owner = self.client_for("sec-uploader")
|
||||
self.assertEqual(
|
||||
jpatch(
|
||||
owner,
|
||||
f"/api/files/J-{self.item.id}/",
|
||||
{"rating": "", "tags": [], "notes": ""},
|
||||
).status_code,
|
||||
200,
|
||||
)
|
||||
|
||||
|
||||
class PrivacyTests(SecurityTestCase):
|
||||
def test_staged_uploads_are_private(self):
|
||||
temp = TempUpload.objects.create(
|
||||
user=self.users["sec-uploader2"],
|
||||
file=SimpleUploadedFile("staged.bin", b"staged"),
|
||||
original_filename="staged.bin",
|
||||
md5=hashlib.md5(b"staged").hexdigest(),
|
||||
size=6,
|
||||
)
|
||||
self.assertEqual(
|
||||
self.client_for("sec-uploader2").get(f"/api/uploads/{temp.id}/file/").status_code,
|
||||
200,
|
||||
)
|
||||
self.assertEqual(
|
||||
self.client_for("sec-uploader").get(f"/api/uploads/{temp.id}/file/").status_code,
|
||||
404,
|
||||
)
|
||||
self.assertEqual(
|
||||
self.client_for("sec-staff").get(f"/api/uploads/{temp.id}/file/").status_code,
|
||||
200,
|
||||
)
|
||||
self.assertEqual(self.guest.get(f"/api/uploads/{temp.id}/file/").status_code, 401)
|
||||
self.assertNotIn(
|
||||
str(temp.id), self.client_for("sec-uploader").get("/api/uploads/").content.decode()
|
||||
)
|
||||
|
||||
def test_similarity_checks_are_private(self):
|
||||
check = SimilarityCheck.objects.create(
|
||||
user=self.users["sec-uploader2"],
|
||||
file=SimpleUploadedFile("similar.bin", b"similar"),
|
||||
original_filename="similar.bin",
|
||||
md5=hashlib.md5(b"similar").hexdigest(),
|
||||
size=7,
|
||||
)
|
||||
self.assertEqual(
|
||||
self.client_for("sec-uploader2").get(f"/api/similarity/{check.id}/").status_code,
|
||||
200,
|
||||
)
|
||||
self.assertEqual(
|
||||
self.client_for("sec-uploader").get(f"/api/similarity/{check.id}/").status_code,
|
||||
404,
|
||||
)
|
||||
self.assertEqual(
|
||||
self.client_for("sec-uploader").get(f"/api/similarity/{check.id}/file/").status_code,
|
||||
404,
|
||||
)
|
||||
self.assertEqual(
|
||||
self.client_for("sec-staff").get(f"/api/similarity/{check.id}/").status_code,
|
||||
200,
|
||||
)
|
||||
|
||||
|
||||
class CredentialEncryptionTests(SecurityTestCase):
|
||||
def test_keys_are_encrypted_at_rest_and_only_owners_see_them(self):
|
||||
client = self.client_for("sec-user")
|
||||
response = client.put(
|
||||
"/api/auth/e621/",
|
||||
data=json.dumps(
|
||||
{"username": "tester", "api_key": "abc123def456", "base_url": "https://e621.net"}
|
||||
),
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertEqual(response.json()["api_key"], "abc123def456")
|
||||
user = User.objects.get(pk=self.users["sec-user"].pk)
|
||||
self.assertTrue(user.e621_api_key.startswith("enc:"))
|
||||
self.assertNotIn("abc123def456", user.e621_api_key)
|
||||
self.assertTrue(user.e621_configured)
|
||||
# Other users never see it.
|
||||
other = self.client_for("sec-staff")
|
||||
self.assertNotIn("abc123def456", other.get("/api/auth/me/").content.decode())
|
||||
self.assertNotIn("abc123def456", other.get("/api/auth/e621/").content.decode())
|
||||
|
||||
def test_undecryptable_values_read_as_not_configured(self):
|
||||
user = User.objects.get(pk=self.users["sec-user"].pk)
|
||||
user.e621_username = "tester"
|
||||
user.e621_api_key = "enc:not-a-valid-token"
|
||||
user.save(update_fields=["e621_username", "e621_api_key"])
|
||||
user.refresh_from_db()
|
||||
self.assertEqual(user.e621_api_key_plain, "")
|
||||
self.assertFalse(user.e621_configured)
|
||||
|
||||
|
||||
class ThrottleTests(SecurityTestCase):
|
||||
def test_login_is_rate_limited(self):
|
||||
codes = [
|
||||
self.guest.post(
|
||||
"/api/auth/token/",
|
||||
data=json.dumps({"username": "nobody", "password": "wrong"}),
|
||||
content_type="application/json",
|
||||
).status_code
|
||||
for _ in range(6)
|
||||
]
|
||||
self.assertEqual(codes[:5], [400] * 5)
|
||||
self.assertEqual(codes[5], 429)
|
||||
|
||||
def test_anonymous_polling_is_not_throttled(self):
|
||||
self.assertEqual(
|
||||
{self.guest.get("/api/status/").status_code for _ in range(12)}, {200}
|
||||
)
|
||||
|
||||
|
||||
class RemoteUrlTests(SecurityTestCase):
|
||||
def test_allowlist(self):
|
||||
for url in ("file:///etc/passwd", "http://127.0.0.1:1/", "https://evil.example/"):
|
||||
with self.assertRaises(services.RemoteUrlError):
|
||||
services.validate_remote_url(url)
|
||||
self.assertEqual(
|
||||
services.validate_remote_url("https://static1.e621.net/data/x.png"),
|
||||
"https://static1.e621.net/data/x.png",
|
||||
)
|
||||
|
||||
def test_download_creation_rejects_internal_urls(self):
|
||||
uploader = self.client_for("sec-uploader")
|
||||
for url in ("http://127.0.0.1:1/", "http://192.168.1.1/", "file:///etc/passwd"):
|
||||
response = jpost(
|
||||
uploader, "/api/online/downloads/", {"url": url, "filename": "x.bin"}
|
||||
)
|
||||
self.assertEqual(response.status_code, 400, url)
|
||||
self.assertEqual(DownloadTask.objects.count(), 0)
|
||||
|
||||
def test_guest_proxy_rejects_internal_urls(self):
|
||||
response = self.guest.get(
|
||||
"/api/online/file/?url=http%3A%2F%2F127.0.0.1%3A1%2F"
|
||||
)
|
||||
self.assertEqual(response.status_code, 400)
|
||||
@@ -61,7 +61,14 @@ if os.getenv("TRUST_PROXY_HEADERS", "false").lower() == "true":
|
||||
|
||||
|
||||
def _git_commit_hash():
|
||||
"""Short git hash of the running build, mirroring the original app."""
|
||||
"""Short git hash of the running build, mirroring the original app.
|
||||
|
||||
Containers rarely ship the .git directory, so an explicit GIT_COMMIT_HASH
|
||||
environment variable (baked in at image build time) wins when present.
|
||||
"""
|
||||
configured = os.getenv("GIT_COMMIT_HASH", "").strip()
|
||||
if configured:
|
||||
return configured[:12]
|
||||
try:
|
||||
return subprocess.check_output(
|
||||
["git", "rev-parse", "--short", "HEAD"],
|
||||
@@ -98,6 +105,9 @@ INSTALLED_APPS = [
|
||||
|
||||
MIDDLEWARE = [
|
||||
"django.middleware.security.SecurityMiddleware",
|
||||
# Serves the Django admin's static files in production (collected at
|
||||
# image build time); harmless in dev.
|
||||
"whitenoise.middleware.WhiteNoiseMiddleware",
|
||||
# Must sit above CommonMiddleware so preflights are answered early.
|
||||
"corsheaders.middleware.CorsMiddleware",
|
||||
"django.contrib.sessions.middleware.SessionMiddleware",
|
||||
@@ -173,6 +183,16 @@ USE_TZ = True
|
||||
STATIC_URL = "static/"
|
||||
STATIC_ROOT = BASE_DIR / "staticfiles"
|
||||
|
||||
STORAGES = {
|
||||
"default": {
|
||||
"BACKEND": "django.core.files.storage.FileSystemStorage",
|
||||
},
|
||||
"staticfiles": {
|
||||
# No manifest: works whether or not collectstatic ran (dev included).
|
||||
"BACKEND": "whitenoise.storage.CompressedStaticFilesStorage",
|
||||
},
|
||||
}
|
||||
|
||||
MEDIA_URL = "/media/"
|
||||
MEDIA_ROOT = BASE_DIR / "media"
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@ djangorestframework>=3.17
|
||||
django-filter>=25.1
|
||||
django-cors-headers>=4.9
|
||||
cryptography>=46.0
|
||||
gunicorn>=23.0
|
||||
whitenoise>=6.8
|
||||
Pillow>=11.0
|
||||
python-dotenv>=1.0
|
||||
requests>=2.32
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# J621 deployment environment — copy to deploy/.env and fill in.
|
||||
#
|
||||
# Compose reads this file for variable substitution AND passes it to the
|
||||
# backend container (env_file), so everything here is visible to Django.
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tailscale
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reusable, untagged auth key (Settings -> Keys -> Generate auth key,
|
||||
# Reusable = on, Tags = none). Auto-approves the device.
|
||||
TS_AUTHKEY=
|
||||
|
||||
# Hostname this deployment gets on the tailnet; the funnel URL becomes
|
||||
# https://<TS_HOSTNAME>.<tailnet>.ts.net. Use distinct names per compose.
|
||||
TS_HOSTNAME=j621
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Django
|
||||
# ---------------------------------------------------------------------------
|
||||
# Long random string. Signs media URLs and encrypts stored e621 API keys —
|
||||
# rotating it invalidates both, so users re-enter their e621 key.
|
||||
SECRET_KEY=change-me-to-a-long-random-string
|
||||
DEBUG=False
|
||||
|
||||
# Hosts Django may be reached on, comma separated, no scheme. Add the tailnet
|
||||
# hostname of every compose that talks to this backend (and keep localhost /
|
||||
# 127.0.0.1 for container health checks).
|
||||
ALLOWED_HOSTS=j621.rainbow-herring.ts.net,localhost,127.0.0.1
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cross-origin access (needed for the separate frontend + backend deploys)
|
||||
# ---------------------------------------------------------------------------
|
||||
# The origin the SPA is served from, e.g. https://j621-frontend.<tailnet>.ts.net
|
||||
# CORS_ALLOWED_ORIGINS=
|
||||
# CSRF_TRUSTED_ORIGINS=
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Database (created by the compose files; change the password)
|
||||
# ---------------------------------------------------------------------------
|
||||
DB_NAME=j621
|
||||
DB_USER=j621
|
||||
DB_PASSWORD=j621
|
||||
DB_ROOT_PASSWORD=j621root
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Optional overrides
|
||||
# ---------------------------------------------------------------------------
|
||||
# GUNICORN_WORKERS=2
|
||||
# GUNICORN_THREADS=4
|
||||
# GUNICORN_TIMEOUT=120
|
||||
|
||||
# Rate limits (per IP anonymous, per account signed in)
|
||||
# THROTTLE_ANON=120/min
|
||||
# THROTTLE_USER=600/min
|
||||
# THROTTLE_LOGIN=5/min
|
||||
# THROTTLE_REGISTER=20/hour
|
||||
# THROTTLE_E621_PROXY=60/hour
|
||||
|
||||
# e621 media hosts the backend may fetch from (downloads, proxies)
|
||||
# E621_MEDIA_HOSTS=static1.e621.net,static2.e621.net,static3.e621.net
|
||||
|
||||
# Ephemeral similarity-check lifetime in minutes
|
||||
# SIMILARITY_TTL_MINUTES=30
|
||||
|
||||
# Registry/tag used by the compose files and push scripts
|
||||
# J621_REGISTRY=gitea.rainbow-herring.ts.net/jakebreath
|
||||
# J621_TAG=latest
|
||||
@@ -0,0 +1,4 @@
|
||||
# Runtime data and state — never commit these.
|
||||
.env
|
||||
data/
|
||||
tailscale-state/
|
||||
@@ -0,0 +1,40 @@
|
||||
# J621-Backend — Django + gunicorn, with migrations applied on start.
|
||||
#
|
||||
# Build context is the repository root, e.g.:
|
||||
# docker build -f deploy/J621-Backend -t j621-backend .
|
||||
#
|
||||
# Needs MariaDB and Redis (see the compose files). ffmpeg is used for video
|
||||
# thumbnails; media/logs live under the mounted volumes.
|
||||
|
||||
# syntax=docker/dockerfile:1
|
||||
|
||||
FROM python:3.14-slim
|
||||
|
||||
# Baked in by the push scripts so the shell's version pill shows the commit
|
||||
# even though the image has no .git directory.
|
||||
ARG GIT_HASH=unknown
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
GIT_COMMIT_HASH=$GIT_HASH
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/requirements.txt ./
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY backend/ ./
|
||||
COPY deploy/backend-entrypoint.sh /usr/local/bin/j621-entrypoint
|
||||
|
||||
RUN chmod +x /usr/local/bin/j621-entrypoint \
|
||||
&& mkdir -p /app/media /app/logs \
|
||||
&& python manage.py collectstatic --noinput
|
||||
|
||||
EXPOSE 8000
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health')" || exit 1
|
||||
|
||||
ENTRYPOINT ["j621-entrypoint"]
|
||||
@@ -0,0 +1,24 @@
|
||||
# J621-Frontend — builds the SPA and serves it as static files.
|
||||
#
|
||||
# Build context is the repository root, e.g.:
|
||||
# docker build -f deploy/J621-Frontend -t j621-frontend .
|
||||
#
|
||||
# This container only serves the SPA (with the index fallback). The public
|
||||
# entry point is the separate "nginx" reverse-proxy service in the compose
|
||||
# files, so API routing stays in one place.
|
||||
|
||||
# syntax=docker/dockerfile:1
|
||||
|
||||
FROM node:22-alpine AS build
|
||||
WORKDIR /app
|
||||
COPY frontend/package.json frontend/package-lock.json ./
|
||||
RUN npm ci --no-audit --no-fund
|
||||
COPY frontend/ ./
|
||||
RUN npm run build
|
||||
|
||||
FROM nginx:1.29-alpine
|
||||
COPY deploy/nginx-frontend.conf /etc/nginx/conf.d/default.conf
|
||||
COPY --from=build /app/dist /usr/share/nginx/html
|
||||
EXPOSE 80
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \
|
||||
CMD wget -q --spider http://127.0.0.1/nginx-health || exit 1
|
||||
@@ -0,0 +1,100 @@
|
||||
# J621 deployment (Docker + Tailscale)
|
||||
|
||||
Three compose variants, all behind an **nginx** service and a **Tailscale**
|
||||
sidecar. Nothing is published on the host's ports and no system nginx or
|
||||
reverse proxy is involved: the sidecar shares the nginx service's network
|
||||
namespace and Tailscale Serve/Funnel exposes it.
|
||||
|
||||
| Compose | Services | Funnel | Serve config |
|
||||
| --- | --- | --- | --- |
|
||||
| `compose.yml` (default) | mariadb, redis, backend, frontend, nginx, tailscale | `https://<host>.<tailnet>.ts.net` → nginx → backend + SPA | `serve.default.json` |
|
||||
| `compose.frontend.yml` | frontend, nginx, tailscale | `https://<host>.<tailnet>.ts.net` → nginx → SPA only | `serve.frontend.json` |
|
||||
| `compose.backend.yml` | mariadb, redis, backend, nginx, tailscale | `https://<host>.<tailnet>.ts.net:8443` → nginx → API | `serve.backend.json` |
|
||||
|
||||
Funnel can only expose ports **443**, **8443** and **10000**, so the separate
|
||||
backend uses 8443. Change it in `serve.backend.json` (and `.env`) if you
|
||||
prefer 10000.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
cd deploy
|
||||
cp .env.example .env # fill in TS_AUTHKEY, SECRET_KEY, ALLOWED_HOSTS, ...
|
||||
docker compose up -d # default: everything on one host
|
||||
# or
|
||||
docker compose -f compose.frontend.yml up -d
|
||||
docker compose -f compose.backend.yml up -d
|
||||
```
|
||||
|
||||
The images are pulled from the Gitea registry; append `--build` (or run the
|
||||
push scripts) if you build locally. Data lives in `deploy/data/`
|
||||
(`media/`, `logs/`, `mariadb/`, `redis/`) and the Tailscale node state in
|
||||
`deploy/tailscale-state/`; both are git-ignored.
|
||||
|
||||
## First start
|
||||
|
||||
The SPA asks where the backend is (`/setup`) in production builds:
|
||||
|
||||
- **Default compose** — leave the field blank (same origin) and everything
|
||||
works through nginx.
|
||||
- **Frontend-only + backend-only** — point the (cross-origin) SPA at the
|
||||
backend's funnel URL, e.g. `https://j621-backend.<tailnet>.ts.net:8443`,
|
||||
and give the backend `CORS_ALLOWED_ORIGINS=https://<frontend-host>.<tailnet>.ts.net`
|
||||
(plus `CSRF_TRUSTED_ORIGINS` for the admin).
|
||||
- Without a backend at all, choose **"Continue without a backend"** to run in
|
||||
local mode (e621 browsing only).
|
||||
|
||||
Any origin and port is fine — the backend builds its media URLs from the
|
||||
forwarded host/proto (`TRUST_PROXY_HEADERS=true` is set in all composes).
|
||||
|
||||
## Images
|
||||
|
||||
Two Dockerfiles, both built from the repository root:
|
||||
|
||||
```bash
|
||||
docker build -f deploy/J621-Frontend -t j621-frontend .
|
||||
docker build --build-arg "GIT_HASH=$(git rev-parse --short HEAD)" \
|
||||
-f deploy/J621-Backend -t j621-backend .
|
||||
```
|
||||
|
||||
`J621-Frontend` builds the SPA and serves it as static files.
|
||||
`J621-Backend` is Django + gunicorn (migrations run on start) and includes
|
||||
ffmpeg for video thumbnails. The `GIT_HASH` build arg is baked into the
|
||||
backend image so the shell's version pill shows the commit (images have no
|
||||
`.git` directory); the push scripts pass it automatically.
|
||||
|
||||
Push multi-arch images to the Gitea registry:
|
||||
|
||||
```bash
|
||||
./push_all.sh # or push_frontend.sh / push_backend.sh [sha]
|
||||
```
|
||||
|
||||
They tag `:latest` and `:<commit-sha>` and expect `docker login
|
||||
gitea.rainbow-herring.ts.net` to succeed.
|
||||
|
||||
## Tests
|
||||
|
||||
The security/permission suite lives in `backend/apps/core/tests/`:
|
||||
|
||||
```bash
|
||||
cd ../backend
|
||||
./venv/bin/python manage.py test apps.core.tests
|
||||
```
|
||||
|
||||
It needs a one-time grant on the database user (test databases are created
|
||||
from scratch):
|
||||
|
||||
```sql
|
||||
GRANT ALL ON `test_j621`.* TO 'j621'@'%';
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- The nginx service is the only entry point: `/api`, `/admin`, `/static` and
|
||||
`/health` go to the backend, everything else to the SPA. Both upstreams are
|
||||
resolved at request time, so the same config works in all three variants.
|
||||
- The compose healthchecks use `/nginx-health` (nginx itself), `/health`
|
||||
(Django + database) and the frontend's static server, so `depends_on:
|
||||
condition: service_healthy` gates the sidecar on a working stack.
|
||||
- `deploy/data/media/library` is the watched folder; drop files there (or use
|
||||
uploads) and run `manage.py scan_files` inside the backend container.
|
||||
Executable
+16
@@ -0,0 +1,16 @@
|
||||
#!/bin/sh
|
||||
# Apply migrations, then hand over to gunicorn.
|
||||
#
|
||||
# Workers/threads are modest on purpose: background jobs (downloads, scans)
|
||||
# live in daemon threads inside the worker and are reaped if a worker dies.
|
||||
set -e
|
||||
|
||||
python manage.py migrate --noinput
|
||||
|
||||
exec gunicorn config.wsgi:application \
|
||||
--bind 0.0.0.0:8000 \
|
||||
--workers "${GUNICORN_WORKERS:-2}" \
|
||||
--threads "${GUNICORN_THREADS:-4}" \
|
||||
--timeout "${GUNICORN_TIMEOUT:-120}" \
|
||||
--access-logfile - \
|
||||
--error-logfile -
|
||||
@@ -0,0 +1,109 @@
|
||||
# J621 — backend-only deployment: API + database + nginx proxy + Tailscale.
|
||||
#
|
||||
# cd deploy && cp .env.example .env # TS_AUTHKEY, SECRET_KEY, hosts, CORS
|
||||
# docker compose -f compose.backend.yml up -d
|
||||
#
|
||||
# Funnel: https://<TS_HOSTNAME>.<tailnet>.ts.net:8443 (an allowed funnel port)
|
||||
# reaches the nginx service, which routes /api, /admin, /static and /health to
|
||||
# Django. "/" answers a small JSON hint because no frontend is attached.
|
||||
#
|
||||
# Because the frontend lives elsewhere it is cross-origin — set in .env:
|
||||
# CORS_ALLOWED_ORIGINS=https://<frontend-host>.<tailnet>.ts.net
|
||||
# CSRF_TRUSTED_ORIGINS=... (same value; only needed for the admin)
|
||||
# and add this host to ALLOWED_HOSTS (comma separated list).
|
||||
|
||||
name: j621-backend-deploy
|
||||
|
||||
services:
|
||||
mariadb:
|
||||
image: mariadb:11.4
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD:-j621root}
|
||||
MARIADB_DATABASE: ${DB_NAME:-j621}
|
||||
MARIADB_USER: ${DB_USER:-j621}
|
||||
MARIADB_PASSWORD: ${DB_PASSWORD:-j621}
|
||||
volumes:
|
||||
- ./data/mariadb:/var/lib/mysql
|
||||
healthcheck:
|
||||
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
command: ["redis-server", "--appendonly", "yes"]
|
||||
volumes:
|
||||
- ./data/redis:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
backend:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-backend:${J621_TAG:-latest}
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/J621-Backend
|
||||
# Bake the commit into the image so the shell's version pill shows it;
|
||||
# the push scripts pass --build-arg themselves.
|
||||
args:
|
||||
GIT_HASH: ${GIT_HASH:-unknown}
|
||||
restart: unless-stopped
|
||||
env_file: [./.env]
|
||||
environment:
|
||||
DB_HOST: mariadb
|
||||
DB_PORT: "3306"
|
||||
REDIS_URL: redis://redis:6379/1
|
||||
TRUST_PROXY_HEADERS: "true"
|
||||
volumes:
|
||||
- ./data/media:/app/media
|
||||
- ./data/logs:/app/logs
|
||||
depends_on:
|
||||
mariadb:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
nginx:
|
||||
image: nginx:1.29-alpine
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./nginx-proxy.conf:/etc/nginx/conf.d/default.conf:ro
|
||||
depends_on:
|
||||
backend:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q --spider http://127.0.0.1/nginx-health || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
tailscale:
|
||||
image: tailscale/tailscale:latest
|
||||
restart: unless-stopped
|
||||
# Shares the nginx service's network namespace: 127.0.0.1:80 is the proxy.
|
||||
network_mode: "service:nginx"
|
||||
environment:
|
||||
TS_AUTHKEY: ${TS_AUTHKEY:?Set TS_AUTHKEY in deploy/.env}
|
||||
TS_HOSTNAME: ${TS_HOSTNAME:-j621-backend}
|
||||
TS_AUTH_ONCE: "true"
|
||||
TS_STATE_DIR: /var/lib/tailscale
|
||||
TS_SERVE_CONFIG: /config/serve.json
|
||||
volumes:
|
||||
- ./tailscale-state:/var/lib/tailscale
|
||||
- ./serve.backend.json:/config/serve.json:ro
|
||||
- /etc/ssl/certs:/etc/ssl/certs:ro
|
||||
depends_on:
|
||||
nginx:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "tailscale", "status"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
@@ -0,0 +1,62 @@
|
||||
# J621 — frontend-only deployment: SPA + nginx proxy + Tailscale sidecar.
|
||||
#
|
||||
# cd deploy && cp .env.example .env # TS_AUTHKEY at minimum
|
||||
# docker compose -f compose.frontend.yml up -d
|
||||
#
|
||||
# Funnel: https://<TS_HOSTNAME>.<tailnet>.ts.net (443) serves the SPA.
|
||||
# On first start the SPA asks for a backend (/setup): point it at a
|
||||
# backend-only deployment (e.g. https://j621-backend.<tailnet>.ts.net:8443),
|
||||
# leave it blank to serve one yourself, or stay in local mode.
|
||||
#
|
||||
# There is no backend in this compose, so /api answers 502 here until the SPA
|
||||
# is configured to call one elsewhere.
|
||||
|
||||
name: j621-frontend-deploy
|
||||
|
||||
services:
|
||||
frontend:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-frontend:${J621_TAG:-latest}
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/J621-Frontend
|
||||
restart: unless-stopped
|
||||
|
||||
nginx:
|
||||
image: nginx:1.29-alpine
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./nginx-proxy.conf:/etc/nginx/conf.d/default.conf:ro
|
||||
depends_on:
|
||||
frontend:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q --spider http://127.0.0.1/nginx-health || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
tailscale:
|
||||
image: tailscale/tailscale:latest
|
||||
restart: unless-stopped
|
||||
# Shares the nginx service's network namespace: 127.0.0.1:80 is the proxy.
|
||||
network_mode: "service:nginx"
|
||||
environment:
|
||||
TS_AUTHKEY: ${TS_AUTHKEY:?Set TS_AUTHKEY in deploy/.env}
|
||||
TS_HOSTNAME: ${TS_HOSTNAME:-j621-frontend}
|
||||
TS_AUTH_ONCE: "true"
|
||||
TS_STATE_DIR: /var/lib/tailscale
|
||||
TS_SERVE_CONFIG: /config/serve.json
|
||||
volumes:
|
||||
- ./tailscale-state:/var/lib/tailscale
|
||||
- ./serve.frontend.json:/config/serve.json:ro
|
||||
- /etc/ssl/certs:/etc/ssl/certs:ro
|
||||
depends_on:
|
||||
nginx:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "tailscale", "status"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
@@ -0,0 +1,116 @@
|
||||
# J621 — default deployment: frontend + backend + database on one Tailscale
|
||||
# host, behind the nginx proxy service (no host nginx, nothing published on
|
||||
# the host's ports).
|
||||
#
|
||||
# cd deploy && cp .env.example .env # fill in TS_AUTHKEY, SECRET_KEY, ...
|
||||
# docker compose up -d # or: docker compose -f compose.yml up -d
|
||||
#
|
||||
# The funnel serves https://<TS_HOSTNAME>.<tailnet>.ts.net (port 443) to the
|
||||
# nginx service, which routes /api, /admin, /static and /health to the
|
||||
# backend and everything else to the SPA. Same origin, so no CORS needed.
|
||||
|
||||
name: j621-deploy
|
||||
|
||||
services:
|
||||
mariadb:
|
||||
image: mariadb:11.4
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD:-j621root}
|
||||
MARIADB_DATABASE: ${DB_NAME:-j621}
|
||||
MARIADB_USER: ${DB_USER:-j621}
|
||||
MARIADB_PASSWORD: ${DB_PASSWORD:-j621}
|
||||
volumes:
|
||||
- ./data/mariadb:/var/lib/mysql
|
||||
healthcheck:
|
||||
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
command: ["redis-server", "--appendonly", "yes"]
|
||||
volumes:
|
||||
- ./data/redis:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
backend:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-backend:${J621_TAG:-latest}
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/J621-Backend
|
||||
# Bake the commit into the image so the shell's version pill shows it;
|
||||
# the push scripts pass --build-arg themselves.
|
||||
args:
|
||||
GIT_HASH: ${GIT_HASH:-unknown}
|
||||
restart: unless-stopped
|
||||
env_file: [./.env]
|
||||
environment:
|
||||
DB_HOST: mariadb
|
||||
DB_PORT: "3306"
|
||||
REDIS_URL: redis://redis:6379/1
|
||||
# The tailnet funnel terminates TLS and forwards the original host/proto.
|
||||
TRUST_PROXY_HEADERS: "true"
|
||||
volumes:
|
||||
- ./data/media:/app/media
|
||||
- ./data/logs:/app/logs
|
||||
depends_on:
|
||||
mariadb:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
frontend:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-frontend:${J621_TAG:-latest}
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/J621-Frontend
|
||||
restart: unless-stopped
|
||||
|
||||
nginx:
|
||||
image: nginx:1.29-alpine
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./nginx-proxy.conf:/etc/nginx/conf.d/default.conf:ro
|
||||
depends_on:
|
||||
backend:
|
||||
condition: service_healthy
|
||||
frontend:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q --spider http://127.0.0.1/nginx-health || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
tailscale:
|
||||
image: tailscale/tailscale:latest
|
||||
restart: unless-stopped
|
||||
# Shares the nginx service's network namespace: 127.0.0.1:80 is the proxy.
|
||||
network_mode: "service:nginx"
|
||||
environment:
|
||||
TS_AUTHKEY: ${TS_AUTHKEY:?Set TS_AUTHKEY in deploy/.env}
|
||||
TS_HOSTNAME: ${TS_HOSTNAME:-j621}
|
||||
TS_AUTH_ONCE: "true"
|
||||
TS_STATE_DIR: /var/lib/tailscale
|
||||
TS_SERVE_CONFIG: /config/serve.json
|
||||
volumes:
|
||||
- ./tailscale-state:/var/lib/tailscale
|
||||
- ./serve.default.json:/config/serve.json:ro
|
||||
- /etc/ssl/certs:/etc/ssl/certs:ro
|
||||
depends_on:
|
||||
nginx:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "tailscale", "status"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
@@ -0,0 +1,30 @@
|
||||
# Static file server for the built SPA (inside the frontend image).
|
||||
# Routing / API proxying is done by the separate nginx service.
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name _;
|
||||
|
||||
root /usr/share/nginx/html;
|
||||
index index.html;
|
||||
|
||||
gzip on;
|
||||
gzip_types text/css application/javascript application/json image/svg+xml;
|
||||
gzip_min_length 1024;
|
||||
|
||||
location = /nginx-health {
|
||||
access_log off;
|
||||
return 200 "ok\n";
|
||||
}
|
||||
|
||||
# Client-side routes: everything that is not a real file gets index.html.
|
||||
location / {
|
||||
try_files $uri /index.html;
|
||||
}
|
||||
|
||||
# Hashed build assets can be cached hard.
|
||||
location /assets/ {
|
||||
expires 30d;
|
||||
add_header Cache-Control "public, immutable";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
# J621 public entry point (the "nginx" service in the compose files).
|
||||
#
|
||||
# Routes:
|
||||
# /api, /admin, /static, /health -> backend:8000 (Django / gunicorn)
|
||||
# everything else -> frontend:80 (SPA static files)
|
||||
#
|
||||
# Both upstreams are variables so the same file works in all three compose
|
||||
# variants: with no frontend on the network "/" answers a small JSON hint,
|
||||
# with no backend the API paths answer 502 until one is configured via /setup.
|
||||
# Nothing is published to the host; the Tailscale sidecar shares this
|
||||
# container's network namespace and funnels to 127.0.0.1:80.
|
||||
|
||||
resolver 127.0.0.11 valid=10s ipv6=off;
|
||||
|
||||
map $http_x_forwarded_proto $j621_forwarded_proto {
|
||||
default $http_x_forwarded_proto;
|
||||
"" $scheme;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name _;
|
||||
|
||||
location = /nginx-health {
|
||||
access_log off;
|
||||
return 200 "ok\n";
|
||||
}
|
||||
|
||||
location ~ ^/(api|admin|static|health)(/|$) {
|
||||
set $j621_backend http://backend:8000;
|
||||
proxy_pass $j621_backend$request_uri;
|
||||
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $j621_forwarded_proto;
|
||||
|
||||
# Uploads and client-processed files can be large; stream them.
|
||||
client_max_body_size 2048m;
|
||||
proxy_request_buffering off;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_send_timeout 600s;
|
||||
}
|
||||
|
||||
location / {
|
||||
set $j621_frontend http://frontend:80;
|
||||
proxy_pass $j621_frontend$request_uri;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $j621_forwarded_proto;
|
||||
|
||||
# Backend-only deployments have no frontend: say so instead of 502.
|
||||
error_page 502 503 504 = @j621_no_frontend;
|
||||
}
|
||||
|
||||
location @j621_no_frontend {
|
||||
default_type application/json;
|
||||
return 200 '{"detail":"J621 API - no frontend is attached to this deployment."}';
|
||||
}
|
||||
}
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/bin/bash
|
||||
# Build and push both J621 images (frontend + backend) to the Gitea registry.
|
||||
#
|
||||
# Usage: ./push_all.sh [commit-sha]
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
SHA="${1:-$(git rev-parse --short HEAD)}"
|
||||
|
||||
./push_frontend.sh "$SHA"
|
||||
./push_backend.sh "$SHA"
|
||||
|
||||
echo "==> Both images pushed with tag '$SHA' (and :latest)."
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/bin/bash
|
||||
# Build and push the J621 backend image (amd64 + arm64) to the private Gitea
|
||||
# registry as :latest and :<commit-sha>.
|
||||
#
|
||||
# Usage: ./push_backend.sh [commit-sha]
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
REGISTRY="gitea.rainbow-herring.ts.net/jakebreath/j621-backend"
|
||||
REGISTRY_HOST="$(printf '%s' "$REGISTRY" | cut -d/ -f1)"
|
||||
SHA="${1:-$(git rev-parse --short HEAD)}"
|
||||
BUILDER=multiarch
|
||||
PLATFORMS="linux/amd64,linux/arm64"
|
||||
|
||||
echo "==> Logging in to $REGISTRY_HOST ..."
|
||||
docker login "$REGISTRY_HOST"
|
||||
|
||||
if ! docker buildx inspect "$BUILDER" >/dev/null 2>&1; then
|
||||
echo "==> Creating buildx builder '$BUILDER' ..."
|
||||
docker buildx create --name "$BUILDER" --driver docker-container \
|
||||
--platform "$PLATFORMS" --bootstrap
|
||||
else
|
||||
docker buildx inspect --bootstrap "$BUILDER" >/dev/null
|
||||
fi
|
||||
|
||||
echo "==> Building + pushing $PLATFORMS -> $REGISTRY:{latest,$SHA} ..."
|
||||
docker buildx build --builder "$BUILDER" --push \
|
||||
--platform "$PLATFORMS" \
|
||||
--build-arg "GIT_HASH=$SHA" \
|
||||
-f deploy/J621-Backend \
|
||||
-t "$REGISTRY:latest" \
|
||||
-t "$REGISTRY:$SHA" \
|
||||
.
|
||||
|
||||
echo "==> Done. On the deploy host:"
|
||||
echo " docker login $REGISTRY_HOST && docker compose pull && docker compose up -d"
|
||||
Executable
+35
@@ -0,0 +1,35 @@
|
||||
#!/bin/bash
|
||||
# Build and push the J621 frontend image (amd64 + arm64) to the private Gitea
|
||||
# registry as :latest and :<commit-sha>.
|
||||
#
|
||||
# Usage: ./push_frontend.sh [commit-sha]
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
REGISTRY="gitea.rainbow-herring.ts.net/jakebreath/j621-frontend"
|
||||
REGISTRY_HOST="$(printf '%s' "$REGISTRY" | cut -d/ -f1)"
|
||||
SHA="${1:-$(git rev-parse --short HEAD)}"
|
||||
BUILDER=multiarch
|
||||
PLATFORMS="linux/amd64,linux/arm64"
|
||||
|
||||
echo "==> Logging in to $REGISTRY_HOST ..."
|
||||
docker login "$REGISTRY_HOST"
|
||||
|
||||
if ! docker buildx inspect "$BUILDER" >/dev/null 2>&1; then
|
||||
echo "==> Creating buildx builder '$BUILDER' ..."
|
||||
docker buildx create --name "$BUILDER" --driver docker-container \
|
||||
--platform "$PLATFORMS" --bootstrap
|
||||
else
|
||||
docker buildx inspect --bootstrap "$BUILDER" >/dev/null
|
||||
fi
|
||||
|
||||
echo "==> Building + pushing $PLATFORMS -> $REGISTRY:{latest,$SHA} ..."
|
||||
docker buildx build --builder "$BUILDER" --push \
|
||||
--platform "$PLATFORMS" \
|
||||
-f deploy/J621-Frontend \
|
||||
-t "$REGISTRY:latest" \
|
||||
-t "$REGISTRY:$SHA" \
|
||||
.
|
||||
|
||||
echo "==> Done. On the deploy host:"
|
||||
echo " docker login $REGISTRY_HOST && docker compose pull && docker compose up -d"
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"TCP": {
|
||||
"8443": {
|
||||
"HTTPS": true
|
||||
}
|
||||
},
|
||||
"Web": {
|
||||
"${TS_CERT_DOMAIN}:8443": {
|
||||
"Handlers": {
|
||||
"/": {
|
||||
"Proxy": "http://127.0.0.1:80"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"AllowFunnel": {
|
||||
"${TS_CERT_DOMAIN}:8443": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"TCP": {
|
||||
"443": {
|
||||
"HTTPS": true
|
||||
}
|
||||
},
|
||||
"Web": {
|
||||
"${TS_CERT_DOMAIN}:443": {
|
||||
"Handlers": {
|
||||
"/": {
|
||||
"Proxy": "http://127.0.0.1:80"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"AllowFunnel": {
|
||||
"${TS_CERT_DOMAIN}:443": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"TCP": {
|
||||
"443": {
|
||||
"HTTPS": true
|
||||
}
|
||||
},
|
||||
"Web": {
|
||||
"${TS_CERT_DOMAIN}:443": {
|
||||
"Handlers": {
|
||||
"/": {
|
||||
"Proxy": "http://127.0.0.1:80"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"AllowFunnel": {
|
||||
"${TS_CERT_DOMAIN}:443": true
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user