From 8ebda6ab20616ab5d0835a2da920e66da2059715 Mon Sep 17 00:00:00 2001 From: JakeBreath Date: Fri, 18 Sep 2026 00:51:29 -0500 Subject: [PATCH] Docker deployment (2 images, 3 composes, Tailscale funnels) + committed security suite Test suite (the manual audit harness, now a real test): - backend/apps/core/tests/test_security.py: 20 transactional tests across guest visibility, object ownership, staged-upload/similarity privacy, staff role boundaries, deletion rules, encrypted credentials, throttling and the download allowlist. Uses temp media folders and clears cache. Needs a one-time GRANT on test_j621 (documented in the module + README). Images (deploy/J621-Frontend, deploy/J621-Backend, repo root as context): - Frontend: node build -> static nginx with SPA fallback, asset caching and an internal health endpoint. - Backend: gunicorn + whitenoise (admin static collected at build), ffmpeg for video thumbnails, migrations applied on start, GIT_HASH build arg so the shell's version pill shows the commit. Composes (distinct project names so they coexist with the dev stack): - compose.yml (both), compose.frontend.yml, compose.backend.yml. - A shared nginx proxy service is the only entry point (no host nginx, no published host ports): /api,/admin,/static,/health -> backend, everything else -> SPA; both upstreams resolve at request time so one config serves all variants. - A Tailscale sidecar per compose shares the nginx network namespace; serve.default/frontend/backend.json use funnel ports 443 and 8443 only (10000 is the remaining allowance) with ${TS_CERT_DOMAIN} substitution. Registry: push_frontend.sh / push_backend.sh / push_all.sh build multi-arch images and push :latest + : to the Gitea registry, following the existing Packs-site pattern. Docs: deploy/README.md + .env.example, ROADMAP section 6 updated, AGENTS.md deployment and test notes. Verified: all three composes validate, both nginx configs pass nginx -t, both images build, the combined stack boots against real MariaDB/Redis (migrations applied, /health ok, whitenoise serving admin static, env=prod, git hash baked in), the proxy serves the SPA and routes /api, and manage.py test apps.core.tests passes 20/20. --- AGENTS.md | 9 + ROADMAP.md | 24 +- backend/apps/core/tests/__init__.py | 0 backend/apps/core/tests/test_security.py | 449 +++++++++++++++++++++++ backend/config/settings.py | 22 +- backend/requirements.txt | 2 + deploy/.env.example | 67 ++++ deploy/.gitignore | 4 + deploy/J621-Backend | 40 ++ deploy/J621-Frontend | 24 ++ deploy/README.md | 100 +++++ deploy/backend-entrypoint.sh | 16 + deploy/compose.backend.yml | 109 ++++++ deploy/compose.frontend.yml | 62 ++++ deploy/compose.yml | 116 ++++++ deploy/nginx-frontend.conf | 30 ++ deploy/nginx-proxy.conf | 65 ++++ deploy/push_all.sh | 13 + deploy/push_backend.sh | 36 ++ deploy/push_frontend.sh | 35 ++ deploy/serve.backend.json | 19 + deploy/serve.default.json | 19 + deploy/serve.frontend.json | 19 + 23 files changed, 1271 insertions(+), 9 deletions(-) create mode 100644 backend/apps/core/tests/__init__.py create mode 100644 backend/apps/core/tests/test_security.py create mode 100644 deploy/.env.example create mode 100644 deploy/.gitignore create mode 100644 deploy/J621-Backend create mode 100644 deploy/J621-Frontend create mode 100644 deploy/README.md create mode 100755 deploy/backend-entrypoint.sh create mode 100644 deploy/compose.backend.yml create mode 100644 deploy/compose.frontend.yml create mode 100644 deploy/compose.yml create mode 100644 deploy/nginx-frontend.conf create mode 100644 deploy/nginx-proxy.conf create mode 100755 deploy/push_all.sh create mode 100755 deploy/push_backend.sh create mode 100755 deploy/push_frontend.sh create mode 100644 deploy/serve.backend.json create mode 100644 deploy/serve.default.json create mode 100644 deploy/serve.frontend.json diff --git a/AGENTS.md b/AGENTS.md index e51c7df..65f99b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,6 +31,15 @@ Project constraints (do not regress): MariaDB/Redis come from docker-compose.yml. - Deployment will be Docker-based (compose); do not add systemd/cron unit files for scheduling — use the container setup for timers/workers. +- deploy/ is the deployment source of truth: J621-Frontend (SPA on static + nginx) and J621-Backend (gunicorn + whitenoise, ffmpeg, migrations on + start), three compose variants (both / frontend-only / backend-only) behind + a shared nginx proxy service, and a Tailscale sidecar per compose whose + serve configs only use funnel ports 443 / 8443 / 10000. Images are pushed + to the Gitea registry with deploy/push_*.sh (multi-arch, :latest + :sha, + GIT_HASH baked in for the version pill). +- Security/permission tests live in backend/apps/core/tests and need a + one-time grant: GRANT ALL ON `test_j621`.* TO 'j621'@'%'; - Same-origin and cross-origin frontends both work, and the SPA can also run with no backend at all: a production build asks on first start (/setup) and stores the answer in localStorage — a URL, "" for same origin, or "none" diff --git a/ROADMAP.md b/ROADMAP.md index be5611a..26e9c94 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -145,14 +145,22 @@ Files now stage first and are resolved before entering the library. ## 6. Infrastructure -- [ ] Guest blacklist refresh on a timer (`refresh_guest_blacklist` via cron/systemd) -- [ ] Follow sync on a timer (`sync_followed_tags` + `sync_followed_pools` via - cron/systemd, e.g. every 30 minutes) -- [ ] Similarity temp cleanup on a timer (`cleanup_similarity` via cron; the - TTL also cleans lazily when new checks are created) -- [ ] Production setup: build the SPA, serve via Nginx (static + `/media` + - `/library`), systemd unit for Waitress -- [ ] Automated tests (backend API + frontend components) +- [ ] Guest blacklist refresh on a timer (in-container scheduler) +- [ ] Follow sync on a timer (in-container scheduler, e.g. every 30 minutes) +- [ ] Similarity temp cleanup on a timer (in-container scheduler; the TTL + also cleans lazily when new checks are created) +- [x] Production setup: two Docker images (SPA on static nginx, API on + gunicorn + whitenoise with ffmpeg) and three compose variants (both / + frontend-only / backend-only) behind a shared nginx proxy service, each + with a Tailscale sidecar and a funnel serve config (ports 443 / 8443 / + 10000 only). Multi-arch images are pushed to the Gitea registry by + `deploy/push_*.sh` +- [x] Security/permission test suite (`manage.py test apps.core.tests`): + guest visibility, object ownership, upload/similarity privacy, role + boundaries, deletion rules, encrypted credentials, throttling, SSRF + allowlist +- [ ] Broader automated tests (more backend API coverage + frontend + components) - [x] Backfill e621 metadata for items downloaded before metadata was stored — covered by the match scan (`/api/matches/` scope `all`, or per-item Recheck) and `manage.py match_e621 --scope all` diff --git a/backend/apps/core/tests/__init__.py b/backend/apps/core/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/apps/core/tests/test_security.py b/backend/apps/core/tests/test_security.py new file mode 100644 index 0000000..461bf6a --- /dev/null +++ b/backend/apps/core/tests/test_security.py @@ -0,0 +1,449 @@ +"""Security regression suite. + +Covers the checks that must never live on the SPA side alone: guest +visibility, object ownership, staged-upload/similarity privacy, staff role +boundaries, account deletion rules, encrypted credentials, throttling and +the SSRF allowlist for server-side downloads. + +Run: backend/venv/bin/python manage.py test apps.core.tests + +The database user needs rights to create the test database once, e.g.: + GRANT ALL ON `test_j621`.* TO 'j621'@'%'; +""" + +import hashlib +import json +import shutil +import tempfile +import time +from pathlib import Path + +from django.contrib.auth import get_user_model +from django.core import signing +from django.core.cache import cache +from django.core.files.uploadedfile import SimpleUploadedFile +from django.test import Client, TestCase, override_settings + +from rest_framework.authtoken.models import Token + +from apps.library import services +from apps.library.models import ( + DownloadTask, + MediaItem, + MediaLocation, + SimilarityCheck, + TempUpload, +) +from apps.library.services import MEDIA_FILE_SALT + +User = get_user_model() + + +def jpost(client, path, body=None): + return client.post(path, data=json.dumps(body or {}), content_type="application/json") + + +def jpatch(client, path, body=None): + return client.patch(path, data=json.dumps(body or {}), content_type="application/json") + + +class SecurityTestCase(TestCase): + """Shared fixtures: throwaway accounts for every role and temp folders.""" + + @classmethod + def setUpClass(cls): + super().setUpClass() + cls._tmp = tempfile.mkdtemp(prefix="j621-security-") + cls._media = Path(cls._tmp) / "media" + cls._watched = cls._media / "library" + cls._watched.mkdir(parents=True, exist_ok=True) + cls._settings = override_settings( + MEDIA_ROOT=str(cls._media), + WATCHED_FOLDER=str(cls._watched), + ) + cls._settings.enable() + + @classmethod + def tearDownClass(cls): + cls._settings.disable() + shutil.rmtree(cls._tmp, ignore_errors=True) + super().tearDownClass() + + def setUp(self): + cache.clear() # throttle counters and cached payloads + self.users = {} + for name, role in ( + ("sec-user", "user"), + ("sec-uploader", "uploader"), + ("sec-uploader2", "uploader"), + ("sec-staff", "staff"), + ("sec-staff2", "staff"), + ): + user = User.objects.create_user( + username=name, password="audit-pass-123456" + ) + user.role = role + user.save(update_fields=["role"]) + self.users[name] = user + self.admin = User.objects.create_superuser( + username="sec-admin", password="audit-pass-123456" + ) + self.tokens = { + name: Token.objects.create(user=user).key + for name, user in self.users.items() + } + self.tokens["sec-admin"] = Token.objects.create(user=self.admin).key + + def client_for(self, name): + client = Client() + client.defaults["HTTP_AUTHORIZATION"] = f"Token {self.tokens[name]}" + return client + + @property + def guest(self): + return Client() + + def make_item(self, label, *, hidden=False, owner=None, tags=None): + path = self._watched / f"{label}.bin" + path.write_bytes(b"audit-" + label.encode()) + item = MediaItem.objects.create( + md5=hashlib.md5(label.encode()).hexdigest(), + size=path.stat().st_size, + uploaded_by=owner, + tags=tags or [], + ) + MediaLocation.objects.create( + item=item, path=str(path), rel_path=path.name, mtime=time.time() + ) + if hidden: + MediaItem.objects.filter(pk=item.pk).update(hidden_from_guests=True) + item.refresh_from_db() + return item + + def old_signature(self, item, action="raw", age=3 * 86400): + """A valid signature minted `age` seconds ago.""" + real_time = signing.time + + class Backdated: + def time(self): + return real_time.time() - age + + try: + signing.time = Backdated() + return signing.dumps( + {"item": item.id, "user": self.users["sec-uploader"].id, "action": action}, + salt=MEDIA_FILE_SALT, + ) + finally: + signing.time = real_time + + +class GuestVisibilityTests(SecurityTestCase): + def setUp(self): + super().setUp() + self.visible = self.make_item("visible", owner=self.users["sec-uploader"]) + self.hidden = self.make_item( + "hidden", + hidden=True, + owner=self.users["sec-uploader"], + tags=["sec_audit_hidden_tag"], + ) + + def test_guest_list_and_retrieve_hide_protected_items(self): + response = self.guest.get( + f"/api/files/?j_ids=J-{self.visible.id},J-{self.hidden.id}" + ) + self.assertEqual( + {row["j_id"] for row in response.json()["results"]}, + {f"J-{self.visible.id}"}, + ) + self.assertEqual(self.guest.get(f"/api/files/J-{self.hidden.id}/").status_code, 404) + self.assertEqual( + self.guest.get(f"/api/files/J-{self.hidden.id}/raw/").status_code, 404 + ) + self.assertEqual( + self.guest.get(f"/api/files/J-{self.hidden.id}/thumbnail/").status_code, 404 + ) + self.assertEqual(self.guest.get(f"/api/files/{self.hidden.md5}/").status_code, 404) + self.assertEqual( + self.guest.get(f"/api/files/J-{self.hidden.id}/neighbors/").status_code, 404 + ) + + def test_guest_lookup_and_tag_cloud_hide_protected_items(self): + response = jpost( + self.guest, + "/api/files/lookup/", + {"md5s": [self.visible.md5, self.hidden.md5]}, + ) + self.assertEqual(response.json()["found"], [self.visible.md5]) + cloud = self.guest.get("/api/tags/cloud/").json()["tags"] + self.assertNotIn("sec_audit_hidden_tag", [entry["tag"] for entry in cloud]) + + def test_authenticated_users_and_signed_urls_see_protected_items(self): + uploader = self.client_for("sec-uploader") + self.assertEqual(uploader.get(f"/api/files/J-{self.hidden.id}/").status_code, 200) + signed = signing.dumps( + {"item": self.hidden.id, "user": self.users["sec-uploader"].id, "action": "raw"}, + salt=MEDIA_FILE_SALT, + ) + self.assertEqual( + self.guest.get(f"/api/files/J-{self.hidden.id}/raw/?sig={signed}").status_code, + 200, + ) + + def test_signature_integrity(self): + signed = signing.dumps( + {"item": self.hidden.id, "user": self.users["sec-uploader"].id, "action": "raw"}, + salt=MEDIA_FILE_SALT, + ) + raw = f"/api/files/J-{self.hidden.id}/raw/" + thumbnail = f"/api/files/J-{self.hidden.id}/thumbnail/" + # Tampered signature (same action). + self.assertEqual(self.guest.get(f"{raw}?sig={signed[:-4]}AAAA").status_code, 404) + # Valid signature, wrong action. + self.assertEqual(self.guest.get(f"{thumbnail}?sig={signed}").status_code, 404) + # Expired signature (minted three days ago). + expired = self.old_signature(self.hidden) + self.assertEqual(self.guest.get(f"{raw}?sig={expired}").status_code, 404) + + +class RoleBoundaryTests(SecurityTestCase): + def test_non_uploader_is_read_only(self): + user = self.client_for("sec-user") + self.assertEqual(user.get("/api/storage/").status_code, 403) + self.assertEqual(user.get("/api/duplicates/md5/").status_code, 403) + self.assertEqual( + jpost(user, "/api/duplicates/visual/", {"j_id": "J-1"}).status_code, 403 + ) + self.assertEqual(user.get("/api/uploads/").status_code, 403) + self.assertEqual(user.get("/api/online/downloads/").status_code, 403) + self.assertEqual(user.get("/api/matches/").status_code, 403) + self.assertEqual(user.get("/api/stats/").status_code, 403) + self.assertEqual(user.get("/api/users/").status_code, 403) + self.assertEqual(jpost(user, "/api/delete/", {"j_ids": []}).status_code, 403) + self.assertEqual(jpost(user, "/api/temp/clear/").status_code, 403) + + def test_uploader_and_staff_reach_their_tools(self): + uploader = self.client_for("sec-uploader") + staff = self.client_for("sec-staff") + self.assertEqual(uploader.get("/api/storage/").status_code, 200) + self.assertEqual(jpost(uploader, "/api/delete/", {"j_ids": []}).status_code, 200) + self.assertEqual(uploader.get("/api/duplicates/md5/").status_code, 200) + self.assertEqual(staff.get("/api/stats/").status_code, 200) + self.assertEqual(staff.get("/api/users/").status_code, 200) + + def test_only_admins_change_staff_roles(self): + staff = self.client_for("sec-staff") + user_id = self.users["sec-user"].id + self.assertEqual( + jpatch(staff, f"/api/users/{user_id}/", {"role": "staff"}).status_code, + 403, + ) + self.assertEqual( + jpatch( + staff, + f"/api/users/{self.users['sec-staff2'].id}/", + {"role": "user"}, + ).status_code, + 403, + ) + admin = self.client_for("sec-admin") + self.assertEqual( + jpatch(admin, f"/api/users/{user_id}/", {"role": "uploader"}).status_code, + 200, + ) + + def test_deletion_rules(self): + staff = self.client_for("sec-staff") + admin = self.client_for("sec-admin") + self.assertEqual( + staff.delete(f"/api/users/{self.users['sec-user'].id}/").status_code, 204 + ) + self.assertEqual( + staff.delete(f"/api/users/{self.users['sec-staff2'].id}/").status_code, 403 + ) + self.assertEqual( + staff.delete(f"/api/users/{self.users['sec-staff'].id}/").status_code, 400 + ) + self.assertEqual( + admin.delete(f"/api/users/{self.users['sec-staff2'].id}/").status_code, 204 + ) + + def test_deleting_a_user_keeps_library_items(self): + owner = self.users["sec-uploader2"] + item = self.make_item("orphan", owner=owner) + self.client_for("sec-admin").delete(f"/api/users/{owner.id}/") + item.refresh_from_db() + self.assertIsNone(item.uploaded_by_id) + + +class ItemOwnershipTests(SecurityTestCase): + def setUp(self): + super().setUp() + self.item = self.make_item("owned", owner=self.users["sec-uploader"]) + + def test_other_uploaders_cannot_touch_items(self): + other = self.client_for("sec-uploader2") + self.assertEqual( + jpost(other, f"/api/files/J-{self.item.id}/match/", {"post_id": "x"}).status_code, + 403, + ) + self.assertEqual( + jpost(other, f"/api/files/J-{self.item.id}/unlink/").status_code, 403 + ) + self.assertEqual( + other.post(f"/api/files/J-{self.item.id}/optimize/").status_code, 403 + ) + self.assertEqual( + jpatch(other, f"/api/files/J-{self.item.id}/", {"rating": "s"}).status_code, + 403, + ) + + def test_owner_and_staff_pass_the_permission_check(self): + for name in ("sec-uploader", "sec-staff"): + client = self.client_for(name) + # An invalid post id proves authorization happened: validation answers. + self.assertEqual( + jpost(client, f"/api/files/J-{self.item.id}/match/", {"post_id": "x"}).status_code, + 400, + ) + self.assertEqual( + client.post(f"/api/files/J-{self.item.id}/optimize/").status_code, 400 + ) + owner = self.client_for("sec-uploader") + self.assertEqual( + jpatch( + owner, + f"/api/files/J-{self.item.id}/", + {"rating": "", "tags": [], "notes": ""}, + ).status_code, + 200, + ) + + +class PrivacyTests(SecurityTestCase): + def test_staged_uploads_are_private(self): + temp = TempUpload.objects.create( + user=self.users["sec-uploader2"], + file=SimpleUploadedFile("staged.bin", b"staged"), + original_filename="staged.bin", + md5=hashlib.md5(b"staged").hexdigest(), + size=6, + ) + self.assertEqual( + self.client_for("sec-uploader2").get(f"/api/uploads/{temp.id}/file/").status_code, + 200, + ) + self.assertEqual( + self.client_for("sec-uploader").get(f"/api/uploads/{temp.id}/file/").status_code, + 404, + ) + self.assertEqual( + self.client_for("sec-staff").get(f"/api/uploads/{temp.id}/file/").status_code, + 200, + ) + self.assertEqual(self.guest.get(f"/api/uploads/{temp.id}/file/").status_code, 401) + self.assertNotIn( + str(temp.id), self.client_for("sec-uploader").get("/api/uploads/").content.decode() + ) + + def test_similarity_checks_are_private(self): + check = SimilarityCheck.objects.create( + user=self.users["sec-uploader2"], + file=SimpleUploadedFile("similar.bin", b"similar"), + original_filename="similar.bin", + md5=hashlib.md5(b"similar").hexdigest(), + size=7, + ) + self.assertEqual( + self.client_for("sec-uploader2").get(f"/api/similarity/{check.id}/").status_code, + 200, + ) + self.assertEqual( + self.client_for("sec-uploader").get(f"/api/similarity/{check.id}/").status_code, + 404, + ) + self.assertEqual( + self.client_for("sec-uploader").get(f"/api/similarity/{check.id}/file/").status_code, + 404, + ) + self.assertEqual( + self.client_for("sec-staff").get(f"/api/similarity/{check.id}/").status_code, + 200, + ) + + +class CredentialEncryptionTests(SecurityTestCase): + def test_keys_are_encrypted_at_rest_and_only_owners_see_them(self): + client = self.client_for("sec-user") + response = client.put( + "/api/auth/e621/", + data=json.dumps( + {"username": "tester", "api_key": "abc123def456", "base_url": "https://e621.net"} + ), + content_type="application/json", + ) + self.assertEqual(response.status_code, 200) + self.assertEqual(response.json()["api_key"], "abc123def456") + user = User.objects.get(pk=self.users["sec-user"].pk) + self.assertTrue(user.e621_api_key.startswith("enc:")) + self.assertNotIn("abc123def456", user.e621_api_key) + self.assertTrue(user.e621_configured) + # Other users never see it. + other = self.client_for("sec-staff") + self.assertNotIn("abc123def456", other.get("/api/auth/me/").content.decode()) + self.assertNotIn("abc123def456", other.get("/api/auth/e621/").content.decode()) + + def test_undecryptable_values_read_as_not_configured(self): + user = User.objects.get(pk=self.users["sec-user"].pk) + user.e621_username = "tester" + user.e621_api_key = "enc:not-a-valid-token" + user.save(update_fields=["e621_username", "e621_api_key"]) + user.refresh_from_db() + self.assertEqual(user.e621_api_key_plain, "") + self.assertFalse(user.e621_configured) + + +class ThrottleTests(SecurityTestCase): + def test_login_is_rate_limited(self): + codes = [ + self.guest.post( + "/api/auth/token/", + data=json.dumps({"username": "nobody", "password": "wrong"}), + content_type="application/json", + ).status_code + for _ in range(6) + ] + self.assertEqual(codes[:5], [400] * 5) + self.assertEqual(codes[5], 429) + + def test_anonymous_polling_is_not_throttled(self): + self.assertEqual( + {self.guest.get("/api/status/").status_code for _ in range(12)}, {200} + ) + + +class RemoteUrlTests(SecurityTestCase): + def test_allowlist(self): + for url in ("file:///etc/passwd", "http://127.0.0.1:1/", "https://evil.example/"): + with self.assertRaises(services.RemoteUrlError): + services.validate_remote_url(url) + self.assertEqual( + services.validate_remote_url("https://static1.e621.net/data/x.png"), + "https://static1.e621.net/data/x.png", + ) + + def test_download_creation_rejects_internal_urls(self): + uploader = self.client_for("sec-uploader") + for url in ("http://127.0.0.1:1/", "http://192.168.1.1/", "file:///etc/passwd"): + response = jpost( + uploader, "/api/online/downloads/", {"url": url, "filename": "x.bin"} + ) + self.assertEqual(response.status_code, 400, url) + self.assertEqual(DownloadTask.objects.count(), 0) + + def test_guest_proxy_rejects_internal_urls(self): + response = self.guest.get( + "/api/online/file/?url=http%3A%2F%2F127.0.0.1%3A1%2F" + ) + self.assertEqual(response.status_code, 400) diff --git a/backend/config/settings.py b/backend/config/settings.py index 5c9521c..21286d7 100644 --- a/backend/config/settings.py +++ b/backend/config/settings.py @@ -61,7 +61,14 @@ if os.getenv("TRUST_PROXY_HEADERS", "false").lower() == "true": def _git_commit_hash(): - """Short git hash of the running build, mirroring the original app.""" + """Short git hash of the running build, mirroring the original app. + + Containers rarely ship the .git directory, so an explicit GIT_COMMIT_HASH + environment variable (baked in at image build time) wins when present. + """ + configured = os.getenv("GIT_COMMIT_HASH", "").strip() + if configured: + return configured[:12] try: return subprocess.check_output( ["git", "rev-parse", "--short", "HEAD"], @@ -98,6 +105,9 @@ INSTALLED_APPS = [ MIDDLEWARE = [ "django.middleware.security.SecurityMiddleware", + # Serves the Django admin's static files in production (collected at + # image build time); harmless in dev. + "whitenoise.middleware.WhiteNoiseMiddleware", # Must sit above CommonMiddleware so preflights are answered early. "corsheaders.middleware.CorsMiddleware", "django.contrib.sessions.middleware.SessionMiddleware", @@ -173,6 +183,16 @@ USE_TZ = True STATIC_URL = "static/" STATIC_ROOT = BASE_DIR / "staticfiles" +STORAGES = { + "default": { + "BACKEND": "django.core.files.storage.FileSystemStorage", + }, + "staticfiles": { + # No manifest: works whether or not collectstatic ran (dev included). + "BACKEND": "whitenoise.storage.CompressedStaticFilesStorage", + }, +} + MEDIA_URL = "/media/" MEDIA_ROOT = BASE_DIR / "media" diff --git a/backend/requirements.txt b/backend/requirements.txt index 31dafb8..da4c7e9 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -3,6 +3,8 @@ djangorestframework>=3.17 django-filter>=25.1 django-cors-headers>=4.9 cryptography>=46.0 +gunicorn>=23.0 +whitenoise>=6.8 Pillow>=11.0 python-dotenv>=1.0 requests>=2.32 diff --git a/deploy/.env.example b/deploy/.env.example new file mode 100644 index 0000000..cafa2f3 --- /dev/null +++ b/deploy/.env.example @@ -0,0 +1,67 @@ +# J621 deployment environment — copy to deploy/.env and fill in. +# +# Compose reads this file for variable substitution AND passes it to the +# backend container (env_file), so everything here is visible to Django. + +# --------------------------------------------------------------------------- +# Tailscale +# --------------------------------------------------------------------------- +# Reusable, untagged auth key (Settings -> Keys -> Generate auth key, +# Reusable = on, Tags = none). Auto-approves the device. +TS_AUTHKEY= + +# Hostname this deployment gets on the tailnet; the funnel URL becomes +# https://..ts.net. Use distinct names per compose. +TS_HOSTNAME=j621 + +# --------------------------------------------------------------------------- +# Django +# --------------------------------------------------------------------------- +# Long random string. Signs media URLs and encrypts stored e621 API keys — +# rotating it invalidates both, so users re-enter their e621 key. +SECRET_KEY=change-me-to-a-long-random-string +DEBUG=False + +# Hosts Django may be reached on, comma separated, no scheme. Add the tailnet +# hostname of every compose that talks to this backend (and keep localhost / +# 127.0.0.1 for container health checks). +ALLOWED_HOSTS=j621.rainbow-herring.ts.net,localhost,127.0.0.1 + +# --------------------------------------------------------------------------- +# Cross-origin access (needed for the separate frontend + backend deploys) +# --------------------------------------------------------------------------- +# The origin the SPA is served from, e.g. https://j621-frontend..ts.net +# CORS_ALLOWED_ORIGINS= +# CSRF_TRUSTED_ORIGINS= + +# --------------------------------------------------------------------------- +# Database (created by the compose files; change the password) +# --------------------------------------------------------------------------- +DB_NAME=j621 +DB_USER=j621 +DB_PASSWORD=j621 +DB_ROOT_PASSWORD=j621root + +# --------------------------------------------------------------------------- +# Optional overrides +# --------------------------------------------------------------------------- +# GUNICORN_WORKERS=2 +# GUNICORN_THREADS=4 +# GUNICORN_TIMEOUT=120 + +# Rate limits (per IP anonymous, per account signed in) +# THROTTLE_ANON=120/min +# THROTTLE_USER=600/min +# THROTTLE_LOGIN=5/min +# THROTTLE_REGISTER=20/hour +# THROTTLE_E621_PROXY=60/hour + +# e621 media hosts the backend may fetch from (downloads, proxies) +# E621_MEDIA_HOSTS=static1.e621.net,static2.e621.net,static3.e621.net + +# Ephemeral similarity-check lifetime in minutes +# SIMILARITY_TTL_MINUTES=30 + +# Registry/tag used by the compose files and push scripts +# J621_REGISTRY=gitea.rainbow-herring.ts.net/jakebreath +# J621_TAG=latest diff --git a/deploy/.gitignore b/deploy/.gitignore new file mode 100644 index 0000000..56ab077 --- /dev/null +++ b/deploy/.gitignore @@ -0,0 +1,4 @@ +# Runtime data and state — never commit these. +.env +data/ +tailscale-state/ diff --git a/deploy/J621-Backend b/deploy/J621-Backend new file mode 100644 index 0000000..cfecb37 --- /dev/null +++ b/deploy/J621-Backend @@ -0,0 +1,40 @@ +# J621-Backend — Django + gunicorn, with migrations applied on start. +# +# Build context is the repository root, e.g.: +# docker build -f deploy/J621-Backend -t j621-backend . +# +# Needs MariaDB and Redis (see the compose files). ffmpeg is used for video +# thumbnails; media/logs live under the mounted volumes. + +# syntax=docker/dockerfile:1 + +FROM python:3.14-slim + +# Baked in by the push scripts so the shell's version pill shows the commit +# even though the image has no .git directory. +ARG GIT_HASH=unknown +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + GIT_COMMIT_HASH=$GIT_HASH + +WORKDIR /app + +RUN apt-get update \ + && apt-get install -y --no-install-recommends ffmpeg \ + && rm -rf /var/lib/apt/lists/* + +COPY backend/requirements.txt ./ +RUN pip install --no-cache-dir -r requirements.txt + +COPY backend/ ./ +COPY deploy/backend-entrypoint.sh /usr/local/bin/j621-entrypoint + +RUN chmod +x /usr/local/bin/j621-entrypoint \ + && mkdir -p /app/media /app/logs \ + && python manage.py collectstatic --noinput + +EXPOSE 8000 +HEALTHCHECK --interval=30s --timeout=5s --start-period=30s \ + CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health')" || exit 1 + +ENTRYPOINT ["j621-entrypoint"] diff --git a/deploy/J621-Frontend b/deploy/J621-Frontend new file mode 100644 index 0000000..79610c0 --- /dev/null +++ b/deploy/J621-Frontend @@ -0,0 +1,24 @@ +# J621-Frontend — builds the SPA and serves it as static files. +# +# Build context is the repository root, e.g.: +# docker build -f deploy/J621-Frontend -t j621-frontend . +# +# This container only serves the SPA (with the index fallback). The public +# entry point is the separate "nginx" reverse-proxy service in the compose +# files, so API routing stays in one place. + +# syntax=docker/dockerfile:1 + +FROM node:22-alpine AS build +WORKDIR /app +COPY frontend/package.json frontend/package-lock.json ./ +RUN npm ci --no-audit --no-fund +COPY frontend/ ./ +RUN npm run build + +FROM nginx:1.29-alpine +COPY deploy/nginx-frontend.conf /etc/nginx/conf.d/default.conf +COPY --from=build /app/dist /usr/share/nginx/html +EXPOSE 80 +HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \ + CMD wget -q --spider http://127.0.0.1/nginx-health || exit 1 diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 0000000..25194a6 --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,100 @@ +# J621 deployment (Docker + Tailscale) + +Three compose variants, all behind an **nginx** service and a **Tailscale** +sidecar. Nothing is published on the host's ports and no system nginx or +reverse proxy is involved: the sidecar shares the nginx service's network +namespace and Tailscale Serve/Funnel exposes it. + +| Compose | Services | Funnel | Serve config | +| --- | --- | --- | --- | +| `compose.yml` (default) | mariadb, redis, backend, frontend, nginx, tailscale | `https://..ts.net` → nginx → backend + SPA | `serve.default.json` | +| `compose.frontend.yml` | frontend, nginx, tailscale | `https://..ts.net` → nginx → SPA only | `serve.frontend.json` | +| `compose.backend.yml` | mariadb, redis, backend, nginx, tailscale | `https://..ts.net:8443` → nginx → API | `serve.backend.json` | + +Funnel can only expose ports **443**, **8443** and **10000**, so the separate +backend uses 8443. Change it in `serve.backend.json` (and `.env`) if you +prefer 10000. + +## Usage + +```bash +cd deploy +cp .env.example .env # fill in TS_AUTHKEY, SECRET_KEY, ALLOWED_HOSTS, ... +docker compose up -d # default: everything on one host +# or +docker compose -f compose.frontend.yml up -d +docker compose -f compose.backend.yml up -d +``` + +The images are pulled from the Gitea registry; append `--build` (or run the +push scripts) if you build locally. Data lives in `deploy/data/` +(`media/`, `logs/`, `mariadb/`, `redis/`) and the Tailscale node state in +`deploy/tailscale-state/`; both are git-ignored. + +## First start + +The SPA asks where the backend is (`/setup`) in production builds: + +- **Default compose** — leave the field blank (same origin) and everything + works through nginx. +- **Frontend-only + backend-only** — point the (cross-origin) SPA at the + backend's funnel URL, e.g. `https://j621-backend..ts.net:8443`, + and give the backend `CORS_ALLOWED_ORIGINS=https://..ts.net` + (plus `CSRF_TRUSTED_ORIGINS` for the admin). +- Without a backend at all, choose **"Continue without a backend"** to run in + local mode (e621 browsing only). + +Any origin and port is fine — the backend builds its media URLs from the +forwarded host/proto (`TRUST_PROXY_HEADERS=true` is set in all composes). + +## Images + +Two Dockerfiles, both built from the repository root: + +```bash +docker build -f deploy/J621-Frontend -t j621-frontend . +docker build --build-arg "GIT_HASH=$(git rev-parse --short HEAD)" \ + -f deploy/J621-Backend -t j621-backend . +``` + +`J621-Frontend` builds the SPA and serves it as static files. +`J621-Backend` is Django + gunicorn (migrations run on start) and includes +ffmpeg for video thumbnails. The `GIT_HASH` build arg is baked into the +backend image so the shell's version pill shows the commit (images have no +`.git` directory); the push scripts pass it automatically. + +Push multi-arch images to the Gitea registry: + +```bash +./push_all.sh # or push_frontend.sh / push_backend.sh [sha] +``` + +They tag `:latest` and `:` and expect `docker login +gitea.rainbow-herring.ts.net` to succeed. + +## Tests + +The security/permission suite lives in `backend/apps/core/tests/`: + +```bash +cd ../backend +./venv/bin/python manage.py test apps.core.tests +``` + +It needs a one-time grant on the database user (test databases are created +from scratch): + +```sql +GRANT ALL ON `test_j621`.* TO 'j621'@'%'; +``` + +## Notes + +- The nginx service is the only entry point: `/api`, `/admin`, `/static` and + `/health` go to the backend, everything else to the SPA. Both upstreams are + resolved at request time, so the same config works in all three variants. +- The compose healthchecks use `/nginx-health` (nginx itself), `/health` + (Django + database) and the frontend's static server, so `depends_on: + condition: service_healthy` gates the sidecar on a working stack. +- `deploy/data/media/library` is the watched folder; drop files there (or use + uploads) and run `manage.py scan_files` inside the backend container. diff --git a/deploy/backend-entrypoint.sh b/deploy/backend-entrypoint.sh new file mode 100755 index 0000000..f86ed54 --- /dev/null +++ b/deploy/backend-entrypoint.sh @@ -0,0 +1,16 @@ +#!/bin/sh +# Apply migrations, then hand over to gunicorn. +# +# Workers/threads are modest on purpose: background jobs (downloads, scans) +# live in daemon threads inside the worker and are reaped if a worker dies. +set -e + +python manage.py migrate --noinput + +exec gunicorn config.wsgi:application \ + --bind 0.0.0.0:8000 \ + --workers "${GUNICORN_WORKERS:-2}" \ + --threads "${GUNICORN_THREADS:-4}" \ + --timeout "${GUNICORN_TIMEOUT:-120}" \ + --access-logfile - \ + --error-logfile - diff --git a/deploy/compose.backend.yml b/deploy/compose.backend.yml new file mode 100644 index 0000000..55e0a42 --- /dev/null +++ b/deploy/compose.backend.yml @@ -0,0 +1,109 @@ +# J621 — backend-only deployment: API + database + nginx proxy + Tailscale. +# +# cd deploy && cp .env.example .env # TS_AUTHKEY, SECRET_KEY, hosts, CORS +# docker compose -f compose.backend.yml up -d +# +# Funnel: https://..ts.net:8443 (an allowed funnel port) +# reaches the nginx service, which routes /api, /admin, /static and /health to +# Django. "/" answers a small JSON hint because no frontend is attached. +# +# Because the frontend lives elsewhere it is cross-origin — set in .env: +# CORS_ALLOWED_ORIGINS=https://..ts.net +# CSRF_TRUSTED_ORIGINS=... (same value; only needed for the admin) +# and add this host to ALLOWED_HOSTS (comma separated list). + +name: j621-backend-deploy + +services: + mariadb: + image: mariadb:11.4 + restart: unless-stopped + environment: + MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD:-j621root} + MARIADB_DATABASE: ${DB_NAME:-j621} + MARIADB_USER: ${DB_USER:-j621} + MARIADB_PASSWORD: ${DB_PASSWORD:-j621} + volumes: + - ./data/mariadb:/var/lib/mysql + healthcheck: + test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] + interval: 5s + timeout: 5s + retries: 20 + + redis: + image: redis:7-alpine + restart: unless-stopped + command: ["redis-server", "--appendonly", "yes"] + volumes: + - ./data/redis:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 5s + timeout: 5s + retries: 20 + + backend: + image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-backend:${J621_TAG:-latest} + build: + context: .. + dockerfile: deploy/J621-Backend + # Bake the commit into the image so the shell's version pill shows it; + # the push scripts pass --build-arg themselves. + args: + GIT_HASH: ${GIT_HASH:-unknown} + restart: unless-stopped + env_file: [./.env] + environment: + DB_HOST: mariadb + DB_PORT: "3306" + REDIS_URL: redis://redis:6379/1 + TRUST_PROXY_HEADERS: "true" + volumes: + - ./data/media:/app/media + - ./data/logs:/app/logs + depends_on: + mariadb: + condition: service_healthy + redis: + condition: service_healthy + + nginx: + image: nginx:1.29-alpine + restart: unless-stopped + volumes: + - ./nginx-proxy.conf:/etc/nginx/conf.d/default.conf:ro + depends_on: + backend: + condition: service_healthy + healthcheck: + test: ["CMD-SHELL", "wget -q --spider http://127.0.0.1/nginx-health || exit 1"] + interval: 30s + timeout: 3s + retries: 3 + start_period: 10s + + tailscale: + image: tailscale/tailscale:latest + restart: unless-stopped + # Shares the nginx service's network namespace: 127.0.0.1:80 is the proxy. + network_mode: "service:nginx" + environment: + TS_AUTHKEY: ${TS_AUTHKEY:?Set TS_AUTHKEY in deploy/.env} + TS_HOSTNAME: ${TS_HOSTNAME:-j621-backend} + TS_AUTH_ONCE: "true" + TS_STATE_DIR: /var/lib/tailscale + TS_SERVE_CONFIG: /config/serve.json + volumes: + - ./tailscale-state:/var/lib/tailscale + - ./serve.backend.json:/config/serve.json:ro + - /etc/ssl/certs:/etc/ssl/certs:ro + depends_on: + nginx: + condition: service_healthy + healthcheck: + test: ["CMD", "tailscale", "status"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 30s diff --git a/deploy/compose.frontend.yml b/deploy/compose.frontend.yml new file mode 100644 index 0000000..32e0089 --- /dev/null +++ b/deploy/compose.frontend.yml @@ -0,0 +1,62 @@ +# J621 — frontend-only deployment: SPA + nginx proxy + Tailscale sidecar. +# +# cd deploy && cp .env.example .env # TS_AUTHKEY at minimum +# docker compose -f compose.frontend.yml up -d +# +# Funnel: https://..ts.net (443) serves the SPA. +# On first start the SPA asks for a backend (/setup): point it at a +# backend-only deployment (e.g. https://j621-backend..ts.net:8443), +# leave it blank to serve one yourself, or stay in local mode. +# +# There is no backend in this compose, so /api answers 502 here until the SPA +# is configured to call one elsewhere. + +name: j621-frontend-deploy + +services: + frontend: + image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-frontend:${J621_TAG:-latest} + build: + context: .. + dockerfile: deploy/J621-Frontend + restart: unless-stopped + + nginx: + image: nginx:1.29-alpine + restart: unless-stopped + volumes: + - ./nginx-proxy.conf:/etc/nginx/conf.d/default.conf:ro + depends_on: + frontend: + condition: service_healthy + healthcheck: + test: ["CMD-SHELL", "wget -q --spider http://127.0.0.1/nginx-health || exit 1"] + interval: 30s + timeout: 3s + retries: 3 + start_period: 10s + + tailscale: + image: tailscale/tailscale:latest + restart: unless-stopped + # Shares the nginx service's network namespace: 127.0.0.1:80 is the proxy. + network_mode: "service:nginx" + environment: + TS_AUTHKEY: ${TS_AUTHKEY:?Set TS_AUTHKEY in deploy/.env} + TS_HOSTNAME: ${TS_HOSTNAME:-j621-frontend} + TS_AUTH_ONCE: "true" + TS_STATE_DIR: /var/lib/tailscale + TS_SERVE_CONFIG: /config/serve.json + volumes: + - ./tailscale-state:/var/lib/tailscale + - ./serve.frontend.json:/config/serve.json:ro + - /etc/ssl/certs:/etc/ssl/certs:ro + depends_on: + nginx: + condition: service_healthy + healthcheck: + test: ["CMD", "tailscale", "status"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 30s diff --git a/deploy/compose.yml b/deploy/compose.yml new file mode 100644 index 0000000..6c177a5 --- /dev/null +++ b/deploy/compose.yml @@ -0,0 +1,116 @@ +# J621 — default deployment: frontend + backend + database on one Tailscale +# host, behind the nginx proxy service (no host nginx, nothing published on +# the host's ports). +# +# cd deploy && cp .env.example .env # fill in TS_AUTHKEY, SECRET_KEY, ... +# docker compose up -d # or: docker compose -f compose.yml up -d +# +# The funnel serves https://..ts.net (port 443) to the +# nginx service, which routes /api, /admin, /static and /health to the +# backend and everything else to the SPA. Same origin, so no CORS needed. + +name: j621-deploy + +services: + mariadb: + image: mariadb:11.4 + restart: unless-stopped + environment: + MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD:-j621root} + MARIADB_DATABASE: ${DB_NAME:-j621} + MARIADB_USER: ${DB_USER:-j621} + MARIADB_PASSWORD: ${DB_PASSWORD:-j621} + volumes: + - ./data/mariadb:/var/lib/mysql + healthcheck: + test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] + interval: 5s + timeout: 5s + retries: 20 + + redis: + image: redis:7-alpine + restart: unless-stopped + command: ["redis-server", "--appendonly", "yes"] + volumes: + - ./data/redis:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 5s + timeout: 5s + retries: 20 + + backend: + image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-backend:${J621_TAG:-latest} + build: + context: .. + dockerfile: deploy/J621-Backend + # Bake the commit into the image so the shell's version pill shows it; + # the push scripts pass --build-arg themselves. + args: + GIT_HASH: ${GIT_HASH:-unknown} + restart: unless-stopped + env_file: [./.env] + environment: + DB_HOST: mariadb + DB_PORT: "3306" + REDIS_URL: redis://redis:6379/1 + # The tailnet funnel terminates TLS and forwards the original host/proto. + TRUST_PROXY_HEADERS: "true" + volumes: + - ./data/media:/app/media + - ./data/logs:/app/logs + depends_on: + mariadb: + condition: service_healthy + redis: + condition: service_healthy + + frontend: + image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-frontend:${J621_TAG:-latest} + build: + context: .. + dockerfile: deploy/J621-Frontend + restart: unless-stopped + + nginx: + image: nginx:1.29-alpine + restart: unless-stopped + volumes: + - ./nginx-proxy.conf:/etc/nginx/conf.d/default.conf:ro + depends_on: + backend: + condition: service_healthy + frontend: + condition: service_healthy + healthcheck: + test: ["CMD-SHELL", "wget -q --spider http://127.0.0.1/nginx-health || exit 1"] + interval: 30s + timeout: 3s + retries: 3 + start_period: 10s + + tailscale: + image: tailscale/tailscale:latest + restart: unless-stopped + # Shares the nginx service's network namespace: 127.0.0.1:80 is the proxy. + network_mode: "service:nginx" + environment: + TS_AUTHKEY: ${TS_AUTHKEY:?Set TS_AUTHKEY in deploy/.env} + TS_HOSTNAME: ${TS_HOSTNAME:-j621} + TS_AUTH_ONCE: "true" + TS_STATE_DIR: /var/lib/tailscale + TS_SERVE_CONFIG: /config/serve.json + volumes: + - ./tailscale-state:/var/lib/tailscale + - ./serve.default.json:/config/serve.json:ro + - /etc/ssl/certs:/etc/ssl/certs:ro + depends_on: + nginx: + condition: service_healthy + healthcheck: + test: ["CMD", "tailscale", "status"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 30s diff --git a/deploy/nginx-frontend.conf b/deploy/nginx-frontend.conf new file mode 100644 index 0000000..737fc44 --- /dev/null +++ b/deploy/nginx-frontend.conf @@ -0,0 +1,30 @@ +# Static file server for the built SPA (inside the frontend image). +# Routing / API proxying is done by the separate nginx service. + +server { + listen 80; + server_name _; + + root /usr/share/nginx/html; + index index.html; + + gzip on; + gzip_types text/css application/javascript application/json image/svg+xml; + gzip_min_length 1024; + + location = /nginx-health { + access_log off; + return 200 "ok\n"; + } + + # Client-side routes: everything that is not a real file gets index.html. + location / { + try_files $uri /index.html; + } + + # Hashed build assets can be cached hard. + location /assets/ { + expires 30d; + add_header Cache-Control "public, immutable"; + } +} diff --git a/deploy/nginx-proxy.conf b/deploy/nginx-proxy.conf new file mode 100644 index 0000000..c9c8419 --- /dev/null +++ b/deploy/nginx-proxy.conf @@ -0,0 +1,65 @@ +# J621 public entry point (the "nginx" service in the compose files). +# +# Routes: +# /api, /admin, /static, /health -> backend:8000 (Django / gunicorn) +# everything else -> frontend:80 (SPA static files) +# +# Both upstreams are variables so the same file works in all three compose +# variants: with no frontend on the network "/" answers a small JSON hint, +# with no backend the API paths answer 502 until one is configured via /setup. +# Nothing is published to the host; the Tailscale sidecar shares this +# container's network namespace and funnels to 127.0.0.1:80. + +resolver 127.0.0.11 valid=10s ipv6=off; + +map $http_x_forwarded_proto $j621_forwarded_proto { + default $http_x_forwarded_proto; + "" $scheme; +} + +server { + listen 80; + server_name _; + + location = /nginx-health { + access_log off; + return 200 "ok\n"; + } + + location ~ ^/(api|admin|static|health)(/|$) { + set $j621_backend http://backend:8000; + proxy_pass $j621_backend$request_uri; + + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Proto $j621_forwarded_proto; + + # Uploads and client-processed files can be large; stream them. + client_max_body_size 2048m; + proxy_request_buffering off; + proxy_read_timeout 600s; + proxy_send_timeout 600s; + } + + location / { + set $j621_frontend http://frontend:80; + proxy_pass $j621_frontend$request_uri; + + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Proto $j621_forwarded_proto; + + # Backend-only deployments have no frontend: say so instead of 502. + error_page 502 503 504 = @j621_no_frontend; + } + + location @j621_no_frontend { + default_type application/json; + return 200 '{"detail":"J621 API - no frontend is attached to this deployment."}'; + } +} diff --git a/deploy/push_all.sh b/deploy/push_all.sh new file mode 100755 index 0000000..43df575 --- /dev/null +++ b/deploy/push_all.sh @@ -0,0 +1,13 @@ +#!/bin/bash +# Build and push both J621 images (frontend + backend) to the Gitea registry. +# +# Usage: ./push_all.sh [commit-sha] +set -euo pipefail +cd "$(dirname "$0")" + +SHA="${1:-$(git rev-parse --short HEAD)}" + +./push_frontend.sh "$SHA" +./push_backend.sh "$SHA" + +echo "==> Both images pushed with tag '$SHA' (and :latest)." diff --git a/deploy/push_backend.sh b/deploy/push_backend.sh new file mode 100755 index 0000000..b9d5130 --- /dev/null +++ b/deploy/push_backend.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# Build and push the J621 backend image (amd64 + arm64) to the private Gitea +# registry as :latest and :. +# +# Usage: ./push_backend.sh [commit-sha] +set -euo pipefail +cd "$(dirname "$0")/.." + +REGISTRY="gitea.rainbow-herring.ts.net/jakebreath/j621-backend" +REGISTRY_HOST="$(printf '%s' "$REGISTRY" | cut -d/ -f1)" +SHA="${1:-$(git rev-parse --short HEAD)}" +BUILDER=multiarch +PLATFORMS="linux/amd64,linux/arm64" + +echo "==> Logging in to $REGISTRY_HOST ..." +docker login "$REGISTRY_HOST" + +if ! docker buildx inspect "$BUILDER" >/dev/null 2>&1; then + echo "==> Creating buildx builder '$BUILDER' ..." + docker buildx create --name "$BUILDER" --driver docker-container \ + --platform "$PLATFORMS" --bootstrap +else + docker buildx inspect --bootstrap "$BUILDER" >/dev/null +fi + +echo "==> Building + pushing $PLATFORMS -> $REGISTRY:{latest,$SHA} ..." +docker buildx build --builder "$BUILDER" --push \ + --platform "$PLATFORMS" \ + --build-arg "GIT_HASH=$SHA" \ + -f deploy/J621-Backend \ + -t "$REGISTRY:latest" \ + -t "$REGISTRY:$SHA" \ + . + +echo "==> Done. On the deploy host:" +echo " docker login $REGISTRY_HOST && docker compose pull && docker compose up -d" diff --git a/deploy/push_frontend.sh b/deploy/push_frontend.sh new file mode 100755 index 0000000..fd69d38 --- /dev/null +++ b/deploy/push_frontend.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# Build and push the J621 frontend image (amd64 + arm64) to the private Gitea +# registry as :latest and :. +# +# Usage: ./push_frontend.sh [commit-sha] +set -euo pipefail +cd "$(dirname "$0")/.." + +REGISTRY="gitea.rainbow-herring.ts.net/jakebreath/j621-frontend" +REGISTRY_HOST="$(printf '%s' "$REGISTRY" | cut -d/ -f1)" +SHA="${1:-$(git rev-parse --short HEAD)}" +BUILDER=multiarch +PLATFORMS="linux/amd64,linux/arm64" + +echo "==> Logging in to $REGISTRY_HOST ..." +docker login "$REGISTRY_HOST" + +if ! docker buildx inspect "$BUILDER" >/dev/null 2>&1; then + echo "==> Creating buildx builder '$BUILDER' ..." + docker buildx create --name "$BUILDER" --driver docker-container \ + --platform "$PLATFORMS" --bootstrap +else + docker buildx inspect --bootstrap "$BUILDER" >/dev/null +fi + +echo "==> Building + pushing $PLATFORMS -> $REGISTRY:{latest,$SHA} ..." +docker buildx build --builder "$BUILDER" --push \ + --platform "$PLATFORMS" \ + -f deploy/J621-Frontend \ + -t "$REGISTRY:latest" \ + -t "$REGISTRY:$SHA" \ + . + +echo "==> Done. On the deploy host:" +echo " docker login $REGISTRY_HOST && docker compose pull && docker compose up -d" diff --git a/deploy/serve.backend.json b/deploy/serve.backend.json new file mode 100644 index 0000000..1b62b38 --- /dev/null +++ b/deploy/serve.backend.json @@ -0,0 +1,19 @@ +{ + "TCP": { + "8443": { + "HTTPS": true + } + }, + "Web": { + "${TS_CERT_DOMAIN}:8443": { + "Handlers": { + "/": { + "Proxy": "http://127.0.0.1:80" + } + } + } + }, + "AllowFunnel": { + "${TS_CERT_DOMAIN}:8443": true + } +} diff --git a/deploy/serve.default.json b/deploy/serve.default.json new file mode 100644 index 0000000..30a210b --- /dev/null +++ b/deploy/serve.default.json @@ -0,0 +1,19 @@ +{ + "TCP": { + "443": { + "HTTPS": true + } + }, + "Web": { + "${TS_CERT_DOMAIN}:443": { + "Handlers": { + "/": { + "Proxy": "http://127.0.0.1:80" + } + } + } + }, + "AllowFunnel": { + "${TS_CERT_DOMAIN}:443": true + } +} diff --git a/deploy/serve.frontend.json b/deploy/serve.frontend.json new file mode 100644 index 0000000..30a210b --- /dev/null +++ b/deploy/serve.frontend.json @@ -0,0 +1,19 @@ +{ + "TCP": { + "443": { + "HTTPS": true + } + }, + "Web": { + "${TS_CERT_DOMAIN}:443": { + "Handlers": { + "/": { + "Proxy": "http://127.0.0.1:80" + } + } + } + }, + "AllowFunnel": { + "${TS_CERT_DOMAIN}:443": true + } +}