Run the periodic commands in a scheduler service (no host cron)
Adds deploy/scheduler-entrypoint.sh to the backend image (entrypoint j621-scheduler) and a scheduler service to the four composes that have a backend. It waits until the database and migrations are ready, runs every job once, then keeps to the intervals: sync_followed_tags + sync_followed_pools every 30 min (J621_SYNC_EVERY) cleanup_similarity hourly (J621_CLEAN_EVERY) refresh_guest_blacklist daily (J621_BLACKLIST_EVERY) It shares the backend image, media volume and env file, so commands see the same library and database; failures are logged and retried next interval. Output goes to docker compose logs scheduler; the frontend-only composes have no backend and therefore no scheduler. Verified against a real stack: the scheduler waited for migrations, ran all four commands on start (follow syncs as anonymous, similarity cleanup, and a guest blacklist refresh that pulled the real 14-tag list from e621), and kept looping. All six composes still validate.
This commit is contained in:
@@ -49,6 +49,12 @@ DB_ROOT_PASSWORD=j621root
|
||||
# GUNICORN_THREADS=4
|
||||
# GUNICORN_TIMEOUT=120
|
||||
|
||||
# Scheduler intervals in seconds (the "scheduler" service runs the periodic
|
||||
# management commands; see deploy/README.md)
|
||||
# J621_SYNC_EVERY=1800
|
||||
# J621_CLEAN_EVERY=3600
|
||||
# J621_BLACKLIST_EVERY=86400
|
||||
|
||||
# Rate limits (per IP anonymous, per account signed in)
|
||||
# THROTTLE_ANON=120/min
|
||||
# THROTTLE_USER=600/min
|
||||
|
||||
+2
-1
@@ -28,8 +28,9 @@ RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY backend/ ./
|
||||
COPY deploy/backend-entrypoint.sh /usr/local/bin/j621-entrypoint
|
||||
COPY deploy/scheduler-entrypoint.sh /usr/local/bin/j621-scheduler
|
||||
|
||||
RUN chmod +x /usr/local/bin/j621-entrypoint \
|
||||
RUN chmod +x /usr/local/bin/j621-entrypoint /usr/local/bin/j621-scheduler \
|
||||
&& mkdir -p /app/media /app/logs \
|
||||
&& python manage.py collectstatic --noinput
|
||||
|
||||
|
||||
@@ -123,6 +123,23 @@ Push multi-arch images to the Gitea registry:
|
||||
They tag `:latest` and `:<commit-sha>` and expect `docker login
|
||||
gitea.rainbow-herring.ts.net` to succeed.
|
||||
|
||||
## Scheduled jobs
|
||||
|
||||
Compose files with a backend also run a **`scheduler`** service — the same
|
||||
backend image with a different entrypoint, so no host cron is involved:
|
||||
|
||||
| Job | Default interval | Env override |
|
||||
| --- | --- | --- |
|
||||
| `sync_followed_tags` + `sync_followed_pools` | every 30 minutes | `J621_SYNC_EVERY` |
|
||||
| `cleanup_similarity` | hourly | `J621_CLEAN_EVERY` |
|
||||
| `refresh_guest_blacklist` | daily | `J621_BLACKLIST_EVERY` |
|
||||
|
||||
It waits for the database and migrations before its first run, runs every job
|
||||
once on start, then keeps to the intervals (failures are logged and retried
|
||||
next round). Output goes to `docker compose logs scheduler`. Intervals are
|
||||
seconds, set in `deploy/.env`. The frontend-only composes have no backend, so
|
||||
no scheduler.
|
||||
|
||||
## Tests
|
||||
|
||||
The security/permission suite lives in `backend/apps/core/tests/`:
|
||||
|
||||
@@ -68,6 +68,33 @@ services:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
# Periodic maintenance inside the deployment (no host cron): follow syncs,
|
||||
# similarity cleanup and the guest blacklist refresh.
|
||||
scheduler:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-backend:${J621_TAG:-latest}
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/J621-Backend
|
||||
args:
|
||||
GIT_HASH: ${GIT_HASH:-unknown}
|
||||
restart: unless-stopped
|
||||
entrypoint: ["j621-scheduler"]
|
||||
env_file: [./.env]
|
||||
environment:
|
||||
DB_HOST: mariadb
|
||||
DB_PORT: "3306"
|
||||
REDIS_URL: redis://redis:6379/1
|
||||
SYNC_EVERY: ${J621_SYNC_EVERY:-1800}
|
||||
CLEAN_EVERY: ${J621_CLEAN_EVERY:-3600}
|
||||
BLACKLIST_EVERY: ${J621_BLACKLIST_EVERY:-86400}
|
||||
volumes:
|
||||
- ./data/media:/app/media
|
||||
depends_on:
|
||||
mariadb:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
nginx:
|
||||
image: nginx:1.29-alpine
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -68,6 +68,33 @@ services:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
# Periodic maintenance inside the deployment (no host cron): follow syncs,
|
||||
# similarity cleanup and the guest blacklist refresh.
|
||||
scheduler:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-backend:${J621_TAG:-latest}
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/J621-Backend
|
||||
args:
|
||||
GIT_HASH: ${GIT_HASH:-unknown}
|
||||
restart: unless-stopped
|
||||
entrypoint: ["j621-scheduler"]
|
||||
env_file: [./.env]
|
||||
environment:
|
||||
DB_HOST: mariadb
|
||||
DB_PORT: "3306"
|
||||
REDIS_URL: redis://redis:6379/1
|
||||
SYNC_EVERY: ${J621_SYNC_EVERY:-1800}
|
||||
CLEAN_EVERY: ${J621_CLEAN_EVERY:-3600}
|
||||
BLACKLIST_EVERY: ${J621_BLACKLIST_EVERY:-86400}
|
||||
volumes:
|
||||
- ./data/media:/app/media
|
||||
depends_on:
|
||||
mariadb:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
nginx:
|
||||
image: nginx:1.29-alpine
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -67,6 +67,33 @@ services:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
# Periodic maintenance inside the deployment (no host cron): follow syncs,
|
||||
# similarity cleanup and the guest blacklist refresh.
|
||||
scheduler:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-backend:${J621_TAG:-latest}
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/J621-Backend
|
||||
args:
|
||||
GIT_HASH: ${GIT_HASH:-unknown}
|
||||
restart: unless-stopped
|
||||
entrypoint: ["j621-scheduler"]
|
||||
env_file: [./.env]
|
||||
environment:
|
||||
DB_HOST: mariadb
|
||||
DB_PORT: "3306"
|
||||
REDIS_URL: redis://redis:6379/1
|
||||
SYNC_EVERY: ${J621_SYNC_EVERY:-1800}
|
||||
CLEAN_EVERY: ${J621_CLEAN_EVERY:-3600}
|
||||
BLACKLIST_EVERY: ${J621_BLACKLIST_EVERY:-86400}
|
||||
volumes:
|
||||
- ./data/media:/app/media
|
||||
depends_on:
|
||||
mariadb:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
frontend:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-frontend:${J621_TAG:-latest}
|
||||
build:
|
||||
|
||||
@@ -66,6 +66,33 @@ services:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
# Periodic maintenance inside the deployment (no host cron): follow syncs,
|
||||
# similarity cleanup and the guest blacklist refresh.
|
||||
scheduler:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-backend:${J621_TAG:-latest}
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/J621-Backend
|
||||
args:
|
||||
GIT_HASH: ${GIT_HASH:-unknown}
|
||||
restart: unless-stopped
|
||||
entrypoint: ["j621-scheduler"]
|
||||
env_file: [./.env]
|
||||
environment:
|
||||
DB_HOST: mariadb
|
||||
DB_PORT: "3306"
|
||||
REDIS_URL: redis://redis:6379/1
|
||||
SYNC_EVERY: ${J621_SYNC_EVERY:-1800}
|
||||
CLEAN_EVERY: ${J621_CLEAN_EVERY:-3600}
|
||||
BLACKLIST_EVERY: ${J621_BLACKLIST_EVERY:-86400}
|
||||
volumes:
|
||||
- ./data/media:/app/media
|
||||
depends_on:
|
||||
mariadb:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
frontend:
|
||||
image: ${J621_REGISTRY:-gitea.rainbow-herring.ts.net/jakebreath}/j621-frontend:${J621_TAG:-latest}
|
||||
build:
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
#!/bin/sh
|
||||
# Periodic maintenance for a J621 deployment (runs in the "scheduler"
|
||||
# service; no host cron involved).
|
||||
#
|
||||
# sync_followed_tags + sync_followed_pools every SYNC_EVERY seconds (30 min)
|
||||
# cleanup_similarity every CLEAN_EVERY seconds (1 h)
|
||||
# refresh_guest_blacklist every BLACKLIST_EVERY (24 h)
|
||||
#
|
||||
# Waits for the database and migrations to be ready, runs everything once,
|
||||
# then keeps checking. A failing command is logged and retried next interval,
|
||||
# so a temporary e621 outage is not fatal.
|
||||
set -u
|
||||
|
||||
SYNC_EVERY="${SYNC_EVERY:-1800}"
|
||||
CLEAN_EVERY="${CLEAN_EVERY:-3600}"
|
||||
BLACKLIST_EVERY="${BLACKLIST_EVERY:-86400}"
|
||||
|
||||
log() { echo "[scheduler] $(date -Iseconds) $*"; }
|
||||
|
||||
run() {
|
||||
log "running: $*"
|
||||
if "$@"; then
|
||||
log "done: $*"
|
||||
else
|
||||
log "FAILED (retrying at the next interval): $*"
|
||||
fi
|
||||
}
|
||||
|
||||
command -v python >/dev/null || { log "python not found"; exit 1; }
|
||||
|
||||
log "waiting for the database and migrations..."
|
||||
until python manage.py migrate --check >/dev/null 2>&1; do
|
||||
sleep 10
|
||||
done
|
||||
log "database ready; intervals: sync=${SYNC_EVERY}s cleanup=${CLEAN_EVERY}s blacklist=${BLACKLIST_EVERY}s"
|
||||
|
||||
now=$(date +%s)
|
||||
last_sync=$((now - SYNC_EVERY))
|
||||
last_clean=$((now - CLEAN_EVERY))
|
||||
last_blacklist=$((now - BLACKLIST_EVERY))
|
||||
|
||||
while true; do
|
||||
now=$(date +%s)
|
||||
|
||||
if [ $((now - last_sync)) -ge "$SYNC_EVERY" ]; then
|
||||
last_sync=$now
|
||||
run python manage.py sync_followed_tags
|
||||
run python manage.py sync_followed_pools
|
||||
fi
|
||||
|
||||
if [ $((now - last_clean)) -ge "$CLEAN_EVERY" ]; then
|
||||
last_clean=$now
|
||||
run python manage.py cleanup_similarity
|
||||
fi
|
||||
|
||||
if [ $((now - last_blacklist)) -ge "$BLACKLIST_EVERY" ]; then
|
||||
last_blacklist=$now
|
||||
run python manage.py refresh_guest_blacklist
|
||||
fi
|
||||
|
||||
sleep 30
|
||||
done
|
||||
Reference in New Issue
Block a user