Authentik 26.8.x expired runner
bruh
Authentik Troubleshooting Runbook: Stuck Task Foreign Keys, Corrupted Image Layers & unpigz Pull Errors
Environment: Docker Compose (postgresql, server, worker) - Authentik 2026.8.x
Compose directory: /home/alex/docker/authentik
Last updated: 2026-11-14
Overview
This runbook documents how to diagnose and recover from three related issues encountered in the Authentik Docker Compose stack:
- Dramatiq worker crash loop caused by a foreign key constraint violation on
authentik_tasks_task/authentik_tasks_tasklog. - Container startup failure caused by corrupted Python bytecode (
ValueError: bad marshal data) inside the cached Docker image. - Docker image pull failure caused by
unpigzreportingcorrupted -- crc32 mismatchduring layer extraction.
1. Worker Crash Loop: authentik_tasks_task Foreign Key Violation
Symptoms
authentik-workerrestarts every 3 seconds with aCRITICALDramatiq error:Consumer encountered a connection error: update or delete on table "authentik_tasks_task" violates foreign key constraint "authentik_tasks_task_task_id_a82f0835_fk_authentik" on table "authentik_tasks_tasklog" DETAIL: Key (message_id)=(<TASK_UUID>) is still referenced from table "authentik_tasks_tasklog".authentik-dblogs repeatedERROR+STATEMENT: COMMITmessages for the samemessage_id.
Cause
The worker's task broker (django_dramatiq_postgres) attempts to delete a completed/expired task from authentik_tasks_task, while child rows in authentik_tasks_tasklog (or authentik_tasks_taskdependency) still reference it. Because Django creates PostgreSQL foreign keys as DEFERRABLE INITIALLY DEFERRED, the constraint fails at COMMIT, rolling back the transaction and retrying in an infinite loop.
Warning: Do not drop the foreign key constraint (
authentik_tasks_task_task_id_a82f0835_fk_authentik) in PostgreSQL. Removing it breaks database integrity and can break future Authentik migrations.
Resolution
First, stop the worker so it is not contending on database locks:
cd /home/alex/docker/authentik
docker compose stop worker
Option A: Delete via Django ORM (server must be running)
Using ak shell automatically cascades deletions across all referencing child tables (authentik_tasks_tasklog, authentik_tasks_taskdependency, etc.). Replace <TASK_UUID> with the message_id from the logs:
docker compose exec server ak shell -c "
from authentik.tasks.models import Task
qs = Task.objects.filter(message_id='<TASK_UUID>')
print('Found:', list(qs.values('message_id', 'actor_name', 'state', 'mtime')))
print('Deleted:', qs.delete())
"
Option B: Delete directly via psql (use if server cannot start)
docker compose up -d postgresql
docker compose exec postgresql sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"'
Inside psql, inspect the task and delete child rows before the parent inside a transaction:
SELECT message_id, actor_name, state, mtime
FROM authentik_tasks_task
WHERE message_id = '<TASK_UUID>';
BEGIN;
DELETE FROM authentik_tasks_tasklog
WHERE task_id = '<TASK_UUID>';
DELETE FROM authentik_tasks_taskdependency
WHERE task_id = '<TASK_UUID>'
OR dependency_id = '<TASK_UUID>';
DELETE FROM authentik_tasks_task
WHERE message_id = '<TASK_UUID>';
COMMIT;
Exit psql with \q, then start the worker and monitor logs:
docker compose start worker
docker compose logs -f server worker
If the error immediately repeats with a different message_id
Clear expired completed tasks in small batches via ak shell while worker is stopped, and ensure Authentik is updated to the latest patch release in your branch:
docker compose stop worker
docker compose exec server ak shell -c "
from django.utils import timezone
from authentik.tasks.models import Task
while True:
pks = list(Task.objects.filter(state='done', result_expiry__lt=timezone.now()).values_list('pk', flat=True)[:1000])
if not pks:
break
print('Deleted batch:', Task.objects.filter(pk__in=pks).delete()[0])
"
docker compose start worker
2. Startup Crash: ValueError: bad marshal data (unknown type code)
Symptoms
server, worker, or ak shell crashes during Django/Python initialization while importing a module (e.g., botocore.utils):
File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
ValueError: bad marshal data (unknown type code)
Cause
A .pyc bytecode file inside the container's writable overlay or cached Docker image layer is corrupted (commonly caused by an unclean shutdown, storage glitch, or memory error).
Resolution
-
First try recreating the containers to clear any corrupted container writable layer:
docker compose up -d --force-recreate server worker -
If the error persists in both
serverandworker, the shared base image layer is corrupted. Remove the local image and re-pull it:docker compose rm -sf server worker docker compose config --images | grep authentik | xargs -r docker image rm -f docker compose pull server worker docker compose up -d docker compose logs -f server worker
3. Docker Pull Failure: unpigz: skipping: <stdin>: corrupted -- crc32 mismatch
Symptoms
docker compose pull downloads layers and fails during extraction:
failed to register layer: exit status 1: unpigz: skipping: <stdin>: corrupted -- crc32 mismatch
Cause
Docker automatically invokes /usr/bin/unpigz when present on the host. Certain pigz/unpigz builds can fail CRC32 validation on valid gzipped layers. When unpigz is absent, Docker falls back to its built-in Go gzip decompressor.
Resolution
-
Check disk space, inodes, and kernel logs first to rule out a full filesystem or disk I/O errors:
df -h /var/lib/docker df -i /var/lib/docker dmesg -T | grep -iE 'error|corrupt|ext4|xfs|btrfs|nvme|ata|i/o|mce|edac|segfault' -
Temporarily disable
unpigz, restart Docker, and re-pull:mv /usr/bin/unpigz /usr/bin/unpigz.disabled systemctl restart docker docker compose pull server worker docker compose up -d docker compose logs -f server worker
Note: If corruption errors or unexpected process segfaults (e.g.
libcrypto.so.3segfaults indmesg) appear across unrelated services even withoutunpigz, inspect host hardware health-specifically RAM (memtester/ MemTest86+), BIOS memory timings (XMP/EXPO), and disk SMART status (smartctl).
4. Recommended compose.yaml Hardening
To prevent server and worker from racing postgresql on host reboots, wait for the Postgres healthcheck instead of only container start:
depends_on:
postgresql:
condition: service_healthy
Documented: Gemini Hydrogen