AlexVeeBee's Wiki

Authentik 26.8.x expired runner

bruh

Authentik Troubleshooting Runbook: Stuck Task Foreign Keys, Corrupted Image Layers & unpigz Pull Errors

Environment: Docker Compose (postgresql, server, worker) - Authentik 2026.8.x
Compose directory: /home/alex/docker/authentik
Last updated: 2026-11-14


Overview

This runbook documents how to diagnose and recover from three related issues encountered in the Authentik Docker Compose stack:

  1. Dramatiq worker crash loop caused by a foreign key constraint violation on authentik_tasks_task / authentik_tasks_tasklog.
  2. Container startup failure caused by corrupted Python bytecode (ValueError: bad marshal data) inside the cached Docker image.
  3. Docker image pull failure caused by unpigz reporting corrupted -- crc32 mismatch during layer extraction.

1. Worker Crash Loop: authentik_tasks_task Foreign Key Violation

Symptoms

Cause

The worker's task broker (django_dramatiq_postgres) attempts to delete a completed/expired task from authentik_tasks_task, while child rows in authentik_tasks_tasklog (or authentik_tasks_taskdependency) still reference it. Because Django creates PostgreSQL foreign keys as DEFERRABLE INITIALLY DEFERRED, the constraint fails at COMMIT, rolling back the transaction and retrying in an infinite loop.

Warning: Do not drop the foreign key constraint (authentik_tasks_task_task_id_a82f0835_fk_authentik) in PostgreSQL. Removing it breaks database integrity and can break future Authentik migrations.

Resolution

First, stop the worker so it is not contending on database locks:

cd /home/alex/docker/authentik
docker compose stop worker

Option A: Delete via Django ORM (server must be running)

Using ak shell automatically cascades deletions across all referencing child tables (authentik_tasks_tasklog, authentik_tasks_taskdependency, etc.). Replace <TASK_UUID> with the message_id from the logs:

docker compose exec server ak shell -c "
from authentik.tasks.models import Task
qs = Task.objects.filter(message_id='<TASK_UUID>')
print('Found:', list(qs.values('message_id', 'actor_name', 'state', 'mtime')))
print('Deleted:', qs.delete())
"

Option B: Delete directly via psql (use if server cannot start)

docker compose up -d postgresql
docker compose exec postgresql sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"'

Inside psql, inspect the task and delete child rows before the parent inside a transaction:

SELECT message_id, actor_name, state, mtime
FROM authentik_tasks_task
WHERE message_id = '<TASK_UUID>';

BEGIN;

DELETE FROM authentik_tasks_tasklog
WHERE task_id = '<TASK_UUID>';

DELETE FROM authentik_tasks_taskdependency
WHERE task_id = '<TASK_UUID>'
   OR dependency_id = '<TASK_UUID>';

DELETE FROM authentik_tasks_task
WHERE message_id = '<TASK_UUID>';

COMMIT;

Exit psql with \q, then start the worker and monitor logs:

docker compose start worker
docker compose logs -f server worker

If the error immediately repeats with a different message_id

Clear expired completed tasks in small batches via ak shell while worker is stopped, and ensure Authentik is updated to the latest patch release in your branch:

docker compose stop worker
docker compose exec server ak shell -c "
from django.utils import timezone
from authentik.tasks.models import Task

while True:
    pks = list(Task.objects.filter(state='done', result_expiry__lt=timezone.now()).values_list('pk', flat=True)[:1000])
    if not pks:
        break
    print('Deleted batch:', Task.objects.filter(pk__in=pks).delete()[0])
"
docker compose start worker

2. Startup Crash: ValueError: bad marshal data (unknown type code)

Symptoms

server, worker, or ak shell crashes during Django/Python initialization while importing a module (e.g., botocore.utils):

  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
ValueError: bad marshal data (unknown type code)

Cause

A .pyc bytecode file inside the container's writable overlay or cached Docker image layer is corrupted (commonly caused by an unclean shutdown, storage glitch, or memory error).

Resolution

  1. First try recreating the containers to clear any corrupted container writable layer:

    docker compose up -d --force-recreate server worker
    
  2. If the error persists in both server and worker, the shared base image layer is corrupted. Remove the local image and re-pull it:

    docker compose rm -sf server worker
    docker compose config --images | grep authentik | xargs -r docker image rm -f
    docker compose pull server worker
    docker compose up -d
    docker compose logs -f server worker
    

3. Docker Pull Failure: unpigz: skipping: <stdin>: corrupted -- crc32 mismatch

Symptoms

docker compose pull downloads layers and fails during extraction:

failed to register layer: exit status 1: unpigz: skipping: <stdin>: corrupted -- crc32 mismatch

Cause

Docker automatically invokes /usr/bin/unpigz when present on the host. Certain pigz/unpigz builds can fail CRC32 validation on valid gzipped layers. When unpigz is absent, Docker falls back to its built-in Go gzip decompressor.

Resolution

  1. Check disk space, inodes, and kernel logs first to rule out a full filesystem or disk I/O errors:

    df -h /var/lib/docker
    df -i /var/lib/docker
    dmesg -T | grep -iE 'error|corrupt|ext4|xfs|btrfs|nvme|ata|i/o|mce|edac|segfault'
    
  2. Temporarily disable unpigz, restart Docker, and re-pull:

    mv /usr/bin/unpigz /usr/bin/unpigz.disabled
    systemctl restart docker
    docker compose pull server worker
    docker compose up -d
    docker compose logs -f server worker
    

Note: If corruption errors or unexpected process segfaults (e.g. libcrypto.so.3 segfaults in dmesg) appear across unrelated services even without unpigz, inspect host hardware health-specifically RAM (memtester / MemTest86+), BIOS memory timings (XMP/EXPO), and disk SMART status (smartctl).


To prevent server and worker from racing postgresql on host reboots, wait for the Postgres healthcheck instead of only container start:

depends_on:
  postgresql:
    condition: service_healthy

Documented: Gemini Hydrogen