Media Worker — Concurrency, Retries, and Shutdown

Architectural role

The media worker is a stateless, CPU-bound service:

• Pulls optimize-request events from Kafka.
• Uses local disk as ephemeral scratch space.
• Reads/writes object storage.
• Publishes optimize-response events (success or structured failure).

It should never hold authoritative user state — Postgres remains source of truth on the API side.

Concurrency model

Two-level parallelism

Kafka ReadMessage loop (single goroutine typical)
    │
    └──► ProcessMediaEvent()  ── spawns ──► processJobWithRetry() goroutine per message
                      │
                      └── acquire workerPool slot (buffered channel, cap = N)
                              └── processJob()  // FFmpeg / libvips
                              └── release slot

Config knobs (illustrative)

Backpressure: When pool saturated, new goroutines block on workerPool <- inside the retry loop — Kafka consumer may still read messages unless flow control is added upstream. Monitor consumer lag as the real backpressure signal.

Per-job lifecycle

job := { id, event, status=PENDING, retryCount=0 }
register activeJobs[job.id]

loop retryCount <= RETRY_COUNT:
    acquire pool
    err = processJob(job)  // download → transform → upload
    release pool
    if err == nil: status=COMPLETED; break
    retryCount++
    sleep(retryCount * 1 second)   // linear backoff

if still failed: status=FAILED
sendProcessingResult(job)          // always
unregister activeJobs[job.id]

processJob internals

1. MkdirAll(/tmp/{jobId})
2. DownloadFile(bucket, key) → input path
3. detectMediaType — image branch vs video branch
4. Upload N outputs with deterministic keys
5. defer RemoveAll(/tmp/{jobId})

Disk exhaustion: Large concurrent jobs × file size can fill /tmp. Mitigate: pool size cap, separate volume, job input size limits enforced at API gate.

Result publishing semantics

Always publish to response topic — even on failure:

{ "success": false, "error": "failed to download file from S3: ..." }

API consumer must handle success: false without throwing (avoid poison retry loops on API side).

If SendProcessingResult fails (broker down):

• Job marked complete locally.
• Object variants may exist in bucket without DB update — orphan variants until reconciliation.

Metric: kafka_publish_failures_total.

Retry vs Kafka redelivery

Idempotency: Deterministic output keys ({userId}/videos/{mediaId}/mp4/...) make retries safe (overwrite). Non-deterministic UUID suffixes in keys would create garbage on retry.

Video-specific partial failure

Thumbnail generation failure is often logged as warning while MP4 upload proceeds:

if thumbnailErr != nil:
    log.Warn(...)
    // continue

Document this for support teams — "video plays but no poster" is a known degraded state, not always a failed Kafka message.

HTTP colocation in same binary

Typical routes in the same process:

Isolation option: Split sync transcode to separate deployment so Kafka workers stay saturated without HTTP starvation.

Graceful shutdown sequence

On SIGTERM:

1. http.Server.Shutdown(30s) — stop new HTTP.
2. Cancel Kafka consumer context — stop reading new messages.
3. Poll activeJobs == 0 with sleep loop — drain in-flight transcodes.
4. Exit.

Kubernetes: Ensure terminationGracePeriodSeconds > worst-case transcode duration + 30s HTTP drain, or in-flight work is killed mid-FFmpeg.

Observability (OpenTelemetry pattern)

Span hierarchy example:

media.process_event
  └── media.process_job_with_retry
        └── media.process_job
              ├── image.compress | media.process_video
              └── kafka.send_processing_result

Attributes to rely on in dashboards: event.s3_key, media.type, job.retry_count, processing.success.

Failure mode matrix