Skip to main content

Worker Architecture

Current Design​

Urblock uses BullMQ for async job processing. All 16 queues (token deploy, tx submit, tx confirm, webhook send, etc.) are currently processed in-process within the main API server.

This simplifies deployment but means CPU-intensive operations (e.g., large batch deploys) compete with HTTP request handling.

To separate workers from the API:

1. Set WORKER_MODE environment variable​

# API-only (no queue processors)
WORKER_MODE=api

# Workers-only (no HTTP routes)
WORKER_MODE=workers

# Both (current behavior, suitable for dev/small deployments)
WORKER_MODE=both

2. Docker Compose setup​

services:
  api:
    build: ./apps/api
    environment:
      - WORKER_MODE=api
    ports:
      - "3000:3000"

  workers:
    build: ./apps/api
    environment:
      - WORKER_MODE=workers
    deploy:
      replicas: 2  # Scale workers independently

3. Implementation​

The QueueModule conditionally registers processors:

// In queue.module.ts
const isWorkerMode = process.env.WORKER_MODE !== 'api';
const processors = isWorkerMode
  ? [DeadLetterProcessor, TxSubmitProcessor, TxConfirmProcessor, ...]
  : [];

This allows the same codebase to run in both modes without code splitting.

Benefits​

  • Independent scaling: Run 5 workers while keeping 2 API replicas
  • Resource isolation: Worker OOMs don't crash the API
  • Deployment: Roll out worker changes without API downtime
  • Monitoring: Separate metrics/alerts for API latency vs queue throughput