Scaling background jobs in Rails without losing your weekend

Background jobs start simple: enqueue a task, process it later. The trouble arrives once volume grows and a single stuck queue starts delaying everything behind it.

Separate queues by latency, not by feature

The most common mistake is one queue per feature area. What actually matters is how quickly a job must run. A password reset email and a nightly report have nothing in common except that both are asynchronous.

• urgent — user-visible, must run within seconds
• default — everything that should finish within a few minutes
• low — reports, backfills and cleanup that can wait

Make every job idempotent

Retries are inevitable, so a job must be safe to run twice. Guard on a persisted state transition rather than assuming the job runs exactly once.

class ChargeInvoiceJob < ApplicationJob
  retry_on Net::OpenTimeout, wait: :polynomially_longer, attempts: 5

  def perform(invoice_id)
    invoice = Invoice.find(invoice_id)
    return if invoice.paid? # already handled by a previous attempt

    invoice.charge!
  end
end

That single guard clause removes an entire category of duplicate-charge incidents.