Laravel Jobs, Queues, Batches, and Redis: A Field Guide

Laravel’s queue system is one of those features you can use for years without really understanding what’s happening underneath. You call SomeJob::dispatch(), a worker somewhere picks it up, and life goes on. But the moment a job mysteriously runs twice, or your failed_jobs table fills up overnight, or Redis OOMs because of a job backlog you forgot about, you suddenly need to understand the moving parts. This is the field guide I wish I’d had. 🐘

What a “Job” Actually Is

A Laravel Job is a plain PHP class that represents a unit of background work — sending an email, generating a Portable Document Format (PDF) report, syncing a record to an external Application Programming Interface (API). You hand it to the queue, and a separate worker process picks it up and runs it later.

The cast of characters:

Job class — your code. A class that uses the Dispatchable trait and implements a handle() method.
Queue connection — where jobs are stored. Configured in config/queue.php. Common drivers: sync (run immediately, no queue), database, redis, sqs, beanstalkd.
Worker — a long-running PHP process started by php artisan queue:work. It pulls jobs off the queue and runs them.
Coordinator (optional) — Redis, when used as the driver, also acts as the lock store and pub/sub fabric for things like batches and unique jobs.

The Database Tables Laravel Uses

Even if you end up running on Redis, the database tables tell you what Laravel is conceptually tracking. The three you’ll see most often:

jobs — the pending queue. Used only when the queue driver is database. Each row is one serialized job payload waiting to be picked up.
failed_jobs — the graveyard. Used regardless of driver. When a job throws and exhausts its retry attempts, it lands here with its exception trace.
job_batches — batch metadata. One row per batch, tracking total_jobs, pending_jobs, failed_jobs, cancelled_at, and the serialized then / catch / finally callbacks.

You also indirectly touch the cache / cache_locks tables when you use job middleware like WithoutOverlapping or the ShouldBeUnique contract — but only if your cache driver is database. With Redis, those locks live in Redis instead, which is much faster under contention.

Create the tables with the built-in artisan generators:

1
2
3
4

php artisan queue:table
php artisan queue:failed-table
php artisan queue:batches-table
php artisan migrate

Each command publishes a migration; migrate applies them. ✨

Dispatching a Job

A minimal job class looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

<?php

namespace App\Jobs;

use App\Models\Invoice;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class SendInvoiceEmail implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

public int $tries = 3;
public int $timeout = 60;

public function __construct(public Invoice $invoice) {}

public function handle(): void
{
// ...send the email
}
}

Dispatching is one line:

1
2
3
4
5
6
7
8

SendInvoiceEmail::dispatch($invoice);

// or pin it to a specific queue / connection / delay
SendInvoiceEmail::dispatch($invoice)
->onConnection('redis')
->onQueue('high')
->delay(now()->addMinutes(5))
->afterCommit();

That afterCommit() at the end is one of the most important methods in this whole post. We’ll come back to it in the gotchas section. 💡

Batching With Bus::batch

Sometimes you have a thousand things to do and you want to know when they’re all done. That’s what Bus::batch is for:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;
use Throwable;

$batch = Bus::batch(
$invoices->map(fn ($invoice) => new SendInvoiceEmail($invoice))
)
->name('Send monthly invoices')
->allowFailures()
->onQueue('emails')
->then(function (Batch $batch) {
// All jobs completed successfully (or failures were allowed)
})
->catch(function (Batch $batch, Throwable $e) {
// First failure observed
})
->finally(function (Batch $batch) {
// Batch finished, success or not
})
->dispatch();

return $batch->id; // a UUID you can use to poll status later

Behind the scenes, Laravel inserts a row into job_batches with total_jobs = 1000, pending_jobs = 1000, and decrements pending_jobs as each child job completes. The then / catch / finally closures are serialized into the row and fired when the appropriate transition happens.

The allowFailures() call is important: without it, the first job that throws stops the rest of the batch from continuing. With it, every job runs and you can inspect $batch->failedJobs at the end.

Worker / Queue Command Lines

The headline command is queue:work. There’s also queue:listen, which restarts the framework on every job — useful only in local development if you want code changes to apply without restarting. In production, you always use queue:work under a process supervisor.

1
2
3
4
5
6
7
8
9

php artisan queue:work redis \
--queue=high,default,low \
--tries=3 \
--backoff=5,15,60 \
--timeout=60 \
--memory=256 \
--max-jobs=1000 \
--max-time=3600 \
--sleep=3

What each flag does:

–queue=high,default,low — pull from these queues in priority order. high drains first.
–tries=3 — retry a failing job twice before sending it to failed_jobs.
–backoff=5,15,60 — wait 5s before the first retry, 15s before the second, 60s before the third.
–timeout=60 — kill the job if it runs longer than 60 seconds.
–memory=256 — restart the worker if memory usage exceeds 256MB. Cheap insurance against leaks.
–max-jobs=1000 — restart the worker after processing 1000 jobs. Also cheap insurance.
–max-time=3600 — restart the worker after running for an hour.
–sleep=3 — when the queue is empty, sleep 3 seconds before polling again. Only matters for the database driver; Redis uses blocking pops.

The maintenance commands you’ll reach for:

1
2
3
4
5
6
7
8
9

php artisan queue:failed # list failed jobs
php artisan queue:retry all # retry every failed job
php artisan queue:retry 5 # retry job with id 5
php artisan queue:forget 5 # delete a failed job
php artisan queue:flush # delete ALL failed jobs (careful)
php artisan queue:prune-failed --hours=48 # delete failed jobs older than 48h
php artisan queue:prune-batches --hours=48 # delete finished batches older than 48h
php artisan queue:clear redis high # nuke all pending jobs on a queue
php artisan queue:restart # signal all workers to gracefully restart

In production, you almost always run workers under Supervisor. A typical config looks like this:

1
2
3
4
5
6
7
8
9
10
11
12

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work redis --queue=high,default --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/log/laravel/worker.log
stopwaitsecs=70

Note stopwaitsecs=70 — it should be greater than –timeout so Supervisor gives an in-flight job time to finish before sending SIGKILL.

Redis in the Mix

The database driver is fine for small applications, but it has two real costs: every poll is a SQL query (so –sleep matters), and lock contention on the jobs table grows nonlinearly with worker count. Switch to redis when you have more than a few workers or you need sub-second latency between dispatch and execution.

Behind the scenes, the Redis driver uses a handful of keys per queue. For a queue called default:

queues:default — the list of jobs waiting to be processed.
queues:default:delayed — a sorted set of jobs scheduled for the future. The score is the run-after timestamp.
queues:default:reserved — a sorted set of jobs currently being processed. The score is the lease expiration. If a worker dies, the lease expires and the job is re-queued.
queues:default:notify — a list used for blocking pops. This is the magic that makes Redis-backed queues feel instant: a worker does a BLPOP on this key and wakes up the moment a job is dispatched.

The atomic claim-a-job operation is implemented as a Lua script, so two workers can never reserve the same job. This is the part you really, really don’t want to reinvent yourself. 🛡️

For Redis-backed queues, the production answer is Laravel Horizon — a dashboard and process supervisor that replaces hand-rolled Supervisor configs. Install it, configure your queues and worker counts in config/horizon.php, and run:

1
2
3

php artisan horizon # start the master + workers
php artisan horizon:terminate # graceful shutdown for deploys
php artisan horizon:status

Horizon also gives you per-queue throughput graphs, recent job inspection, and runtime tagging — well worth the install if you’re already on Redis.

Gotchas (the actual reason you’re reading this)

These are the ones I’ve personally been bitten by. None are bugs; all are design tradeoffs you need to know.

1. Dispatching inside a database transaction

If you dispatch a job inside a DB::transaction(), a fast worker can pick it up before the transaction commits. The job then tries to load a row that doesn’t exist yet, and you get a ModelNotFoundException that’s impossible to reproduce locally.

Two fixes. Per dispatch:

1	SendInvoiceEmail::dispatch($invoice)->afterCommit();

Or globally, in config/queue.php:

1
2
3
4
5
6
7
8
9
10

'connections' => [
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'retry_after' => 90,
'block_for' => null,
'after_commit' => true, // <-- this
],
],

2. retry_after MUST be greater than –timeout

The worker’s –timeout is how long the worker waits before killing the job. The connection’s retry_after is how long Laravel waits before assuming the job is dead and putting it back on the queue. If retry_after is less than –timeout, the supervisor re-queues the job while the original worker is still happily running it. You get two simultaneous executions and any side effect (emails, charges, webhooks) happens twice.

Rule of thumb: retry_after = timeout + 30.

3. Code changes are NOT picked up automatically

The worker boots the framework once and holds it in memory forever. Deploying new code doesn’t change the running worker’s behavior at all. You must run php artisan queue:restart after every deploy. That command signals the workers to exit cleanly, and Supervisor (or Horizon) starts them again with the new code.

Many hours of confused debugging have been spent on this. Make it part of your deploy script and never think about it again.

4. Jobs serialize their constructor arguments

The SerializesModels trait stores only the model’s class and primary key, then re-fetches the row when the job runs. This is great — except if the row has been deleted between dispatch and execution, the job blows up with ModelNotFoundException.

For jobs that should tolerate a soft-deleted parent, either pass the ID and load with withTrashed(), or override getRestoredPropertyValue(). For jobs where the row really must exist, log loudly enough that the failure is obvious.

5. failed_jobs only catches thrown exceptions

A job that never throws but loops forever or silently no-ops never reaches failed_jobs. The only safety net is –timeout — set it. A timeout-killed job goes to failed_jobs with a MaxAttemptsExceededException after retries are exhausted.

6. Bus::batch does not respect afterCommit by default

The per-dispatch afterCommit() method does not exist on the batch builder. If you dispatch a batch inside a transaction, wrap it yourself:

1
2
3
4
5
6
7
8

DB::transaction(function () use ($invoices) {
// ...do transactional work

DB::afterCommit(function () use ($invoices) {
Bus::batch($invoices->map(fn ($i) => new SendInvoiceEmail($i)))
->dispatch();
});
});

7. Closures are not queueable out of the box

dispatch(function () { … }) looks tempting but it requires laravel/serializable-closure and a signed app key, and many edge cases (use statements, non-serializable captures) break it. Don’t. Write a real job class.

8. Redis memory pressure with large backlogs

Every queued job lives entirely in Redis memory until it runs. A million jobs at 5KB each is 5GB of Redis. If your payload is large or your backlog is unbounded, either trim the payload (pass an ID, fetch on the other side), or use the database driver for that specific queue where backlog matters more than latency.

9. Horizon’s auto-balancer can starve long-running queues

Horizon’s auto balancing strategy reallocates workers based on queue length every few seconds. If a queue’s jobs are long-running (say, 10-minute video transcodes), Horizon sees “queue empty, reassign workers elsewhere” and the next batch waits. Use the simple strategy for that supervisor, or pin a minimum worker count.

Closing Thoughts

The queue layer is the boring kind of magical: it works exactly the same on top of sync, database, redis, or sqs. You develop against database, ship to redis, and your code doesn’t change. The cost of that abstraction is knowing the gotchas above — none of them are bugs, but all of them have bitten me at least once, and the diagnosis is much faster the second time. 🎉

If you take three things away: use afterCommit() whenever you dispatch inside a transaction, always run queue:restart on deploy, and keep retry_after greater than your worker timeout. Everything else, you can learn the hard way. 🛠️