When a Composer package vanishes from GitHub: don’t panic, and don’t delete vendor/

Today our CI/CD pipeline went red on a job that hadn’t been touched in months. The .gitlab-ci.yml was untouched. The branch built fine yesterday. composer install exploded.

The relevant chunk of the failure log:

1
2
3
4
5
6
7
8
Failed to download acme/some-nova-tool from dist:
  https://api.github.com/repos/old-owner/some-nova-tool/zipball/24bd3d8...
  HTTP/2 404

In Git.php line 657:
  Failed to execute git clone --mirror -- https://github.com/old-owner/some-nova-tool.git ...
  remote: Repository not found.
  fatal: repository 'https://github.com/old-owner/some-nova-tool.git/' not found

The package itself was still listed on Packagist — but the canonical GitHub repo it points to had been deleted. Even better: both the original repo and the namespace-renamed fork it had been moved to were gone. Packagist had quietly marked the package as frozen with a tiny note: “This package’s canonical repository appears to be gone and the package has been frozen as a result.” 💀

The package was tiny (a Laravel Nova permissions tool) but load-bearing — twelve files in our codebase imported a trait from it, plus a service provider registration. Removing it was not an option for today.

Why local dev kept working

Here’s the part I want you to internalize before anything else: do not rm -rf vendor/ when you hit this kind of failure. Not on your laptop, not on the developer machine of whoever first reports the issue. 🛑

The vendor/ directory is your last copy of that package’s source code. Composer downloaded it months ago from a repository that, today, no longer exists. If you blow away vendor/ and re-run composer install, you will get the exact 404 the CI runner got, and now you have no way to recover the source short of finding a teammate whose vendor/ is still warm.

Tell your team the same thing. The instinct on a broken composer install is to nuke vendor/ and try again. That instinct is wrong here. Until you have a plan, treat the existing vendor/ tree like an artifact you’d lose forever if you deleted it — because that’s what it is.

The recovery: copy, fork, host it yourself

Once you have a backed-up copy of the package source, the recovery is straightforward. The shape of the fix:

  1. Copy the package source out of vendor/ into a scratch directory.
  2. Push it to a Git host you control (your company’s GitLab, a personal GitHub org, wherever).
  3. Tag a version on your fork.
  4. Tell composer.json to look at your fork instead of Packagist.

Step one and two:

1
2
3
4
5
6
7
8
9
mkdir /tmp/some-nova-tool && cd /tmp/some-nova-tool
cp -R ~/projects/myapp/vendor/acme/some-nova-tool/. .
git init -b main
git add .
git commit -m "Import acme/some-nova-tool source (upstream deleted)"
git tag v1.0.8-beta.0
git remote add origin https://gitlab.example.com/internal/some-nova-tool.git
git push -u origin main
git push origin v1.0.8-beta.0

A note on the tag. The locked commit in our composer.lock was on the dev-main branch, several months past the package’s last tagged release (v1.0.7). Rather than invent a v4.0.0 from thin air, I anchored the tag to actual upstream history: v1.0.8-beta.0 — “newer than 1.0.7, not stable, exact snapshot of where main was the day upstream disappeared.” The version string is arbitrary as long as it’s valid SemVer; pick one that won’t lie to a future reader. 🪦

Then in composer.json, add a VCS repository entry pointing at your fork and pin the version:

1
2
3
4
5
6
7
8
9
10
11
{
  "require": {
    "acme/some-nova-tool": "v1.0.8-beta.0"
  },
  "repositories": [
    {
      "type": "vcs",
      "url": "https://gitlab.example.com/internal/some-nova-tool.git"
    }
  ]
}

Crucially, keep the package name the same — acme/some-nova-tool. Composer’s package name and the autoload PSR-4 namespace are what your application code references. If you change the package name, every use Acme\SomeNovaTool\… statement across your codebase breaks. Keep the name; just change where Composer looks for it.

Regenerate the lockfile with the new source:

1
composer update acme/some-nova-tool --with-dependencies

Commit composer.json and composer.lock together and your CI runs green again. The next developer to composer install on a cold cache will pull from your fork and never know there was ever a problem.

Two small details that bit us

HTTPS vs SSH. Make sure the repository URL in composer.json is HTTPS, not SSH. Your laptop probably has an SSH key on the host; CI runners don’t, and they almost always authenticate via an HTTPS token (composer config –global gitlab-token.gitlab.example.com $TOKEN). One of them is in your shell config; the other has to work in a fresh container with only env vars. If they don’t agree, CI fails with auth errors that look nothing like the original 404.

Packagist will not save you. The package page may still resolve — the metadata lives on Packagist, not on GitHub — but the dist URL embedded in that metadata points at GitHub. Composer reads the dist URL, fetches it, gets a 404, falls back to a git clone, gets another 404, and gives up. Once the upstream Git host is gone, Packagist is just a tombstone. 🪦

The lessons, in one sentence each

  • Vendor is your backup. A populated vendor/ tree is the only copy of a deleted package you’ll ever have. Treat it like data, not cache.
  • Pin to tags, not branches. Tracking dev-main means “whatever HEAD is” — fine until HEAD is gone. A pinned tag on a fork you control is reproducible forever.
  • Self-host anything load-bearing. If a third-party package is woven into a dozen of your files, the cost of mirroring it on a Git host you control is one afternoon. The cost of not doing it is the day it disappears and your CI is red and you can’t ship.

Software supply-chain rot is a real thing. Repos get deleted, packages get unpublished, maintainers leave platforms, accounts get suspended. The defensive move costs almost nothing and pays off the one day you really need it. 🛡️

Posted in PHP | Tagged , , , , | Comments Off on When a Composer package vanishes from GitHub: don’t panic, and don’t delete vendor/

Local HTTPS in 5 minutes with Caddy 🔒

I used to dread setting up https for local development. Self-signed certs got the browser to scream. Editing nginx.conf for two hostnames felt like building a cathedral. Caddy changed all that for me — it’s a tiny single-binary web server that does automatic HTTPS out of the box. Point it at a hostname, and it either gets a real Let’s Encrypt cert (for public domains) or generates and trusts a local cert (for development) — without you running certbot, openssl req, or anything else.

This post is the cheat sheet I wish I’d had: install it, point it at a local app, get https in five minutes. ⏱️

Install

Caddy is a single static binary. The package managers wrap it nicely:

1
2
3
4
5
6
7
8
# macOS
brew install caddy

# Debian / Ubuntu
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install caddy

The Debian install also drops a caddy systemd service and a default Caddyfile at /etc/caddy/Caddyfile. On macOS, Homebrew puts the example Caddyfile under /opt/homebrew/etc/Caddyfile (Apple Silicon) or /usr/local/etc/Caddyfile (Intel).

The simplest possible Caddyfile

Caddy reads a config file called the Caddyfile — a tiny domain-specific format that maps hostnames to behaviours. The smallest useful one:

1
2
3
myapp.local {
    reverse_proxy localhost:8000
}

Three lines. “When something asks for myapp.local, terminate TLS and forward the plaintext request to localhost:8000.” Caddy generates a local certificate, installs the matching root CA into your system trust store the first time it runs (you’ll be prompted for a password), and serves https://myapp.local with a green padlock — provided myapp.local resolves to your machine. Add a line to /etc/hosts:

1
127.0.0.1   myapp.local

Run it:

1
2
3
4
5
6
7
8
# Foreground (good for trying it out)
caddy run --config /opt/homebrew/etc/Caddyfile

# As a background service (Debian / systemd)
sudo systemctl enable --now caddy

# Reload after editing the Caddyfile (no downtime)
sudo systemctl reload caddy

Bring your own cert

Sometimes you don’t want Caddy’s auto-generated cert — maybe you’ve already created one with mkcert, or you’ve been issued a cert by your team’s internal CA. Tell Caddy where the .pem files live with the tls directive:

1
2
3
4
myapp.example.com {
    tls /path/to/myapp.example.com.pem /path/to/myapp.example.com-key.pem
    reverse_proxy localhost:8000
}

The first argument is the certificate (full chain), the second is the private key. Caddy stops trying to auto-issue and just uses what you gave it.

Generating a development cert with mkcert is the path of least resistance — install it once, run mkcert -install (which adds its CA to your system trust store), then for any hostname:

1
2
mkcert myapp.example.com
# Creates myapp.example.com.pem and myapp.example.com-key.pem in the current directory

Multiple sites in one block

If you have several hostnames that should share the same TLS settings and proxy target — common with multi-tenant local development — put them on one line, comma-separated:

1
2
3
4
5
6
7
myapp.local, one.myapp.local, two.myapp.local {
    tls /path/to/myapp.local+2.pem /path/to/myapp.local+2-key.pem
    reverse_proxy localhost:80 {
        header_up Host {host}
        header_up X-Forwarded-Proto https
    }
}

Two things worth noticing in that block:

  • header_up Host {host} forwards the original Host header to the upstream — important when your app routes by hostname (multi-tenant, virtual hosts, etc.). Without this, the upstream sees localhost and may not know which tenant is being requested.
  • header_up X-Forwarded-Proto https tells the upstream that the original connection was https. Frameworks like Laravel, Django, and Rails need this to generate correct absolute URLs and to enforce secure-cookie flags.

The +2 in the cert filename is an mkcert convention: when you generate a cert for multiple hostnames, mkcert names the file after the first one and appends +N for the count of additional SANs (Subject Alternative Names).

Useful global options

The block at the very top of the Caddyfile, wrapped in plain { … } with no hostname, is the global options block. The two I reach for most:

1
2
3
{
    auto_https disable_redirects
}

By default, Caddy auto-redirects http:// traffic to https://. Useful in production, occasionally annoying locally — for example, if you’re testing a service that’s already running on port 80 with its own non-https endpoint, the redirect gets in the way. disable_redirects turns that off but keeps the auto-cert magic. Other handy globals:

1
2
3
4
5
{
    debug                                        # verbose logs while iterating
    email you@example.com                        # used by Let's Encrypt for cert expiry warnings
    storage file_system /var/lib/caddy           # where issued certs are cached
}

The thing that won me over

Once you’ve used Caddy for a week, going back to nginx + certbot for a new project feels strange. The Caddyfile fits on a Post-it. There’s no separate cron job to renew certs — Caddy renews them itself. There’s no special config for HTTP/2 or HTTP/3 — they’re on by default. And when the site doesn’t load, the error message tells you why in one sentence, not via a stack trace from journalctl.

It’s not a replacement for nginx everywhere — at high traffic, behind a CDN, or as a proxy for very specialised workloads, nginx still has the edge. But for personal sites, internal tools, and local development, Caddy is hard to beat. 🎉

Posted in Web Development | Tagged , , | Comments Off on Local HTTPS in 5 minutes with Caddy 🔒

Spatie activity_log: which method writes to which column? 🐘

If you’re using spatie/laravel-activitylog, you’ve probably written something like activity()->event(…)->log(…) a hundred times without thinking about where each piece lands in the database. The fluent API is friendly, but the column mapping isn’t obvious until you go look — so here it is in one place.

The package writes to a single table called activity_log. Every chained method on the builder corresponds to one column on that row. 💡

1
2
3
4
5
6
7
activity()
    ->useLog('SyncCampaignUsersJob')
    ->event('Sync user without detaching')
    ->performedOn($campaign)
    ->causedBy($actor)
    ->withProperties(['chunk' => '3/20', 'count' => 100])
    ->log('Processing chunk 3/20 (100 users).');

That single fluent call writes one row. Here’s the full mapping:

Method Column(s) What it stores
useLog(“string”) log_name Filterable bucket like “Auth” or “SyncCampaignUsersJob”. Never leave empty — defaults to the literal string “default”, which makes filtering useless.
log(“string”) description Free-form, human-readable message. Returned by $activity->description.
event(“string”) event Short verb-ish label like “created”, “updated”, “Synced user with detaching”. Useful for grouping similar actions.
performedOn($model) subject_type + subject_id Polymorphic reference to the affected model. e.g. “App\Models\Campaign” + 6.
causedBy($user) causer_type + causer_id Polymorphic reference to the actor. Pass a model instance or just the ID.
withProperties([…]) properties Arbitrary JSON. Great for structured context: counts, IDs, batch labels, before/after diffs.

The empty-useLog gotcha 🪤

Here’s the failure mode worth burning into memory. This call:

1
2
3
4
5
activity()
    ->event('Sync user without detaching')
    ->performedOn($campaign)
    ->causedBy($actor)
    ->log('Synced 100 users.');

…silently writes log_name = “default”. Six months later you open the activity log dashboard, filter by log name, and you’re staring at 47,000 rows in the default bucket. Always add useLog() with a meaningful string. The class name of the job or service writing the log is a perfectly fine default — future-you will thank present-you when grep’ing through audit history.

One more nuance: causer_type

If you pass an integer to causedBy(), the package needs to know what model that ID points to. By default it assumes your auth user model (set in config/auth.php). If your causers are sometimes a User and sometimes a SystemActor or a tenant model, pass the model instance instead of the ID — the polymorphic columns will resolve correctly and querying back becomes painless.

That’s the whole mental model: one row, one chained call, one column per method. Keep useLog() populated and you’ll have a queryable audit trail instead of a blob of “default” entries. 🎯

Posted in PHP | Tagged , , , | Comments Off on Spatie activity_log: which method writes to which column? 🐘

The Null Coalescing Operator: A Small PHP Feature That Quietly Changed Everything

If you’ve been writing PHP for a while, you probably remember the days of nested „isset()” checks cluttering up every template and controller. Since PHP 7, there’s a much cleaner way — and if you haven’t fully embraced it yet, it’s worth a second look.

The null coalescing operator (??) returns the left operand if it exists and isn’t null, otherwise the right. No warnings, no notices, no ceremony.

1
2
3
4
5
6
7
8
9
<?php
// The old way — verbose and easy to get wrong
$username = isset($_GET['user']) ? $_GET['user'] : 'guest';

// With null coalescing — same behavior, far less noise
$username = $_GET['user'] ?? 'guest';

// It chains too, which is where it really shines
$config = $userConfig['theme'] ?? $siteConfig['theme'] ?? 'default';

PHP 7.4 took it a step further with the null coalescing assignment operator (??=), which only assigns if the variable is currently null or unset:

1
2
3
4
5
6
7
8
9
<?php
$options = ['timeout' => 30];

// Only set 'retries' if it isn't already defined
$options['retries'] ??= 3;
$options['timeout'] ??= 60; // stays 30 — already set

print_r($options);
// Array ( [timeout] => 30 [retries] => 3 )

One subtle thing to keep in mind: ?? only reacts to null or unset — not to falsy values like „0″, „””, or „false”. That’s usually what you want, but it’s a meaningful difference from the older ?: (Elvis) operator, which falls back on any falsy value.

1
2
3
4
5
<?php
$count = 0;

echo $count ?? 10;  // prints 0 — because 0 is not null
echo $count ?: 10;  // prints 10 — because 0 is falsy

Small syntax, big quality-of-life improvement. If your codebase still has rows of „isset()” ternaries, refactoring them is one of those low-risk cleanups that pays off every time someone reads the file next. 🐘

Posted in PHP | Tagged | Comments Off on The Null Coalescing Operator: A Small PHP Feature That Quietly Changed Everything

Did You Know? Python’s Walrus Operator Can Make Your Code Cleaner

Did you know? Since Python 3.8, you can use the walrus operator ( := ) to assign a value to a variable as part of an expression. It’s a small piece of syntax that can meaningfully tidy up loops and comprehensions where you’d otherwise compute the same value twice.

Here’s a classic example — reading lines from a file until you hit an empty line:

1
2
3
4
5
6
7
8
9
10
11
# Without the walrus operator
with open("data.txt") as f:
line = f.readline()
while line:
print(line.strip())
line = f.readline()

# With the walrus operator — assign and test in one step
with open("data.txt") as f:
while (line := f.readline()):
print(line.strip())

It’s also handy in list comprehensions when you want to filter on a computed value without recomputing it:

1
2
3
4
5
6
7
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Keep only squares greater than 20, without squaring twice
big_squares = [sq for n in numbers if (sq := n * n) &gt; 20]

print(big_squares)
# [25, 36, 49, 64, 81, 100]

A word of caution: the walrus operator is powerful but easy to overuse. Reach for it when it genuinely removes duplication or makes intent clearer — not just because it’s clever. 🐍

Posted in Python | Tagged | Comments Off on Did You Know? Python’s Walrus Operator Can Make Your Code Cleaner

Did You Know? Python Dictionaries Preserve Insertion Order

Did you know? Since Python 3.7, the built-in

1
dict

type officially preserves the order in which keys are inserted. Before that, if you needed ordering guarantees you had to reach for

1
collections.OrderedDict

. Today, a plain dictionary is enough for most cases.

Here’s a small demonstration:

1
2
3
4
5
6
7
8
9
10
11
12
13
# Keys stay in the order they were added
user = {}
user["name"] = "Ada"
user["role"] = "Author"
user["joined"] = 2026

for key, value in user.items():
    print(f"{key}: {value}")

# Output:
# name: Ada
# role: Author
# joined: 2026

This also means dictionary comprehensions and merges keep a predictable order, which is surprisingly useful when serializing to JSON or building config objects:

1
2
3
4
5
6
7
defaults = {"host": "localhost", "port": 8080}
overrides = {"port": 9090, "debug": True}

# Merge with the | operator (Python 3.9+)
config = defaults | overrides
print(config)
# {'host': 'localhost', 'port': 9090, 'debug': True}

One caveat: ordering is a property of the dictionary, not of equality. Two dicts with the same keys and values are considered equal even if their insertion order differs. 🐍

Posted in Python | Tagged | Comments Off on Did You Know? Python Dictionaries Preserve Insertion Order

BOLA in a Laravel Livewire app: when client-side state is the only thing standing between users and admin actions

A penetration test landed an interesting finding on a Livewire-powered admin panel I work on. The summary on the report read: Broken Object-Level Authorization (BOLA). A standard user can change a tenant-wide “who can access these assets” setting by replaying an administrator’s Livewire request. Severity: Low. Impact: High.

That gap between severity and impact is what made the finding interesting. “Low” because exploitation requires capturing a snapshot from an admin’s session — non-trivial. “High” because the moment you have one, a regular user becomes effectively an administrator. 🪓

What the tester actually did

Two browsers, side by side.

Browser A: logged in as a tenant administrator. Open the asset access settings page, flip the toggle, click Save. While the request is in flight, capture the Livewire snapshot — the JSON blob Livewire posts to /livewire/update containing the component class, the new value, and the cryptographically-signed snapshot of component state. This is normal browser-DevTools work.

Browser B: log out of the admin session. Log in as a plain unprivileged user. Replay the captured request from Browser A, with Browser B’s session cookie. The server processes it. The toggle flips. The standard user has just changed a tenant-wide setting.

The Livewire snapshot’s signature checks out — the snapshot itself is valid. What it’s missing is any check that the user submitting the request is actually allowed to perform the action it represents.

Why this happens in Livewire specifically

If you’ve built REST controllers in Laravel, you’ve reflexively put authorization at the top of your action methods:

1
2
3
4
5
public function update(Request $request, Asset $asset)
{
    $this->authorize('update', $asset);
    // ...
}

Livewire components don’t pattern-match this in your head the same way. The methods you write in a Livewire class — save(), delete(), toggleAccess() — feel like internal helpers. They’re public methods on a PHP object, not endpoints. But Livewire makes them exactly that: every public method is reachable from the browser via a signed snapshot replay. If you don’t authorize them server-side, nothing else does. Blade conditionals that hide UI elements only hide UI elements. The endpoint is open.

The mental shift: every public method on a Livewire component is a controller action, and deserves the same authorization treatment. 🛡️

The fix pattern

I went through every Livewire component in the project and applied the same three-step pattern.

1. Authorize in mount() for the whole component

If a component shouldn’t even be rendered for unauthorized users, fail fast in mount(). This handles the “don’t load it” half of the problem and short-circuits replay attacks against the form itself:

1
2
3
4
5
6
public function mount($context)
{
    $this->authorize('asset:list');
    $this->context = $context;
    // ...
}

2. Authorize on every action method

For each public method that mutates state — save, update, delete, toggleSomething — add an authorize call at the top. Don’t trust that mount() already gated the component, because a replay attack hits the action method directly without re-running mount():

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
public function deleteUser(): void
{
    $this->authorize('user:delete');
    // ... actual deletion
}

public function validateAndSaveUser(): void
{
    if ($this->context === 'createUser') {
        $this->authorize('user:create');
        // ...
    } else {
        $this->authorize('user:edit');
        // ...
    }
}

Note the pattern in the second example: the same component handles two different operations (create and edit) with different ability strings. The authorization check goes inside each branch, so the right ability is enforced for each.

3. Use a base class so it’s the default, not the exception

Across a few dozen components, it’s easy to miss one. We introduced a thin base class that all our Livewire components extend, which trait-includes a customized authorize():

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
namespace App\Auth;

use Illuminate\Foundation\Auth\Access\AuthorizesRequests as BaseAuthorizesRequests;

trait LivewireAuthorizesRequests
{
    use BaseAuthorizesRequests {
        authorize as baseAuthorize;
    }

    public function authorize($ability, $arguments = [])
    {
        return auth()?->user()?->canAccess($ability)
            || $this->baseAuthorize($ability, $arguments);
    }
}

Two small things going on. First, we alias Laravel’s stock authorize to baseAuthorize so we can fall through to it. Second, our app has a custom canAccess on the user that consults a role-ability map living in config/roles.php. The trait gives Livewire components both checks — our app’s role abilities, and stock Laravel policies — with one consistent call site.

4. The harder case: object-scoped authorization

Some abilities are global (“can this user create assets at all?”). Others are per-object (“can this user edit this specific campaign?”). The second one is closer to OWASP’s actual definition of BOLA — the object-level part. We added a sibling helper:

1
2
3
4
5
6
public function authorizeGroupedObject($ability, $groupedObject, $arguments = [])
{
    return (auth()?->user()?->canAccess($ability)
        && $groupedObject?->isAdminAuthorized(auth()->user()))
        || $this->baseAuthorize($ability, $arguments);
}

Used like:

1
$this->authorizeGroupedObject('campaign:edit', $this->campaign);

Both conditions must hold: the user has the role-level ability, and the user has access to the specific group/tenant/owner that this object belongs to. Without the second check, a user who has “campaign:edit” globally could replay a snapshot to edit a campaign in someone else’s group — exactly the BOLA pattern, just with the object identifier in the snapshot instead of the action.

Tests for replay attacks specifically

The most useful thing I added wasn’t the fix — it was a test file that simulates the exact attack. Roughly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public function test_standard_user_cannot_replay_admin_asset_toggle()
{
    $admin = $this->createTenantAdmin();
    $standardUser = $this->createStandardUser();

    // Standard user calls the action directly, as a replay would.
    $this->actingAs($standardUser);
    Livewire::test(AssetsAccess::class, ['context' => 'settings'])
        ->call('save', true)
        ->assertForbidden();

    // And the underlying setting is still the original value.
    $this->assertFalse(Setting::firstWhere('key', 'can_access_assets')->value);
}

The point of this test isn’t that the UI hides the button from the user — that’s not what’s being verified. The point is the action method itself, when called by an unauthorized actor, refuses. That’s the only assertion that catches a replay attack.

I wrote one of these for every component I touched. Feature tests, not unit tests, and Livewire’s Livewire::test() harness makes them concise.

Lessons

  • A signed snapshot is not an authorization check. Livewire’s signature proves the snapshot wasn’t tampered with. It does not prove the current user is allowed to use it. These are different properties; the framework provides the first, you provide the second.
  • Every public method on a Livewire component is a public endpoint. Reason about it the same way you would a controller action. “This is only called from my own Blade view” is wrong — it’s called by anyone who can construct a request to /livewire/update.
  • Hiding UI is not enough. A @can directive in Blade hides a button. It does not protect the action behind the button. Both are needed; only the second one is security.
  • Bake authorization into the base class. If “add $this->authorize(…) to every public method” is a convention, you’ll forget. If authorize is a trait method on the base class and there’s a code review checklist, you’ll forget less. If you go a step further and write a static analyzer that flags Livewire action methods with no authorize() call, you’ll forget least.
  • Test replays directly. Don’t only test the happy path “admin can do thing” and the sad path “button doesn’t show for non-admin.” Also test “non-admin calls the action method directly and is rejected.” That’s the test that maps to the actual attack.

The pentest report rated this Low severity because the attacker needs an admin’s snapshot. In practice, the gap between “can capture an admin’s snapshot” and “is an admin” is whatever the local network conditions are — a shared workstation, a malicious browser extension, a screenshare gone wrong. Do not rely on that gap. Authorize on the server, on every action, every time. 🔐

Posted in Laravel, PHP | Tagged , , , , , , | Comments Off on BOLA in a Laravel Livewire app: when client-side state is the only thing standing between users and admin actions

Azure AD, Google Directory, and SCIM: picking a user-sync story for a multi-tenant Laravel app

Late 2024 I spent a few weeks digging into how a multi-tenant Laravel platform I was working on should let tenant administrators pull users in from external identity providers. The customer asks were predictable — “we use Azure,” “we use Google Workspace,” “can you just hook into our directory?” — and the answer turned out to be more interesting than the question. After looking at Azure Active Directory, Google Directory, and the System for Cross-domain Identity Management (SCIM) protocol, we landed on SCIM as the primary path, with the two cloud-directory options reduced to footnotes. 🐳

This post is a tidied-up version of the investigation notes. If you’re picking a user-sync mechanism for a Software-as-a-Service (SaaS) app and the customer is pointing at one of these three things, the trade-offs below might save you a week.

Why not just speak LDAP to Azure AD directly?

Azure Active Directory (Azure AD) is Microsoft’s cloud identity service — it sits behind Office 365, handles sign-in for Microsoft cloud apps, and is what enterprise customers usually mean when they say “our directory.” The instinct, coming from a traditional on-prem world, is to point an Lightweight Directory Access Protocol (LDAP) client at it and start browsing users.

You can’t. Azure AD does not natively speak LDAP. What it offers instead is one of three pictures, depending on the customer’s deployment:

  1. On-prem AD synced to Azure AD via Azure AD Connect. Your server can speak LDAP to the on-prem Active Directory box, the way it always has. Azure AD is just a downstream replica used for cloud sign-in. The authoritative data still lives on-prem.
  2. Pure cloud Azure AD with no LDAP at all. No LDAP endpoint exposed, anywhere. You either talk to it via the Microsoft Graph REST API, or you don’t talk to it.
  3. Azure AD with Azure AD Domain Services (Azure AD DS) enabled. This spins up a separate managed domain in the cloud that does support LDAP. It’s a paid feature, and it’s a new domain rather than a view into the existing one — the customer would have to decide to migrate into it.

For our app to “just work” with an existing LDAP browser, the customer needed to be in case (1) or (3). Plenty of enterprise customers aren’t — they’re cloud-first, with no on-prem AD and no Domain Services subscription. For those, LDAP is simply not an option, and the realistic alternative is Microsoft Graph.

Graph is a fine API, but adopting it as a sync source means real development work: capture tenant ID, client ID, client secret, and the consented permission scopes (User.Read.All, Directory.Read.All); add a Create-Read-Update-Delete (CRUD) interface for those settings; bring in something like composer require microsoft/microsoft-graph; build the sync loop. None of it is exotic, but it’s all Azure-specific code we’d then have to write again for the next vendor.

One other footnote worth knowing: Azure AD’s free tier covers basic Graph reads, but stress-testing 20,000 users will hit throttling quickly. Azure AD Connect is free with any Azure subscription; Azure AD Domain Services is a premium feature with its own line item. The cost picture is benign for development, less benign for serious load testing.

Google Directory: same destination, different road

Google Directory is the directory layer of Google Workspace — same job as Azure AD, different ecosystem. It manages user accounts, groups, and devices for Workspace tenants and handles sign-in to Gmail, Drive, and the rest.

And just like Azure AD’s cloud-only mode, Google Directory does not speak LDAP. There is no LDAP browser story here at all — no equivalent of Azure AD Domain Services that opens an LDAP port. The only programmatic access is the Admin SDK REST APIs. So whatever LDAP-based extension you’ve been using on the Active Directory side (in our case directorytree/ldaprecord-laravel, which is genuinely lovely for AD work) is just dead weight here.

The Google API client for PHP (composer require google/apiclient) covers the API surface, and the call pattern is similar to Microsoft Graph: OAuth2 service-account credentials, scoped permissions, paginated list endpoints. The schema mismatch is a small extra annoyance — fields we care about, like “manager email” or “team lead,” aren’t always populated in a default Workspace setup, and the customer may need to extend their directory schema via the Admin SDK before our sync sees anything useful.

Cost-wise: Google Workspace doesn’t have a long-term free tier, just a 14-day trial. For development, that’s enough to wire things up; for sustained QA, someone has to pay.

So now we have two cloud directories that don’t speak LDAP, each with its own REST API, its own auth model, and its own schema quirks. If we want to support both, we write the integration twice. This is the moment SCIM starts looking obviously better.

SCIM: the protocol that lets the identity provider do the work

SCIM (System for Cross-domain Identity Management) is a standard for provisioning users between systems. The relevant Request for Comments is RFC 7643 (core schema) plus RFC 7644 (protocol). The pitch, in one sentence: your app exposes a small REST API in a fixed shape, and the customer’s identity provider pushes user changes to it.

That inversion of control is the whole point. Instead of our app polling Azure for users, then polling Google for users, then polling Okta for users — three different APIs, three different auth dances, three different schemas — Azure, Google, and Okta all push the same SCIM-shaped requests to the same endpoint on our side. We write the receiver once. The vendors compete to be good SCIM clients; we just have to be a correct SCIM server.

The terminology is worth getting straight, because it’s a bit counter-intuitive:

  • The identity provider (Azure AD, Google, Okta) is the SCIM client — it initiates requests.
  • Our application is the SCIM service provider — it receives them.

“Client” feels like it should be the consumer, but in SCIM the client is the pusher. Just memorise it; it’ll come up.

After a conversation with a couple of teammates in early December, we settled on SCIM as the path forward, with Azure AD Graph and Google API integrations parked as “maybe later, if a customer specifically asks.” Below is the shape of what we actually built. 🛠️

What the SCIM receiver needs to expose

For Laravel, the arietimmerman/laravel-scim-server package gives you most of the SCIM endpoint scaffolding for free — base routes, schema discovery, the right error envelopes. Standing it up takes an hour. Making it actually map to your domain takes much longer, because every SCIM server eventually becomes opinionated about what the incoming attributes mean.

The endpoints we needed:

  • /api/scim/v2/ — service root, returns capability metadata.
  • /api/scim/v2/Users — supports HTTP POST (create), GET (read/search), PATCH or PUT (update), DELETE (deprovision). A DELETE doesn’t actually nuke the user; it flips their status to suspended, same as our existing LDAP-based flow did.
  • /api/scim/v2/Schemas — schema discovery. The library generates this from your model definitions.
  • /api/scim/v2/Groupsdeliberately left out. Our internal group model doesn’t line up well with SCIM’s, and forcing the mapping would have been more painful than asking customers to manage group membership in-app.

Because the platform was multi-tenant — Stancl’s tenancy library, one database per tenant — we created a fresh route file routes/tenant_api.php rather than co-mingling SCIM with the central admin API. The controller lives at app/Http/Controllers/Tenant/Api/SCIMUserController.php, with the messier translation logic factored into app/Services/Tenant/ScimService.php. Vanilla shape, but worth being explicit because the SCIM library defaults assume a single-tenant Laravel app.

Auth tokens via Sanctum and Jetstream

The SCIM client side (Azure, Okta, etc.) expects a long-lived bearer token. We already had Jetstream + Sanctum wired up for the app, so the natural move was to let tenant admins mint API tokens from their profile page, with a specific SCIM ability scoped on each token. Sanctum handles the storage, expiry, and revocation; Jetstream handles the management UI; we just had to make sure the tokens lived in the tenant database rather than the central one, and reword the permissions checklist so the SCIM ability was discoverable.

The flow from an admin’s perspective:

  1. Open profile page → create API token → name it “Azure SCIM” or similar → tick the SCIM permission.
  2. Copy the one-time-displayed token.
  3. Paste it into the SCIM client configuration in Azure / Okta / Google, alongside the SCIM base URL (which the in-app guide page displays with a copy button).

The “shown only once” pattern is Sanctum’s default and it’s the right one — but you do have to write a sensible warning into the regeneration flow, because admins will absolutely lose the token and try to regenerate, and you want them to understand that the old token stops working the moment they do.

Schema: just two tables

The data model is unglamorous. Sanctum brings its own personal_access_tokens table (publish it with php artisan vendor:publish –provider=”Laravel\Sanctum\SanctumServiceProvider”), and we added one extra table to capture SCIM-specific overflow.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
TABLE personal_access_tokens {
  id              INT        [pk, INCREMENT]
  tokenable_type  VARCHAR
  tokenable_id    INT
  name            VARCHAR
  token           VARCHAR    [UNIQUE]
  abilities       text
  last_used_at    TIMESTAMP
  expires_at      TIMESTAMP
  created_at      TIMESTAMP
  updated_at      TIMESTAMP

  indexes {
    tokenable_type_tokenable_id_index [tokenable_type, tokenable_id]
  }
}

The companion table stores everything that SCIM tells us about a user that we don’t have a first-class column for. Phone numbers, employee numbers, alternate addresses, locale — none of that is in our core users schema, but the SCIM contract is that we acknowledge it and return it on a subsequent GET. So we stash it as JSON.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
TABLE scim_users {
  id           BIGINT     [pk, INCREMENT]
  user_id      BIGINT     [NOT NULL]
  extra_fields json       [DEFAULT NULL]
  created_at   TIMESTAMP  [DEFAULT NULL]
  updated_at   TIMESTAMP  [DEFAULT NULL]

  indexes {
    user_id_index [user_id]
  }

  foreign_keys {
    user_id [REFERENCES users(id), ON DELETE cascade]
  }
}

Two tables, one foreign key, no surprises. Most of the complexity is in the controller layer, not the schema. 💡

The “no vendor actually batches” surprise

RFC 7644 defines a /Bulk endpoint for batch operations. I built scaffolding for it on day one, assumed it would be the hot path for the initial onboarding sync, and started planning a queued-job pipeline to handle the load.

Then I actually watched Azure AD push 50 users.

It sent 50 individual POSTs to /Users, each followed by a GET to confirm the new resource existed. No /Bulk call. Okta does the same. Google Workspace does the same. None of the major SCIM clients actually use the bulk endpoint, despite it being in the spec, because their internal architectures are already issuing one provisioning event per user and there’s no operational benefit to coalescing them. So we ripped out the batch scaffolding and the queued-job pipeline, and treated user creation as a straightforward sequential operation that returns the SCIM-shaped “user created” envelope synchronously. Much simpler. 🎉

Related simplification: we don’t need to keep our own sync logs. The customer’s SCIM client (Azure, Okta, Workspace) keeps a detailed provisioning log on its side, including every error response we return. Building a duplicate log on our side would have been busywork the customer would never look at.

The dev-tunneling problem

One genuinely annoying problem: how does Azure reach a SCIM server running on a developer’s laptop?

For an external-facing customer-installed app this isn’t an issue — Azure hits a public URL. For local development, you need a tunnel. A teammate suggested Laravel Expose, which is a nice piece of software in principle: it gives your local app a public HTTPS URL via a relay server, exactly what we needed. On their laptop it worked perfectly. On mine, the Vite-served UI elements rendered partially, page loads were broken, and SCIM requests kept timing out for reasons I never fully diagnosed. We worked around it for a while by sharing their laptop as the integration-test environment, but it’s the kind of friction you want to remove before more developers join the team. Cloudflare Tunnel and ngrok are the obvious alternatives if Expose doesn’t behave.

For staging, the equivalent question is “how does Azure reach our staging server?” In our case staging was behind the company training Virtual Private Network (VPN), which Azure obviously can’t see. The answer turned out to be straightforward — once the staging server got a real public DNS name and a public-facing route, Azure could talk to it like any other SCIM endpoint. The VPN was incidental to the staging architecture, not load-bearing.

The IP-restriction question

Worth mentioning because it’ll come up the moment a security-conscious customer reviews your SCIM setup: can we restrict the SCIM endpoint to specific source Internet Protocol (IP) addresses?

SCIM doesn’t require it, and arguably doesn’t want it — the protocol assumes the customer is the one initiating requests, and the bearer token is the security boundary. But Azure AD’s outbound IPs do change over time, and some customers will still ask for an allowlist out of habit. Our position was: not yet, ask us again when a customer makes it a deal-breaker. If we do implement it, it goes at the reverse proxy / firewall layer, not inside the app — keeping the controller agnostic about source IPs is the right call.

What I’d tell past-me

Three things, condensed:

  1. If the customer has any plurality of identity providers, SCIM is the answer. Writing one Azure Graph integration is fine. Writing Azure + Google + Okta is a maintenance tax that compounds. SCIM lets each vendor be responsible for translating its directory into a common shape on the wire.
  2. Build the receiver, skip the bulk endpoint, skip your own sync logs. Vendors push one user at a time and keep their own logs. Anything past that is over-engineering.
  3. The dev-environment story is half the work. Local tunneling, staging public reachability, token rotation flows, and “show this token once” UI are the parts that don’t appear in the protocol spec but absolutely show up in onboarding friction. Plan for them up front.

For a tenant-aware Laravel app, the actual code footprint of SCIM is small: a route file, a controller, a service, two tables, a UI page or two, and a Sanctum ability. The conceptual footprint — getting comfortable with “the customer’s directory is in charge, we just listen” — is the bigger shift, and the one that pays off across every future identity-provider integration. 🔐

Posted in Laravel | Tagged , , , , | Comments Off on Azure AD, Google Directory, and SCIM: picking a user-sync story for a multi-tenant Laravel app

Free Azure AD SCIM provisioning to a Laravel app on your laptop, via home router + dynamic DNS

In the last post I sketched why SCIM (System for Cross-domain Identity Management) won out over direct Azure Active Directory (Azure AD) and Google Directory integrations for a multi-tenant Laravel app I was working on. This one is the hands-on follow-up: how to actually get Azure pushing user-provisioning events to a Laravel application running on your laptop, for free. 🐳

The shape of the setup: open a port on your home router, point a free dynamic-Domain-Name-System (dynamic-DNS) hostname at it, and run Laravel Sail behind that hostname. It mirrors the production push flow closely enough to be a useful test rig, gives you a real Azure portal to point at when you’re tuning attribute mappings, and costs nothing beyond the hour or so of one-time setup.

Why not just use Laravel Expose?

Laravel Expose is the obvious answer — it relays public HTTPS requests to a process on your machine, with a friendly Laravel-shaped Command-Line Interface (CLI). On the free tier it works fine for a single-host app. The wrinkle for us was multi-tenancy: the app routes by subdomain, and every tenant lives at tenantname.app.example.com. To exercise that locally, you need a tunnel that gives you a wildcard subdomain, not just a single hostname.

Expose’s wildcard-subdomain feature is paywalled — $60 USD per year plus tax, which lands at roughly $96 once it clears the border. That’s not a lot of money, but it’s enough to think twice when you remember you’ve already got a static-ish home Internet Protocol (IP) address and a router with a port-forwarding screen. So I bypassed it.

Step 1: poke a hole through your home router

Most consumer routers have a “port forwarding” or “service hosting” section. Pick a public port on the Wide-Area-Network (WAN) side and point it at your laptop’s Local-Area-Network (LAN) IP on the port your Laravel Sail container listens on (port 80 by default inside the Sail web container, exposed however your docker-compose.yml maps it — often 80 or 8000 on the host).

One Internet Service Provider (ISP)-specific gotcha that bit me, and might bite you: some ISPs silently block inbound port 80 to residential connections. They don’t explicitly tell you this on the router admin page; the port-forward rule will save happily and then silently drop every connection. The fix is to forward a non-80 port like 8080 instead — most ISPs allow that, and Azure doesn’t care whether your SCIM endpoint lives at port 80 or 8080, since the Tenant Uniform Resource Locator (URL) field lets you specify the port explicitly. If your forward “works” on the router but a phone on cellular data can’t reach it, suspect a residential port-80 block.

Test from outside your network — phone with WiFi off is the easy version:

1
2
# from your phone or any external machine
curl -v http://<your-public-ip>:8080/

If you see your app’s HTML, the tunnel is open. If you get connection-refused or a timeout, it’s either the ISP, the router firewall, the OS firewall, or the wrong LAN IP — work through those in order.

Step 2: a hostname that isn’t your raw IP

Azure will happily accept an IP-address Tenant URL, but the multi-tenant subdomain routing in the app needs a real Domain Name System (DNS) name. no-ip.com gives you a free dynamic-DNS account: you sign up, pick a hostname from one of their domains (mine ended up on redirectme.net, but they have several to choose from), and either run a tiny daemon on your machine or update the IP via their web form whenever your home IP changes.

So now you have something like myapp.redirectme.net pointing at your home IP, and port 8080 on that IP forwarding to your laptop. Putting http://myapp.redirectme.net:8080/scim/v2 into the Tenant URL field on Azure works the same way it would if you were running on a cloud server. The dev-vs-prod difference is mostly: less uptime, and you have to remember to keep the laptop awake during a provisioning test. 💡

For the multi-tenant subdomain wrinkle, you also need per-tenant hostnames. The pragmatic shortcut is to set up one tenant whose domain matches your no-ip hostname exactly. So if your no-ip hostname is myapp.redirectme.net, you want a tenant in your database keyed to that domain. Two small code changes accomplish this.

Step 3: point your Laravel app at the public hostname

Two files need to know about the new hostname. First, .env:

1
APP_URL=http://myapp.redirectme.net

Second, your tenant seeder. If you’re using Stancl’s tenancy library (we were), there’s typically a manual-test seeder somewhere like database/seeders/Tenant/ManualTestSeeder.php that creates your fixture tenant and assigns it a domain. Change that domain to match the no-ip hostname:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class ManualTestSeeder extends Seeder
{
    private $testTenantId = 'test_tenant_one';
    private $testTenantDomain = 'myapp.redirectme.net';
    private $testTenantAdminEmail = 'test_user1@localhost.com';

    public function run()
    {
        $tenant = Tenant::create([
            'id' => $this->testTenantId,
        ]);
        $tenant->domains()->create(['domain' => $this->testTenantDomain]);
        // ... seed an admin user, an API token, etc.
    }
}

Re-seed, then sanity-check that the app loads at http://myapp.redirectme.net:8080/ from a browser on a network that isn’t yours. If you see the tenant’s landing page rather than the central-app landing page, the subdomain routing is doing the right thing.

Step 4: the Azure portal walkthrough

Now to the Azure side. Sign in at portal.azure.com. A work account on Office 365 works; a personal Microsoft account (Hotmail / Outlook / Live) also works, since Azure gives you a free tenant attached to your consumer identity. Either way, you land on the Azure home page with a row of service tiles. The walkthrough is nine clicks.

  1. Click “Enterprise applications”. This is the catalog of apps that have been onboarded into your Azure AD tenant. You’re going to add a new one that represents the Laravel app.
  2. “+ New application” → “Create your own application”. A side panel slides in asking what you’re integrating. Pick “Integrate any other application you don’t find in the gallery (Non-gallery)”. The gallery is for vendors who pre-registered their Enterprise application templates with Microsoft; yours obviously isn’t there.
  3. Name your app. Whatever you like. I used scimtesterapp3 because I’d already created and torn down two others while figuring this out. The name is just a label inside the Azure tenant.
  4. Go back to the Azure home page → click “Users”. This is the directory of users in your Azure AD tenant. Out of the box, you have one user — yourself — and you’ll need at least one more to provision through SCIM. Self-provisioning the owner is a special case and won’t exercise the create-user path.
  5. “+ New user” → “Create new user”. Fill in the Basics tab (user principal name, display name, mail nickname, a password — none of these will ever be used to actually sign in, so don’t sweat the password complexity). Then the Properties tab: email, first name, last name, job title, company name, department, manager. The properties matter because they’re what Azure will hand to your SCIM endpoint when you provision. I usually create someone called Alice Smith (alicesmith@example.com), give her a job title like “Secretary,” a department, and a manager — enough fields populated that the SCIM payload looks realistic.
  6. Confirm the user shows up in the Users list. Two entries now: you, and Alice.
  7. Back to Enterprise applications → your app → “Users and groups” → “+ Add user/group”. Pick Alice from the picker and assign her to the application. This is the bit that says, in Azure-speak, “Alice is in scope for this app’s provisioning.” Without this assignment, even a perfectly-configured SCIM connection won’t push her anywhere.
  8. Open the “Provisioning” blade → “Get started”. Set Provisioning Mode to Automatic. Under Admin Credentials, fill in:
    • Tenant URL: your full public SCIM endpoint, including the path. For our app that’s http://myapp.redirectme.net:8080/scim/v2.
    • Secret Token: an Application Programming Interface (API) token you generated in the Laravel app’s profile-page token UI, with the SCIM ability checked. (See the prior post for how that’s wired up via Sanctum.)

    Hit “Test Connection.” If everything’s right, Azure will get a 200 from your SCIM service-discovery endpoint and confirm that the supplied credentials are authorised. Don’t forget to click Save — Azure’s screen design here is genuinely sneaky, and the Save button is easy to overlook above the form. Without Save, your next step won’t know about the credentials.

  9. “Provision on demand” → pick Alice → run it. Azure walks through a four-step pipeline: import the user from the source, evaluate scoping rules, look up whether the target already has her, then perform the action (create, in this case). If all four steps come back green, your SCIM endpoint just got a real POST /Users from a real identity provider, and Alice now exists in the Laravel app’s tenant users table. 🎉

Once “Provision on demand” works end-to-end, you can flip the provisioning mode to scheduled and Azure will start pushing changes every 40 minutes or so — but for development the on-demand button is the one you’ll live in, because it gives you a per-step success/failure breakdown and lets you iterate on attribute mappings without waiting for the next cycle.

Quirks worth knowing about

A few things that aren’t bugs exactly, but are worth bracing for:

  • Your dev environment vanishes when your laptop sleeps. Port 8080 stops answering, Azure’s next “Provision on demand” attempt fails with a connection error, and you’ll spend thirty seconds confused about what changed. Nothing changed; you closed the lid. I came to see this as a feature, not a bug — it’s a default-deny posture for the rest of the internet when I’m not actively testing.
  • Your home IP changes occasionally. Most consumer ISPs hand out “dynamic” IPs that in practice change once every few months. no-ip’s daemon handles this transparently; if you go the manual-update route, expect to update the IP a couple of times a year when SCIM mysteriously stops connecting.
  • HTTP, not HTTPS. The setup above uses plain HTTP because port-forwarding to a local TLS-terminating container is fiddly. Azure will accept this for SCIM — it complains in the UI but allows it — and for a local dev box the tradeoff is reasonable, because the only data flowing through is test users you made up. For staging or anything resembling production, terminate Transport Layer Security (TLS) at a proper public host. Don’t be the person who ships an HTTP SCIM endpoint with real customer data on it.
  • Azure caches things. When you change attribute mappings in the Provisioning blade, sometimes the next “Provision on demand” run uses the old mapping. Wait a minute, retry, and don’t go diving into your code looking for the bug straight away.

Why this is worth doing

The point of all this is to get a real Azure portal pushing to your code. SCIM has enough vendor-specific edge cases — Azure’s quirks around externalId, the enterprise-extension schema for department and manager, the slight differences between how Azure and Okta encode multi-value attributes — that you really do want to be testing against the actual identity provider, not a stub. Once the tunnel-and-dynamic-DNS scaffolding is in place, the iteration loop is fast: tweak controller code, re-run “Provision on demand,” watch the Azure log and your Laravel log side by side. 🔐

An hour of one-time setup, no recurring cost, and you’ve got a SCIM integration test rig sitting on your dev machine. The next post in this thread will be the equivalent walkthrough for Okta, which differs from Azure in some interesting ways — but the home-network side of the setup stays the same.

Posted in Laravel | Tagged , , , , | Comments Off on Free Azure AD SCIM provisioning to a Laravel app on your laptop, via home router + dynamic DNS

Laravel Jobs, Queues, Batches, and Redis: A Field Guide

Laravel’s queue system is one of those features you can use for years without really understanding what’s happening underneath. You call SomeJob::dispatch(), a worker somewhere picks it up, and life goes on. But the moment a job mysteriously runs twice, or your failed_jobs table fills up overnight, or Redis OOMs because of a job backlog you forgot about, you suddenly need to understand the moving parts. This is the field guide I wish I’d had. 🐘

What a “Job” Actually Is

A Laravel Job is a plain PHP class that represents a unit of background work — sending an email, generating a Portable Document Format (PDF) report, syncing a record to an external Application Programming Interface (API). You hand it to the queue, and a separate worker process picks it up and runs it later.

The cast of characters:

  • Job class — your code. A class that uses the Dispatchable trait and implements a handle() method.
  • Queue connection — where jobs are stored. Configured in config/queue.php. Common drivers: sync (run immediately, no queue), database, redis, sqs, beanstalkd.
  • Worker — a long-running PHP process started by php artisan queue:work. It pulls jobs off the queue and runs them.
  • Coordinator (optional) — Redis, when used as the driver, also acts as the lock store and pub/sub fabric for things like batches and unique jobs.

The Database Tables Laravel Uses

Even if you end up running on Redis, the database tables tell you what Laravel is conceptually tracking. The three you’ll see most often:

  • jobs — the pending queue. Used only when the queue driver is database. Each row is one serialized job payload waiting to be picked up.
  • failed_jobs — the graveyard. Used regardless of driver. When a job throws and exhausts its retry attempts, it lands here with its exception trace.
  • job_batches — batch metadata. One row per batch, tracking total_jobs, pending_jobs, failed_jobs, cancelled_at, and the serialized then / catch / finally callbacks.

You also indirectly touch the cache / cache_locks tables when you use job middleware like WithoutOverlapping or the ShouldBeUnique contract — but only if your cache driver is database. With Redis, those locks live in Redis instead, which is much faster under contention.

Create the tables with the built-in artisan generators:

1
2
3
4
php artisan queue:table
php artisan queue:failed-table
php artisan queue:batches-table
php artisan migrate

Each command publishes a migration; migrate applies them. ✨

Dispatching a Job

A minimal job class looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<?php

namespace App\Jobs;

use App\Models\Invoice;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class SendInvoiceEmail implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public int $tries = 3;
    public int $timeout = 60;

    public function __construct(public Invoice $invoice) {}

    public function handle(): void
    {
        // ...send the email
    }
}

Dispatching is one line:

1
2
3
4
5
6
7
8
SendInvoiceEmail::dispatch($invoice);

// or pin it to a specific queue / connection / delay
SendInvoiceEmail::dispatch($invoice)
    ->onConnection('redis')
    ->onQueue('high')
    ->delay(now()->addMinutes(5))
    ->afterCommit();

That afterCommit() at the end is one of the most important methods in this whole post. We’ll come back to it in the gotchas section. 💡

Batching With Bus::batch

Sometimes you have a thousand things to do and you want to know when they’re all done. That’s what Bus::batch is for:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;
use Throwable;

$batch = Bus::batch(
    $invoices->map(fn ($invoice) => new SendInvoiceEmail($invoice))
)
    ->name('Send monthly invoices')
    ->allowFailures()
    ->onQueue('emails')
    ->then(function (Batch $batch) {
        // All jobs completed successfully (or failures were allowed)
    })
    ->catch(function (Batch $batch, Throwable $e) {
        // First failure observed
    })
    ->finally(function (Batch $batch) {
        // Batch finished, success or not
    })
    ->dispatch();

return $batch->id;   // a UUID you can use to poll status later

Behind the scenes, Laravel inserts a row into job_batches with total_jobs = 1000, pending_jobs = 1000, and decrements pending_jobs as each child job completes. The then / catch / finally closures are serialized into the row and fired when the appropriate transition happens.

The allowFailures() call is important: without it, the first job that throws stops the rest of the batch from continuing. With it, every job runs and you can inspect $batch->failedJobs at the end.

Worker / Queue Command Lines

The headline command is queue:work. There’s also queue:listen, which restarts the framework on every job — useful only in local development if you want code changes to apply without restarting. In production, you always use queue:work under a process supervisor.

1
2
3
4
5
6
7
8
9
php artisan queue:work redis \
    --queue=high,default,low \
    --tries=3 \
    --backoff=5,15,60 \
    --timeout=60 \
    --memory=256 \
    --max-jobs=1000 \
    --max-time=3600 \
    --sleep=3

What each flag does:

  • –queue=high,default,low — pull from these queues in priority order. high drains first.
  • –tries=3 — retry a failing job twice before sending it to failed_jobs.
  • –backoff=5,15,60 — wait 5s before the first retry, 15s before the second, 60s before the third.
  • –timeout=60 — kill the job if it runs longer than 60 seconds.
  • –memory=256 — restart the worker if memory usage exceeds 256MB. Cheap insurance against leaks.
  • –max-jobs=1000 — restart the worker after processing 1000 jobs. Also cheap insurance.
  • –max-time=3600 — restart the worker after running for an hour.
  • –sleep=3 — when the queue is empty, sleep 3 seconds before polling again. Only matters for the database driver; Redis uses blocking pops.

The maintenance commands you’ll reach for:

1
2
3
4
5
6
7
8
9
php artisan queue:failed                # list failed jobs
php artisan queue:retry all              # retry every failed job
php artisan queue:retry 5                # retry job with id 5
php artisan queue:forget 5               # delete a failed job
php artisan queue:flush                  # delete ALL failed jobs (careful)
php artisan queue:prune-failed --hours=48   # delete failed jobs older than 48h
php artisan queue:prune-batches --hours=48  # delete finished batches older than 48h
php artisan queue:clear redis high       # nuke all pending jobs on a queue
php artisan queue:restart                # signal all workers to gracefully restart

In production, you almost always run workers under Supervisor. A typical config looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work redis --queue=high,default --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/log/laravel/worker.log
stopwaitsecs=70

Note stopwaitsecs=70 — it should be greater than –timeout so Supervisor gives an in-flight job time to finish before sending SIGKILL.

Redis in the Mix

The database driver is fine for small applications, but it has two real costs: every poll is a SQL query (so –sleep matters), and lock contention on the jobs table grows nonlinearly with worker count. Switch to redis when you have more than a few workers or you need sub-second latency between dispatch and execution.

Behind the scenes, the Redis driver uses a handful of keys per queue. For a queue called default:

  • queues:default — the list of jobs waiting to be processed.
  • queues:default:delayed — a sorted set of jobs scheduled for the future. The score is the run-after timestamp.
  • queues:default:reserved — a sorted set of jobs currently being processed. The score is the lease expiration. If a worker dies, the lease expires and the job is re-queued.
  • queues:default:notify — a list used for blocking pops. This is the magic that makes Redis-backed queues feel instant: a worker does a BLPOP on this key and wakes up the moment a job is dispatched.

The atomic claim-a-job operation is implemented as a Lua script, so two workers can never reserve the same job. This is the part you really, really don’t want to reinvent yourself. 🛡️

For Redis-backed queues, the production answer is Laravel Horizon — a dashboard and process supervisor that replaces hand-rolled Supervisor configs. Install it, configure your queues and worker counts in config/horizon.php, and run:

1
2
3
php artisan horizon              # start the master + workers
php artisan horizon:terminate    # graceful shutdown for deploys
php artisan horizon:status

Horizon also gives you per-queue throughput graphs, recent job inspection, and runtime tagging — well worth the install if you’re already on Redis.

Gotchas (the actual reason you’re reading this)

These are the ones I’ve personally been bitten by. None are bugs; all are design tradeoffs you need to know.

1. Dispatching inside a database transaction

If you dispatch a job inside a DB::transaction(), a fast worker can pick it up before the transaction commits. The job then tries to load a row that doesn’t exist yet, and you get a ModelNotFoundException that’s impossible to reproduce locally.

Two fixes. Per dispatch:

1
SendInvoiceEmail::dispatch($invoice)->afterCommit();

Or globally, in config/queue.php:

1
2
3
4
5
6
7
8
9
10
'connections' => [
    'redis' => [
        'driver'        => 'redis',
        'connection'    => 'default',
        'queue'         => 'default',
        'retry_after'   => 90,
        'block_for'     => null,
        'after_commit'  => true,    // <-- this
    ],
],

2. retry_after MUST be greater than –timeout

The worker’s –timeout is how long the worker waits before killing the job. The connection’s retry_after is how long Laravel waits before assuming the job is dead and putting it back on the queue. If retry_after is less than –timeout, the supervisor re-queues the job while the original worker is still happily running it. You get two simultaneous executions and any side effect (emails, charges, webhooks) happens twice.

Rule of thumb: retry_after = timeout + 30.

3. Code changes are NOT picked up automatically

The worker boots the framework once and holds it in memory forever. Deploying new code doesn’t change the running worker’s behavior at all. You must run php artisan queue:restart after every deploy. That command signals the workers to exit cleanly, and Supervisor (or Horizon) starts them again with the new code.

Many hours of confused debugging have been spent on this. Make it part of your deploy script and never think about it again.

4. Jobs serialize their constructor arguments

The SerializesModels trait stores only the model’s class and primary key, then re-fetches the row when the job runs. This is great — except if the row has been deleted between dispatch and execution, the job blows up with ModelNotFoundException.

For jobs that should tolerate a soft-deleted parent, either pass the ID and load with withTrashed(), or override getRestoredPropertyValue(). For jobs where the row really must exist, log loudly enough that the failure is obvious.

5. failed_jobs only catches thrown exceptions

A job that never throws but loops forever or silently no-ops never reaches failed_jobs. The only safety net is –timeout — set it. A timeout-killed job goes to failed_jobs with a MaxAttemptsExceededException after retries are exhausted.

6. Bus::batch does not respect afterCommit by default

The per-dispatch afterCommit() method does not exist on the batch builder. If you dispatch a batch inside a transaction, wrap it yourself:

1
2
3
4
5
6
7
8
DB::transaction(function () use ($invoices) {
    // ...do transactional work

    DB::afterCommit(function () use ($invoices) {
        Bus::batch($invoices->map(fn ($i) => new SendInvoiceEmail($i)))
            ->dispatch();
    });
});

7. Closures are not queueable out of the box

dispatch(function () { … }) looks tempting but it requires laravel/serializable-closure and a signed app key, and many edge cases (use statements, non-serializable captures) break it. Don’t. Write a real job class.

8. Redis memory pressure with large backlogs

Every queued job lives entirely in Redis memory until it runs. A million jobs at 5KB each is 5GB of Redis. If your payload is large or your backlog is unbounded, either trim the payload (pass an ID, fetch on the other side), or use the database driver for that specific queue where backlog matters more than latency.

9. Horizon’s auto-balancer can starve long-running queues

Horizon’s auto balancing strategy reallocates workers based on queue length every few seconds. If a queue’s jobs are long-running (say, 10-minute video transcodes), Horizon sees “queue empty, reassign workers elsewhere” and the next batch waits. Use the simple strategy for that supervisor, or pin a minimum worker count.

Closing Thoughts

The queue layer is the boring kind of magical: it works exactly the same on top of sync, database, redis, or sqs. You develop against database, ship to redis, and your code doesn’t change. The cost of that abstraction is knowing the gotchas above — none of them are bugs, but all of them have bitten me at least once, and the diagnosis is much faster the second time. 🎉

If you take three things away: use afterCommit() whenever you dispatch inside a transaction, always run queue:restart on deploy, and keep retry_after greater than your worker timeout. Everything else, you can learn the hard way. 🛠️

Further Reading

  • Laravel Queues documentationlaravel.com/docs/queues. The canonical reference for jobs, batching, retries, and middleware.
  • Laravel Horizon documentationlaravel.com/docs/horizon. Configuration, balancing strategies, metrics, deployment.
  • Job middlewarelaravel.com/docs/queues#job-middleware. Covers WithoutOverlapping, rate limiting, and skipping.
  • Bus::batchlaravel.com/docs/queues#job-batching. Including the before, progress, then, catch, finally callbacks and allowFailures().
Posted in Laravel | Tagged , , , | Comments Off on Laravel Jobs, Queues, Batches, and Redis: A Field Guide