Dispatcher Caching & Invalidation

In the Foundations series we met the Dispatcher as the caching and security layer in front of Publish. In production it is the single biggest factor in how fast your site feels and how much load Publish carries. It's also the source of the most common production complaint in AEM: "I published the change, but the site still shows the old content." This post explains how Dispatcher caching works, how invalidation works, and how to configure and debug both.

How the Dispatcher Cache Works

The Dispatcher cache is file-based. When a request arrives:

1. Dispatcher checks whether it may cache the request
2. If a valid cached file exists in its docroot, it serves the file directly and Publish never sees the request
3. If not, it forwards the request to Publish, and if the response is cacheable it saves the response as a file that mirrors the URL path

So /content/mysite/en/home.html becomes a file at <docroot>/content/mysite/en/home.html.

What Dispatcher Will and Won't Cache

By default, Dispatcher caches a response only when all of these are true:

• The request is a GET (or HEAD)
• The URL has a file extension (.html, .json, .css and so on)
• The URL has no query string (unless you configure /ignoreUrlParams)
• The response status is 200
• The request isn't authenticated, unless you enable /allowAuthorized
• The response doesn't carry headers that forbid caching, such as Dispatcher: no-cache, Cache-Control: no-cache or private, or Pragma: no-cache
• The path isn't denied by your cache rules

Knowing this list solves a large share of "why isn't this cached?" questions.

The Key Configuration Pieces

In a classic (6.5) Dispatcher farm file, the /cache section holds the important settings:

/cache
  {
  /docroot "/opt/dispatcher/cache"
  /statfileslevel "3"
  /allowAuthorized "0"
  /enableTTL "1"

  /rules
    {
    /0000 { /glob "*" /type "allow" }
    /0001 { /glob "/content/mysite/*/account/*" /type "deny" }
    }

  /invalidate
    {
    /0000 { /glob "*" /type "deny" }
    /0001 { /glob "*.html" /type "allow" }
    }

  /ignoreUrlParams
    {
    /0001 { /glob "*" /type "deny" }
    /0002 { /glob "utm_*" /type "allow" }
    }
  }

What each does:

Query Parameters and Cache Hit Ratio

Marketing links like ?utm_source=email make every URL "unique", so each one bypasses the cache. The recommended pattern is an allow-list of ignorable parameters: deny everything, then allow only parameters that never change the content, such as utm_*. In /ignoreUrlParams, an allow rule means "ignore this parameter when deciding whether to cache."

Be careful with the shortcut of allowing *: it would cache one version of a page for every parameter value, including ones that really do change the output (like ?page=2).

How Invalidation Works

Dispatcher doesn't usually delete cached files when content changes. Instead it uses .stat files:

1. When an author activates a page, a flush request reaches Dispatcher
2. Dispatcher touches a .stat file at a certain directory level
3. For any cached file, if the .stat file is newer than the cached file, Dispatcher treats the cached file as stale and fetches a fresh copy from Publish on the next request

Only files that match the /invalidate rules are affected, which is why static files (images, CSS, JS) usually survive an activation while HTML is refreshed.

Understanding /statfileslevel

/statfileslevel decides how much of the cache one activation invalidates. When a file is updated, Dispatcher finds the folder at the configured level and invalidates everything in and below it.

For a typical multi-brand, multi-country, multi-language structure like /content/brand/country/language/...:

Pick the level that matches how your content tree is organized. Too low a number and a single edit flushes far more than necessary (hurting performance on busy sites). Too high a number and related pages, such as navigation that appears on every page, may not be refreshed when you expect.

Delete vs Invalidate

A flush request carries an action:

• Activate: invalidate, by touching the .stat file
• Delete/Deactivate: remove the cached file for the specific path, as well as invalidating

If you want "refresh now" behavior for one resource, a delete flush on that path is the most direct.

Time-Based Invalidation (TTL)

With /enableTTL "1", Dispatcher honors Cache-Control: max-age and Expires headers on responses. A file expires when its TTL passes, even if no activation happened. This is useful for content that changes without being activated, such as feeds, listings or anything driven by external data. Set the headers from Apache (mod_headers) or from your Publish responses.

Who Sends the Flush Request?

On AEM 6.5 or AMS: a Dispatcher Flush replication agent configured on Publish sends an HTTP request to the Dispatcher's /dispatcher/invalidate.cache endpoint after content is replicated. The request carries headers such as CQ-Action, CQ-Handle and Content-Length. The Dispatcher's allowed-clients configuration decides who may send flush requests.

On AEM as a Cloud Service: invalidation is wired up for you. Activating content triggers invalidation automatically, and the Dispatcher configuration you deploy through Cloud Manager includes the flush plumbing. A CDN also sits in front of Dispatcher, so you now have two caches to think about (covered below).

Dispatcher on AEM as a Cloud Service

Cloud Service projects include a dispatcher module based on the Dispatcher SDK. Key differences from a 6.5 setup:

• The configuration is split into folders such as conf.d (Apache) and conf.dispatcher.d (Dispatcher), with sub-folders for available_farms, cache, filters, clientheaders, renders and virtualhosts
• Some files are immutable: Adobe controls them, and the validator rejects changes. A common example is default_invalidate.any
• You validate your configuration locally before deploying, with the SDK's validator
• You can run Dispatcher locally in Docker to test rules and caching before they reach a real environment

Typical commands (from the SDK folder):

# Validate your configuration
./bin/validator full -d out src

# Run Dispatcher locally against your local AEM Publish
./bin/docker_run.sh out host.docker.internal:4503 8080

Script names and arguments can change between SDK versions, so check the README that ships with your SDK download.

Local testing gotcha: because of the immutable files, a local Dispatcher may not behave exactly like a cloud environment for flush testing. If invalidation looks broken locally (for example, the log says a file was "never flushed" or no .stat files appear), confirm the behavior on a cloud Dev environment before assuming your config is wrong.

The CDN Layer

On Cloud Service, a CDN caches in front of Dispatcher. That means:

• Cache-Control headers you set matter twice: they influence the CDN and, with TTL enabled, Dispatcher
• Invalidating Dispatcher does not instantly purge the CDN. A page can be fresh in Dispatcher but stale at the edge until its TTL passes or it is purged
• Set sensible TTLs per content type (short for HTML, long for versioned clientlibs and images) and plan how you will purge in an emergency. Check Adobe's current documentation for the supported purge options

Debugging Stale Content

When "my change isn't showing", work from the user's browser inward:

1. Browser: hard refresh, or test in a private window, to rule out browser caching
2. CDN (Cloud Service): inspect response headers (curl -I https://yoursite/page.html) and look at Age and Cache-Control
3. Dispatcher: check whether the cached file exists and its timestamp, and compare it with the .stat file at your /statfileslevel folder
4. Publish: request the page directly from Publish, bypassing Dispatcher. If it's wrong here, the problem is replication, not caching
5. Author: confirm the page was actually published, and wasn't just saved

Other things to check:

• Was a flush sent? Look in the flush agent log (6.5) or the Dispatcher log
• Does the path match an /invalidate rule? A .json or .xml file won't be invalidated if only *.html is allowed
• Is /statfileslevel too deep or too shallow for the pages that changed? Shared components, like a header, live in a different folder from the page that changed
• Did Publish send no-cache headers, making files uncacheable in the first place?

Temporarily raise the Dispatcher log level to see per-request decisions (why a request wasn't cached, which .stat file applied). Lower it again afterward, since debug logging is noisy.

Best Practices

• Cache as much as possible, and treat every uncacheable page as a deliberate decision
• Pick /statfileslevel to match your content tree, and keep the tree consistent
• Use an allow-list for ignorable query parameters
• Keep personalized content out of cached HTML, and load it client-side or with Sling Dynamic Include (SDI)
• Use versioned clientlibs so static assets can have long TTLs
• Review /rules and /filter together, since cache rules decide what's stored and filters decide what's reachable at all
• On Cloud Service, validate locally and test on Dev before promoting Dispatcher changes
• Monitor your cache hit ratio. A healthy site serves the vast majority of HTML and assets from cache

What's Next

Dispatcher invalidation depends on content actually reaching Publish. In the next post we look at replication, how content moves from Author to Publish on 6.5 and on Cloud Service, and how to troubleshoot it when it doesn't.

Next in this series: [Troubleshooting Replication Agents]