Cutting Solr Requests per Page View from Three to One

The Drupal platform I work on has dozens of sites, each with listing pages built on Search API and Solr: search results, articles, products, and a few more. They worked, but they felt slow. A filter click took somewhere between 700 ms and 1.5 seconds,. Before tuning anything, I wanted to know exactly what a single page view cost.

Make the requests visible

Solr traffic is invisible from the browser. The page talks to Drupal, and Drupal talks to Solr. To see what was actually sent, I installed Search API Solr's development submodule locally. It logs every request Drupal sends to Solr, with its parameters, and every response:

# ships with search_api_solr, needs the devel module
composer require --dev drupal/devel
drush en search_api_solr_devel -y

Opening one listing page and reading the log showed the problem right away. One page view meant three Solr requests:

  1. The expected one. Drupal renders the listing block, queries Solr, and gets results.
  2. A second one from Views. Every Solr-backed view had an Attachment display in addition to the block. They were left over from the original site build and weren't used anywhere. But an attachment is rendered together with the display it's attached to, so each block render ran its query twice.
  3. A third one from the browser. After the page loaded, front-end code sent the same query again through Views AJAX, with the same filters, before the user had touched anything.

So the cost of a page view was three times what it should have been, and the most expensive request of the three happened after the page already looked finished.

Why the browser asked again

The third request made the least sense, so I followed it. On pages with a "load more" button, the load-more link didn't carry the currently selected filter tags. If the user clicked it, the next page would have ignored the filters. My guess is that the extra request on page load was added deliberately to work around that: refresh the listing through AJAX so the JavaScript state, including the link, would be right.

It worked, but it paid for a broken link with a Solr query on every page view.

Why filter clicks were slow

The second finding was in the response headers. The Views AJAX endpoint that serves filtered results didn't send a cacheable Cache-Control header, so the CDN treated every response as uncacheable. Every filter click went through to the origin, Drupal ran the view, and Solr ran the query. That was where the 700 ms to 1.5 s went.

Yet those responses are perfect cache material. For the same set of filter tags, the endpoint returns the same HTML to everyone.

The fixes

# before: every filter change reached the origin
cache-control: no-cache, private

# after: the same filters return the same HTML, so the CDN can keep it
cache-control: public, max-age=...

A cache in front of a search endpoint only works if the URL fully describes the result. Here it does: the filters are query parameters, and there's nothing personal on the page. If your listing shows anything user-specific, this is the point where you need to stop and think about Vary or a different cache key.

Rolling it out across many themes

The front-end change lived in the themes, and the platform has several of them. So the rollout went theme by theme: pick one site per theme, clear that site's theme cache, test its listing pages, and only then clear the theme cache on every other site using it. That way a problem in one theme shows up on one site, not on all of them.

Result

One Solr request per page view instead of three, and filter clicks served from the CDN in about 30 ms instead of 700 ms to 1.5 s. None of it required touching Solr itself. It matters even more on a slower search backend: the platform also ran on a search provider that was several times slower per request, and every request we didn't send was one less to wait for. I wrote about that migration in Migrating 88 Drupal Sites from Acquia Search to SearchStax in 6 Hours.

The lesson I took from it: before you optimize a backend, count how many times you call it. The cheapest request is the one you don't send.

← All posts