bilaldemir.dev

Migrating 88 Drupal Sites from Acquia Search to SearchStax in 6 Hours

One of the platforms I work on is a Drupal multisite on Acquia Site Factory: 88 brand and country sites sharing a single codebase, each with its own database, domain and Solr core. Search ran on Acquia Search. Then the decision came to move every site to SearchStax.

The first estimate for doing it by hand was one and a half to two weeks. Create an app per site in the SearchStax UI, copy the host, core, endpoint and token into a spreadsheet, paste them into each site's Search API server, reindex, check, repeat 88 times. It was also the kind of work where one copy-paste mistake silently breaks search on one site and nobody notices for a month.

In the end the switch took six hours, with zero downtime. This is how.

Rule one: never swap in place

The tempting approach is to edit the existing acquia_search_server and point it at SearchStax. I did not do that. Instead the new setup shipped as a second Search API server and a second index, searchstax_server and searchstax_index, living next to the old ones.

That gave me a blue-green switch. Both backends were configured and indexed at the same time, and "which one is live" came down to which index the search views were reading from. Rolling back was a configuration change, not a restore. Once that was true, the rest of the migration stopped being scary.

Talking to an API that is not really documented

SearchStax has a UI for creating apps, and that UI talks to a REST API. The public documentation did not cover the parts I needed, so I opened the browser's network tab, did each step by hand once, and wrote down the requests: create app, list apps, get one app, list models, set search fields, publish, upgrade version, read usage.

On the Acquia side things are easier. Site Factory has a proper API that lists every site on a stack together with its database name and custom domain. The database name is the stable ID I used everywhere, including the app names on SearchStax, so the two lists could be joined later.

// 1. Every site on the stack, with its custom domain
async function getAllSites() {
  const res = await fetch(
    ACSF_API + "/sites?stack_id=" + STACK + "&show_custom_domain=yes&limit=1000",
    { headers: { Authorization: "Basic " + ACSF_TOKEN } },
  );
  return (await res.json()).sites;
}
// 2. One SearchStax app per site, named <prefix>_<db_name>
async function createApp(site) {
  const res = await fetch(SEARCHSTAX_API + "/experience-manager/v2/apps?account=" + ACCOUNT, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: "Token " + SEARCHSTAX_TOKEN,
    },
    body: JSON.stringify({
      name: PREFIX + "_" + site.db_name,
      default: false,
      environment: "Production", // Development / Staging for dev and test
      platform_version_id: PLATFORM_VERSION,
      plan_region_id: REGION,
    }),
  });
  return res.json();
}

for (const site of sites) {
  await createApp(site);
  await sleep(PAUSE); // SearchStax does not like bursts
}

The toolkit was a single file of small numbered steps that I ran one by one from a console: load sites, create apps, fetch app details and merge them into the site list, generate config commands, set search fields, upgrade app versions, collect usage. Numbering the steps sounds trivial, but when you run a migration against 88 sites you want to see exactly which step you are on and re-run only that one.

Generating the Drush commands instead of typing them

Each site needs four values on its Search API server: host, core, update endpoint and write token. All four come from the SearchStax app. So the script read every app, matched it back to the site by its database name, and printed one long chain of drush cset commands with each site's own --uri.

// 3. Turn the API response into one chained Drush command per site
function configCommand(site) {
  const url = new URL(site.searchstax.updateEndpoint);
  const core = url.pathname.replace("/update", "").split("/").at(-1);
  const uri = "--uri=" + site.domain;
  const key = "search_api.server.searchstax_server backend_config.connector_config";

  return [
    "drush cset " + key + ".host " + url.host + " -y " + uri,
    "drush cset " + key + ".core " + core + " -y " + uri,
    "drush cset " + key + ".update_endpoint " + site.searchstax.updateEndpoint + " -y " + uri,
    "drush cset " + key + ".update_token " + site.searchstax.writeToken + " -y " + uri,
  ].join(" && ");
}

console.log(sites.map(configCommand).join(" && "));

I pasted that output into an SSH session on the Acquia server and it configured all 88 sites in one go. Nobody typed a token, and nobody pasted one into the wrong site. The same trick produced the commands for the analytics URL and key.

One small thing that saved time: if the analytics app had failed to provision for a site, the script marked it and skipped that site instead of producing a broken command. I would much rather get a short list of exceptions to fix by hand than a chain that fails halfway through.

Getting the new config into existing sites

The new server and index were part of the exported configuration, but some sites had config states that made a plain import unreliable. For the two critical config objects I loaded the YAML straight into the config factory first, so the objects existed before the full import ran:

drush eval "
  \$data = \Drupal\Component\Serialization\Yaml::decode(
    file_get_contents('../config/default/search_api.server.searchstax_server.yml')
  );
  \Drupal::configFactory()
    ->getEditable('search_api.server.searchstax_server')
    ->setData(\$data)
    ->save();
"

After that the rollout was the usual sequence, run against every site at once with acsf-tools:

# run against every site on the stack with acsf-tools
drush @app.env sfml cim -y
drush @app.env sfml updb -y
drush @app.env sfml search-api-reindex
drush @app.env sfml search-api-index

Two custom Drush commands for the switch

Importing the new server and index doesn't move anything onto them. Every search page on every site is a View, and each of those Views names the index it reads from. Changing them by hand on 88 sites wasn't an option, so I wrote two small Drush commands in one of our custom modules, and ran them on every site with acsf-tools:

drush @app.env sfml migrate:acquia-to-searchstax
drush @app.env sfml migrate:search-api-none-to-tag

migrate:acquia-to-searchstax is the actual switch. It goes through every View on the site and replaces the old index ID with the new one:

#[CLI\Command(
  name: 'migrate:acquia-to-searchstax',
  description: 'Replaces acquia_search_index with searchstax_index in all views configs.'
)]
public function migrateAcquiaToSearchstax(): void {
  $view_ids = $this->configStorage->listAll('views.view.');

  foreach ($view_ids as $view_id) {
    $config = $this->configFactory->getEditable($view_id);
    $data = $config->getRawData();

    $original_json = json_encode($data);
    $updated_json = str_replace('acquia_search_index', 'searchstax_index', $original_json);

    if ($original_json !== $updated_json) {
      $config->setData(json_decode($updated_json, true))->save();
      $this->io()->success("Updated: $view_id");
    } else {
      $this->io()->text("Skipped: $view_id");
    }
  }
}

Encoding the whole config to JSON and doing a string replace looks crude, and it is. But the index ID is a unique machine name, it can appear in many places inside a View (the base table, fields, filters, sorts), and the replacement has to catch every one of them. Walking the nested arrays by hand would have been more code for the same result. Views that don't mention the old index are skipped untouched, so the same command is safe to run on all 88 sites, whatever their search pages look like. It's also trivially reversible: swap the two strings and the site reads from Acquia Search again.

migrate:search-api-none-to-tag uses the same approach for something we found along the way. Search Views were set to the search_api_none cache plugin, so their results were never cached at all. The command replaces it with search_api_tag, Search API's tag-based caching, which keeps results until the index changes. The migration was touching every search View anyway, so it was the cheapest moment to fix that too.

Those two commands were the "switch" moment, and the only step that would ever need to be undone.

Search fields and versions, per app

A new SearchStax app does not know which fields matter for relevance. In the UI you pick them per app and press publish. With 88 apps that is a lot of clicking, so the script did the same three calls for each one: read the app's model ID, post the list of search fields (title, description, body), then publish. Another step upgraded each app to the platform version that matches the Drupal major version.

Pacing turned out to matter more than anything else here. Every call that created or changed something waited before the next one, and between the bulk steps I had to add a full minute: when SearchStax received changes too quickly, it started returning results that made no sense. On top of that, every change took at least five minutes to show up, and queries were three to four times slower than on Acquia Search. The script was never the bottleneck. The platform was.

Rehearse it, then rehearse it again

Before touching production, I ran the full migration on three or four sites in the test environment until search worked on all of them. Then I deleted every SearchStax app I had created and ran the whole thing again from scratch. The first run proves the steps work. The second proves the process works when you follow it exactly as written, with no leftovers from earlier attempts helping you along.

The QA team then went through a fixed checklist on the test sites:

Prepare production before the window

Some work was better done in the weeks before the migration, because each item made the migration itself smaller or safer:

Planning the window

With the rehearsal timings in hand, I could give a real estimate instead of a guess. For a batch of 23 sites it looked like this:

The app creation dominates everything else, and it's almost all waiting on SearchStax. Add one to two hours for the code deploy itself, scale it up to the whole production stack, and the estimate for the migration came out at 12 to 14 hours. That meant booking a long window.

The real migration took six hours, well under the estimate. By then every step had been run twice, and there was nothing left to figure out on the day.

Cleaning up

Once search had been running on SearchStax for a while without complaints, the old setup went away: the Acquia index and server configs were deleted, acquia_search and acquia_connector were uninstalled and removed from composer.json.

drush @app.env sfml cdel search_api.index.acquia_search_index
drush @app.env sfml cdel search_api.server.acquia_search_server
drush @app.env sfml pmu acquia_search
drush @app.env sfml pmu acquia_connector

Things I would tell someone doing the same

The scripts themselves were not clever. What made it fast was the design around them: a reversible switch, one stable ID, and no step that relied on someone typing carefully.

← All posts