bilaldemir.dev

Running a Workflow Platform on 100 MB of RAM Without Redis

Self-hostable workflow tools tend to have a heavy footprint. Before you've built a single workflow, a typical install wants a database, Redis for queues and caching, a worker fleet, sometimes a message broker. For a team that wants to try a tool on a small VPS, that is a lot of moving parts.

BlockNext takes the opposite default. A fresh install runs seven containers and no Redis, and idles at around 100 MB of RAM. PostgreSQL accounts for nearly half of that. Turn on every optional piece and it becomes ten containers at roughly 160 MB. The Go service images are 15 to 32 MB each, built on distroless.

None of this comes from heroic optimization. It comes from one rule: scale-out infrastructure is opt-in.

What Redis is usually for

In a platform like this, Redis typically does five different jobs:

Every one of those exists to share state between instances. A fresh install has exactly one instance. With one process, "shared state" is just memory, and a mutex does the job of a distributed lock.

An in-process provider for everything

Each of those five subsystems is an interface with two implementations: memory and redis. The cache, for example, is a small interface that the rest of the code depends on, and neither the callers nor their tests know which implementation is behind it:

type Service interface {
	// Ping verifies connectivity to the backend.
	Ping(ctx context.Context) error
	// Get returns the value for key, or an empty string if it is absent.
	Get(ctx context.Context, key string) (string, error)
	// Set stores value for key with the given expiration.
	Set(ctx context.Context, key string, value string, expiration time.Duration) error
	// Incr atomically increments the integer value of key and returns the result.
	Incr(ctx context.Context, key string) (int64, error)
	// ...
}

The defaults in .env.example pick the in-process side for all of them:

CACHE_TYPE=memory
BROKER_TYPE=memory
TASK_RUNNER_MODE=embedded
TASK_RUNNER_LEADER_PROVIDER=memory
SEMAPHORE_PROVIDER=memory

The memory versions aren't toys. The semaphore, for example, is a map of organization to holder to expiry behind a mutex, with TTLs and heartbeats. It has the same semantics as the Redis one, just without the network.

The interface has to be designed for the distributed case from the start, though. A memory cache with a Get and a Set is easy. But the code that refreshes OAuth tokens needs an atomic "acquire if free" and an atomic "get and delete". Those have to be in the interface even if the memory version could fake them with a lock. Otherwise switching to Redis later quietly introduces race conditions.

Containers that only start when needed

The second half of the trick is in Docker Compose. Every optional service sits behind a profile named after the setting that would use it, and the active profiles are built from those same settings:

COMPOSE_PROFILES=cache-${CACHE_TYPE},broker-${BROKER_TYPE},lock-${TASK_RUNNER_LEADER_PROVIDER},semaphore-${SEMAPHORE_PROVIDER},task-runner-${TASK_RUNNER_MODE}
cache-redis:
  image: redis
  profiles:
    - cache-redis
    - broker-redis
    - lock-redis
    - semaphore-redis

task-worker:
  profiles:
    - task-runner-queue

With the defaults, the active profiles are cache-memory, broker-memory and so on. No service has those profiles, so no Redis container starts. Switch CACHE_TYPE to redis and the profile becomes cache-redis, which starts Redis. Switch TASK_RUNNER_MODE to queue and both the queue Redis and the task-worker container appear.

There's one source of truth. Nobody has to remember to edit the env file and the compose file, and they can't drift apart.

Durable events: Postgres, not a broker

Some messages may be lost. A progress update to a browser tab that was just closed doesn't matter. Others must not be lost: "this workflow run finished" has to reach whatever reacts to it. BlockNext keeps those two concerns separate. Live updates go through ephemeral pub/sub. Durable events go through a transactional outbox in Postgres.

The event is inserted into an outbox table in the same transaction as the change that caused it. If the transaction rolls back, the event never existed. A relay then claims pending rows in batches:

UPDATE eventbus.outbox_messages AS o
SET
	status = 'processing',
	attempts = o.attempts + 1,
	locked_at = $1
FROM (
	SELECT id AS due_id
	FROM eventbus.outbox_messages
	WHERE
		status = 'pending'
		AND next_attempt_at <= $1
	ORDER BY occurred_at
	LIMIT $2
	FOR UPDATE SKIP LOCKED
) AS due
WHERE o.id = due.due_id
RETURNING ...

FOR UPDATE SKIP LOCKED is what makes this safe with more than one relay. Each relay locks the rows it picked and skips rows another relay already holds, so no event is processed twice at the same time and nobody waits. Failed deliveries get exponential backoff through next_attempt_at, and messages that keep failing end up dead-lettered. A relay that crashed mid-batch would leave rows stuck in processing, so a periodic query puts them back:

UPDATE eventbus.outbox_messages
SET
	status = 'pending',
	locked_at = NULL
WHERE
	status = 'processing'
	AND locked_at < $1

The relay polls instead of using LISTEN/NOTIFY. That costs a little latency, but it's simple, and it works the same whether the relay runs inside the main API process or as a separate event-relay-worker.

The rest of the diet

When to turn Redis on

The moment you run a second instance of the API. Then the cache, the scheduler leader, the semaphore and the live-update broker all need to be shared, and each one can be switched over independently. Queue mode is the next step, when workflow runs should be processed by separate workers instead of inside the API.

Until then, a single instance on in-process providers is not a degraded mode. It's the right setup for that size, and it's what lets someone try the platform on the smallest VPS they have.

← All posts