Redis pub/sub

A Dynamic Snapshot serves all your content from an in-memory snapshot on the server and hot-refreshes it when you publish, with no rebuild and no API call per read. The pattern has two pieces: where the snapshot is read from (snapshotLoader) and who triggers the refresh (refreshSnapshot()).

In this version the JSON lives in Redis. A publisher exports the snapshot with exportSnapshot(), does SET + PUBLISH, and all subscribed instances refresh at once with a single message.

How does it work?

• Source: Redis. The snapshotLoader reads the snapshot key.
• Trigger with pub/sub: instead of one POST per instance, a single PUBLISH notifies all subscribed replicas and each one calls refreshSnapshot().

Note

With this wiring the trigger arrives through the Redis channel, so you no longer need the HTTP refresh endpoint as the main mechanism. The example keeps the POST /api/content-island/refresh from the endpoint version only as an optional manual refresh.

When to choose it?

• Designed for horizontal scaling: refresh N instances with a single message, without per-instance webhooks or N API calls.
• Requires a Redis reachable by all replicas.

Step by step

Start from the endpoint version project. We swap the snapshotLoader for Redis, add the subscription and a publisher.

1. Start Redis and install the dependencies:

   docker run -d --name redis -p 6379:6379 redis:7
   npm i redis

   .env

   REDIS_URL=redis://localhost:6379

2. Create the Redis connection. We register an error handler so a dropped connection doesn’t take down the process:

   src/lib/redis.ts

   import { createClient as createRedis } from 'redis';
   
   export const redis = createRedis({ url: process.env.REDIS_URL });
   
   // node-redis throws on unhandled 'error' events; we log them so a dropped
   // connection doesn't take down the process.
   redis.on('error', err => console.error('Redis error', err));
   
   export const redisReady = redis.connect();

3. Change the snapshotLoader to read from Redis and subscribe to the channel. On first use, each instance listens and refreshes itself when the publisher notifies. If the snapshot doesn’t exist in Redis yet, we don’t cache the failure, so the next request retries:

   src/lib/content-island.ts

   import { createClient } from '@content-island/api-client';
   import { redis, redisReady } from './redis';
   
   const accessToken = process.env.CONTENT_ISLAND_TOKEN!;
   const SNAPSHOT_KEY = 'content-island:snapshot';
   const CHANNEL = 'content-island:updated';
   
   // Express runs as a single long-lived process: a module-level singleton is
   // enough to share a client (and a snapshot) across requests.
   export const contentIslandClient = createClient({
     accessToken,
     mode: 'snapshot',
     // The loader reads the snapshot from Redis.
     snapshotLoader: async () => {
       await redisReady;
       return (await redis.get(SNAPSHOT_KEY)) ?? '';
     },
   });
   
   // On first use, this instance subscribes to the channel and refreshes itself
   // every time the publisher notifies. It also does the initial snapshot load.
   let primed: Promise<unknown> | null = null;
   export function ensureSnapshot() {
     if (!primed) {
       primed = (async () => {
         await redisReady;
         // A Redis connection can't subscribe and run commands at the same
         // time, so the subscriber uses its own duplicated connection.
         const sub = redis.duplicate();
         sub.on('error', err => console.error('Redis subscriber error', err));
         await sub.connect();
         await sub.subscribe(CHANNEL, () => {
           contentIslandClient.refreshSnapshot().catch(console.error);
         });
         await contentIslandClient.refreshSnapshot(); // initial load
       })().catch(err => {
         // Don't cache the failure: if the snapshot doesn't exist in Redis yet,
         // the next request retries instead of reusing a rejected promise.
         primed = null;
         throw err;
       });
     }
     return primed;
   }

4. Create the publisher: it exports, does SET and PUBLISH. Outside the server, it loads the .env with dotenv:

   scripts/publish-redis.mts

   import 'dotenv/config';
   import { exportSnapshot } from '@content-island/api-client';
   import { createClient as createRedis } from 'redis';
   
   const redis = createRedis({ url: process.env.REDIS_URL });
   await redis.connect();
   
   const snapshot = await exportSnapshot({ accessToken: process.env.CONTENT_ISLAND_TOKEN! });
   await redis.set('content-island:snapshot', JSON.stringify(snapshot));
   await redis.publish('content-island:updated', '1');
   
   await redis.quit();
   console.log('✅ snapshot published to Redis');

   Add a script to your package.json to launch it:

   package.json

   "scripts": {
     "publish:redis": "tsx scripts/publish-redis.mts"
   }

5. Try it with two instances (that’s the whole point). Do a first load into Redis and start two APIs on different ports:

   npm run publish:redis    # first load into Redis
   
   PORT=3000 npm run dev    # in one terminal
   PORT=3001 npm run dev    # in another

   Publish new content and run npm run publish:redis just once. Query GET / on both instances: both update their exportedAt at the same time, with a single PUBLISH.

Example

Full code (includes the docker-compose.yml): content-island/examples-dynamic-snapshot → express/03-redis-pub-sub.

References

• Snapshot mode · exportSnapshot() · refreshSnapshot()
• Redis (Docker)