Notifier, now with room to grow

My homelab has a handful of apps that need to tell people things: an email here, a Slack message or a phone push there. Rather than give every app its own SMTP credentials and retry logic, they all hand messages to one small service I wrote: notifier.

Notifier is a Go service with a REST and a gRPC API. You give it a notification ("email this person, using this account") and it takes care of the rest: it queues the message, sends it through one of the current impls for SMTP, Slack, or ntfy, retries with backoff when a provider hiccups, and lets you look up what happened later. My toolbox app, Forge, uses it to send Secret Santa assignments, among other things.

The problem: one copy, all in memory

Until now, notifier kept everything in memory: the queue, the history of what had been sent, even the API keys and rate limits. For a single instance that's simple and fast. But it meant:

• It could only ever run as one copy. A second copy would have its own separate memory: a notification sent through one would be invisible to the other.
• Restarts lost work. Anything queued, or waiting to retry, disappeared on every deploy.

The fix: Postgres as both memory and to-do list

Notifier can now keep its state in Postgres. A single notifications table is both the record of every message and the work queue. Each copy of the service has a few workers, and they grab work like this:

UPDATE notifications
SET status = 'processing',
    locked_until = now() + interval '60 seconds'
WHERE id IN (
  SELECT id FROM notifications
  WHERE status IN ('queued', 'retrying')
    AND next_attempt_at <= now()
  ORDER BY next_attempt_at
  LIMIT 5
  FOR UPDATE SKIP LOCKED
)
RETURNING *;

FOR UPDATE SKIP LOCKED is the key part: if another copy is already claiming a row, Postgres skips it instead of waiting, so any number of copies can pull from the same queue without stepping on each other. Each claim is a lease. If a copy dies mid-send, the lease expires and another copy picks the message up. Retries and scheduled sends are just a future next_attempt_at, so they survive restarts too.

API keys and rate limits moved into Postgres as well, so a key revoked on one copy stops working on all of them.

The in-memory mode is still there and still the default, because it's perfect for development or a single small instance.

Things I found along the way

A change like this is a good excuse to look hard at everything around it:

• Cancelling didn't actually cancel. The notification was marked cancelled but still sent. Same story for retrying: it could send twice.
• Scheduled sends weren't scheduled. The API accepted a "send at" time and then ignored it.
• A typo in the config file turned authentication off. On a parse error, the service quietly fell back to built-in defaults. Now a bad config is a hard failure.
• The vulnerability scanner hadn't been scanning. Once it actually ran, it found 14 known vulnerabilities in the shipped binary, mostly because Go 1.25 had stopped getting security fixes. Moving to Go 1.26 and a few dependency bumps cleared all of them.

My favourite bug of the bunch only appeared in testing: notifications would occasionally sit in the queue for a few milliseconds longer than they should. Postgres was running in a container whose clock was slightly behind my system's, and the service was stamping "due now" with its clock while Postgres compared against its own. The fix was to let the database's clock be the only clock that matters.

What's next

Version v0.1.8 is running on one of my clusters now, still as a single pod using the in-memory store. Moving it onto Postgres and scaling to multiple replicas is next.

After that, I want better visibility: real delivery counters and a "how far behind is the queue" metric I can alert on.

The code is on my Gitea if you want to poke around.