Queues are not a database

A queue is a pipe. It is very good at holding a message for a few seconds while something else gets ready to read it. Every problem I have had with queues came from asking one to hold state instead.

The symptoms

• You query the queue to find out whether something happened
• You need a message to stay until "later", where later is not defined
• You care which message is second in line
• You want to update a message that is already queued

Each of these is a database question being asked of a pipe.

What to do instead

Write the fact to the database first, then queue a pointer to it.

const { insertedId } = await db.collection("jobs").insertOne({
  type: "reindex",
  status: "pending",
  createdAt: new Date(),
});

await channel.sendToQueue("jobs", Buffer.from(String(insertedId)));

The queue now carries twelve bytes and no meaning. If a worker dies mid-job, the row is still there with status: "pending", and a sweeper picks it up. If the broker loses the message entirely — which it will, once — the sweeper still picks it up. The queue became an optimisation rather than a source of truth, which is the only job it is good at.

The one that cost the most

We once stored delivery slots as scheduled messages, because the broker supported delays and it saved writing a table. It worked until we needed to answer "which slots are booked for Thursday". There was no way to ask. We rebuilt it against a table in two days, and the delay feature went back to being what it should have been: a retry timer.