9 September 2026¶
1 change:
- The queue measured itself and then deleted the measurements
The queue measured itself and then deleted the measurements¶
execution_time and cpu_time are recorded per task. queue:cleanup deleted them an hour
later, and nothing aggregated them first — so on an installation running it
@hourly --hours=1, completed rows survived about seventy-five minutes and then the numbers
were gone for good.
Three consequences, in the order of what they cost:
1. queue:health --window gave a wrong answer past the retention, silently.
throughput() counts arrivals and completions among rows that are still there. Ask for a
day — which the option invites — and it counted the ninety minutes that survived, reported
the rate as if that were the day, and exited 0. Not an error, not a warning: a number
that looks like the answer to the question that was asked.
2. Wait time was recorded nowhere at all. The most useful number a queue has — how long
a task sat before a worker took it — existed only as startedat - createdat on rows about
to be deleted. throughput() did not compute it, and after the purge nothing could.
Running the aggregate over one installation's live purge set, before any of this was written:
01:00 16,130 tasks avg wait 9.48 s max wait 40 s
others ~2,300 tasks avg wait 0.60 s max wait 1–10 s
A nightly burst where queue latency went fifteen times normal. Invisible that night and every night before it. Nothing in the framework would have shown it, and nothing recorded enough to show it afterwards.
3. cpu_time was collected and thrown away — a column added two days earlier, and an
installation still could not answer "is this task type slower than last week".
Added¶
queuestats, one row per task type per hour, written byQueueManager::purgeOldTasks()in the same transaction as theDELETEit summarises. That equivalence — a counted row is a deleted row — is the whole correctness argument: no row can be summarised twice, and none deleted without being summarised.
In purgeOldTasks() and not in queue:cleanup, which is the design decision rather
than an implementation detail: the method is what destroys the data, so every caller is
covered — the framework's command, an application's own cron, a manual console call. In
the command it would be one more thing to remember, and an installation that has forked
the command (the reporting one had) could not adopt it at all.
~104 ms over a purge set of 40,840 rows producing 12 buckets, once an hour, on the connection that was about to scan the same rows anyway.
-
QueueManager::history()besidethroughput(). The difference between them is the point:throughput()reads live rows and can only answer for as long as the retention keeps them; this reads the buckets, which outlive them. -
queue:health --history[=hours], which renders it — and the live path now says when its window reaches past the surviving rows. Detected rather than configured: the retention is an argument on somebody else's crontab, and the oldest surviving completion is a fact the connection has (QueueManager::retentionHorizon()).
The shape, and what it deliberately is not¶
| Volume | one row per type per hour — three types is ~72 rows a day, 26k a year |
| Therefore | no hypertable, no compression, no retention policy, no spool |
| Stored | sums, counts and maxima. Never averages |
| Absent | percentiles |
The change log needed hypertables because volume was the problem there. Here there is none,
and anything heavier than a plain table with PRIMARY KEY (type, bucket) would be building
for a problem that does not exist.
Sums and maxima, never averages, because that is what makes the upsert additive and additive is what makes it safe: a second cleanup inside the same hour adds to the bucket instead of overwriting it. Means are derived at read.
Percentiles are deliberately absent. percentile_cont does not merge across two inserts
into one bucket, so storing p95 would mean either a wrong number or a second pass over rows
that no longer exist. max and the derived mean find the problem — the burst above shows in
both.
Statuses are broken out, and the framework's own retention is why. queue:cleanup keeps
warnings for $hours * 10, which on that installation made warning the dominant
population: 38,041 rows against 25,861 completed. A roll-up that counted only "tasks" would
have described the small half.
Two things a test found¶
-
max_execcould be replaced instead of taken. The first test for the additive upsert purged the smaller task second, somax_exec = VALUES(max_exec)— a replace rather than aGREATEST— gave the same answer and removingGREATESTreddened nothing. Ordering the arrival the other way is the only way that assertion means what it says: a maximum overwritten by a later, smaller purge reports the last hour instead of the worst. -
schedule:list's new line reported the opposite of the truth. It askedWorkerLock::isHeld(), which answers did this object take the lock — always false for an object the command just constructed. So it said "noworkprocess" on an installation where one was running, which is the same class of wrong answer the line exists to prevent.isHeldByAnother()reads the file and checks the pid.
schedule:list says whether anything is executing the schedule¶
Not part of the roll-up, and from the same report. Listing reads definitions; running needs a process, and nothing checked for one.
That installation had forked DaemonOrchestrator without extending the framework's, so
includeScheduler() and schedulerProcess() never ran — no work process, no
schedule:run in the crontab. schedule:list printed six due jobs and none of them had
ever executed: spool:drain every minute, timescale:drain hourly,
auth:token-cleanup, auth:twofactor-cleanup, mail:prune and auth:webhook-deliver
every five minutes. It took reading ps to find.
The command now ends with a line saying whether a work process holds the lock, how stale
its heartbeat is, and — when it does not — that these n tasks are registered and not
executing, plus the caveat that a schedule:run crontab line cannot be detected from there.
So the absence of the message is never a claim that nothing runs.
A fork of the orchestrator is a silent way to lose the whole schedule, and this is the command somebody runs when they wonder about it.
What a --limit purge does¶
Nothing, to the roll-up, and it says so. DELETE … LIMIT removes an arbitrary subset of what
the predicate matches, so an aggregate over the predicate would count rows that are still
there and the next call would count them again. Getting it right would mean selecting the ids
first and aggregating exactly those — one statement into three, and a bounded purge holding a
list. The limit exists to keep one DELETE off a lock on a table left for months; the normal
path summarises.
Documentation¶
Pramnos_Queue_Guide.mdgains The history the queue used to delete, with the burst that prompted it, the table of what is stored and why percentiles are not, and a warning box on--windownot reaching past the purge.Pramnos_Workers_And_Daemons_Guide.md— the schedule section gains the box onschedule:listand the orchestrator fork, including the instruction to extendPramnos\Console\DaemonOrchestratorrather thanCommandBase.