Workers & Daemons Guide¶

Everything for console commands, long-running workers, and the daemon supervisor — one place, with worked examples for every case.

The short version. Three classes cover almost every case:

CommandBase — the base for every console command (adds a single-instance lock).

ProcessQueue — a ready-made queue worker (extends CommandBase).

DaemonOrchestrator — supervises long-running daemons.

The daemon plumbing (locking, heartbeats, stop-signals, systemd notifications) lives in four small standalone primitives. You almost never construct them yourself — CommandBase composes them for you. They exist so the classes above share one implementation, and so a plain script that is not a command can reuse the same building blocks. If you only ever extend CommandBase / ProcessQueue / DaemonOrchestrator, you can skip straight to those sections.

The layers¶

 Primitives  (plain PHP · no base class · optional · compose freely)
 ┌───────────────┬────────────────┬─────────────┬──────────────────┐
 │  WorkerLock   │ WorkerReloader │ SignalStop  │ SystemdNotifier  │
 │ lock+heartbeat│ code/settings  │ graceful    │ sd_notify        │
 │ +.stop+wedged │ reload         │ SIGTERM/INT │ (Type=notify)    │
 └───────▲───────┴───────▲────────┴──────▲──────┴────────▲─────────┘
         └───────────────┴── composed by ┴───────────────┘
                              │
                    ┌─────────┴──────────┐
                    │    CommandBase     │   base for all commands (lock-guarded)
                    └─────────┬──────────┘
                              │ extends
                    ┌─────────┴──────────┐
                    │    ProcessQueue    │   ready-made queue worker
                    └────────────────────┘

 DaemonOrchestrator (extends CommandBase)  ── spawns & supervises ──►  worker processes
                                              (a command, or a bespoke script)

Decision guide — "I want to…"¶

Goal	Use
A one-shot command that must not run twice at once	`CommandBase` → `beginJob()`/`endJob()`
Continuously process the framework queue	`ProcessQueue` subclass
A bespoke long-running loop as a console command	`CommandBase` → `installStopSignals()` + `while (!shouldStop())` + `heartbeat()`
A bespoke long-running loop as a standalone script (no Symfony Console)	the primitives directly (`WorkerLock` [+ `SignalStop`/`SystemdNotifier`/`WorkerReloader`])
Supervise one or more of the above as background daemons	`DaemonOrchestrator` subclass
Report worker/daemon health to a dashboard	`DaemonOrchestrator::status()` + `WorkerLock::readState()`

1. `CommandBase` — the base for commands¶

Extend it for any console command. Two usage modes:

1a. Simple lock-guarded command (one-shot)¶

The classic pattern — guarantees a single instance, cleans up on Ctrl+C:

use Pramnos\Console\CommandBase;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class RebuildSearchIndex extends CommandBase
{
    protected function configure(): void
    {
        $this->setName('search:reindex')->setDescription('Rebuild the search index');
    }

    protected function getJobName(): string { return 'SEARCH_REINDEX'; }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        if (!$this->beginJob($output)) {
            return self::FAILURE;           // already running → prints a message
        }
        try {
            // ... do the work ...
            return self::SUCCESS;
        } finally {
            $this->endJob();                // release + remove the lock
        }
    }
}

beginJob() also installs a SIGINT handler that cleans up and exits — right for a one-shot command. Under the orchestrator, SIGINT is ignored instead.

1b. Long-running worker command (a loop)¶

For a daemon-style loop, use the cooperative stop instead, so a SIGTERM/deploy never cuts a job in half:

protected function execute(InputInterface $input, OutputInterface $output): int
{
    if ($this->checkIfRunning()) {
        $output->writeln('<error>Already running.</error>');
        return self::FAILURE;
    }
    $this->startJob();                 // acquire the lock
    $this->installStopSignals();       // trap SIGTERM + SIGINT → cooperative flag
    $this->systemd()->ready();         // no-op unless under systemd Type=notify

    try {
        while (!$this->shouldStop()) {                 // signal, .stop sentinel or lock removed
            $did = $this->processOneBatch();

            // Records state + pings the systemd watchdog; false ⇒ our lock was
            // taken over by a replacement worker, so stop.
            if (!$this->heartbeat(['jobs_processed' => $did])) {
                break;
            }
            if ($did === 0) {
                sleep(1);              // idle back-off
            }
        }
    } finally {
        $this->endJob();               // release + remove the lock, notify STOPPING
    }
    return self::SUCCESS;
}

The two stop models¶

	`beginJob()` (simple)	`installStopSignals()` (worker)
Traps	SIGINT (ignored under the orchestrator)	SIGTERM, plus SIGINT when run interactively — a supervised worker ignores SIGINT
On signal	clean up + exit now	raise a flag; finish the job, then exit
Loop guard	—	`while (!$this->shouldStop())`
Best for	one-shot commands	long-running loops

Pick one per command; don't mix them.

A supervised worker (one started by DaemonOrchestrator, which exports PRAMNOS_JOB_LOCK_FILE) has no terminal, so a SIGINT reaching it was not addressed to it — an operator pressing Ctrl+C in a shell sharing the process group, a script signalling a group. It is set to SIG_IGN rather than left untrapped, because the default action for SIGINT is to terminate mid-task.

Method reference¶

Method	Description
`getJobName(): string`	Lock name (override; used for the lock file + `.stop` sentinel)
`getJobLockFilePath(): string`	Lock path (default `ROOT/var/<jobName>`)
`getLockStaleSeconds(): int`	Age past which a lock is treated as stale (default 2 h)
`beginJob(OutputInterface, bool $registerShutdown=true): bool`	Guard + acquire; `false` if already running
`endJob(): void`	Release + remove the lock; notify systemd `STOPPING`
`checkIfRunning(): bool`	Is another live instance holding the lock?
`startJob(): void`	Acquire the lock (no guard/handler; for the manual worker pattern)
`heartbeat(array $extra=[]): bool`	Refresh heartbeat + record `$extra` + watchdog; `false` if the lock was taken over
`installStopSignals(?callable $onStop=null): void`	Cooperative SIGTERM/SIGINT graceful stop (SIGINT ignored when supervised)
`shouldStop(): bool`	Loop guard: trapped signal, the `.stop` sentinel, or this worker's lock file removed
`signalStop(): SignalStop` / `systemd(): SystemdNotifier` / `workerLock(): WorkerLock`	The composed primitives, if you need them directly

1c. The schedule, and the two ways to run it¶

The framework has periodic work of its own — buffered writes to flush, queued rows to write into compressed chunks, finished queue items to clear. It declares that itself, in Pramnos\Scheduling\FrameworkSchedule, so an application does not have to know it exists. Either of these runs it:

With cron — one line, and it keeps working when a later framework version adds a task:

* * * * * cd /path/to/app && php pramnos schedule:run >> /dev/null 2>&1

Without cron — one long-running process, for containers, under systemd, or as an image's command:

php pramnos work                    # until told to stop
php pramnos work --once             # one pass, then exit
php pramnos work --interval=30      # check every 30 seconds
php pramnos work --max-runtime=3600 # exit hourly for a supervisor to restart

work holds a single-instance lock and stops cooperatively, so a SIGTERM during a task lets that task finish.

No minute is skipped. A pass runs its tasks one after another, so a slow one pushes the next pass along — passes at 03:19:59 and 03:21:09 would never look at 03:20. Each pass therefore asks about every minute since the last one it asked about, up to now, and runs a task due in any of them once; it then sleeps to the start of the next minute. It looks back at most Work::CATCH_UP_MINUTES (15), so a worker that was stopped for an hour does not replay the hour. A minute already asked about is never asked again, so --interval=30 does not run an every-minute task twice.

A restart loses no minute either. The last minute a pass asked about is kept beside the lock, in var/pramnos-work.lock.minute, and a new process starts after it — within the same 15-minute cap. A deploy that restarts work leaves a minute or two in which no process asks: the old one finishes its pass before the new one starts. A task that fell in that gap now runs when the new process starts. Only a process with no record behind it, the first start of an installation, looks at the current minute alone.

schedule:list shows what is registered, and whether a work process runs it

Listing reads definitions. Running needs a process. A schedule can list every framework job as due while nothing executes any of them — no work process and no schedule:run crontab line.

So schedule:list ends with a line saying whether a work process holds the lock, and says plainly that a schedule:run crontab line cannot be detected from there — so the absence of the message is not a claim that nothing is running.

A fork of the orchestrator is a silent way to lose the whole schedule: a copy that does not extend the framework's class never calls includeScheduler() or schedulerProcess(), so no work process is started. If you subclass it, extend Pramnos\Console\DaemonOrchestrator rather than CommandBase, or run work yourself.

Three things about that lock, because each of them is the sort of decision that looks like an oversight:

--once does not take it. It is the cron equivalent, and a crontab line that refused to run because a work process holds the lock would be a cron line that silently does nothing, once a minute, for ever. The cost is the honest one: run a crontab line and a work daemon on the same installation and a task without withoutOverlapping() can run twice. Pick one shape.
A lock taken over stops the worker. heartbeat() answers false once another process owns the lock — which is what a deploy looks like from inside, the replacement having started before this one was told to stop. Carrying on is how every job runs twice for the length of a rollout.
--max-runtime exits 0. A supervisor is told to restart on exit, and a planned recycle that reports a non-zero status becomes a crash in whatever reads the unit's history.

--interval=0 is clamped to one second rather than taken literally; the loop sleeps in one-second steps so a SIGTERM is noticed within a second instead of at the end of the interval.

With a daemon orchestrator — nothing to do. DaemonOrchestrator supervises work alongside the application's own daemons, so a project that has an orchestrator already runs the schedule. See §3.

The queue worker does not need a controller to exist¶

queue:process asks the application for a controller called Queueitems — the administration screen for queued jobs — because Application\Model will not be constructed without a Controller to hold. An application that has no such screen gets a plain Controller instead (QueueManager::controllerOrPlain()), because a background worker never renders anything with it, and failing at start-up would mean respawn, fail, repeat under a supervisor, with a queue that never drains. An application that has one gets it.

This is not the queue worker¶

queue:process runs background jobs — things an application dispatches and expects within seconds. work runs the schedule — things that happen on the clock. They want opposite things: a queue worker polls constantly to keep latency low, a scheduler sleeps a minute at a time. Run both:

php pramnos work &
php pramnos queue:process --daemon &

One broken job does not stop the pass¶

The argument for one process rather than a crontab line each, and the thing that makes it safe: a task that raises is reported, counted, and stepped over. On an installation without cron this process is the only thing running the background, so a pass that abandoned itself on the first exception would stop the buffered writes, the queue and the cleanup along with the broken job — and keep stopping them every minute, for ever, on the strength of one bug in one task.

Three outcomes, three marks, and only one of them is a failure:


`✓`	the task ran	timed, and logged with its duration
`↷`	`run()` answered false	the no-overlap lock is held by the previous minute's copy — the mechanism working, not a failure
`✗`	the task raised	counted, with the message, and written to the `schedule` channel

The middle row matters more than it looks: counted as a failure, a job that legitimately takes three minutes would report two failures for every success, and the count is what --once turns into an exit code for a cron line.

And a failure is logged, not only printed. work runs under systemd or as a container command, where nothing is attached to the output — a failure that exists only on a terminal nobody is watching is a failure nobody knows about.

How a `command` task is run¶

Scheduler::command('spool:drain') shells out, and the console it shells out to is the one this process is running:

/usr/local/bin/php /var/www/html/myapp.php spool:drain

The running entry point, not a fixed name — a scaffolded application's console is <cliName>.php in the project root, not the framework's pramnos. Define PRAMNOS_BIN to point somewhere else:

define('PRAMNOS_BIN', '/usr/local/bin/php /srv/app/console');

A non-zero exit fails the task. schedule:run prints ✗ Failed and returns non-zero; work counts it and carries on with the rest of the pass. Both write it to schedule.log.

And schedule.log carries the reason, not only the status. The last 2 KB of what the command printed — both streams — is appended to the logged message:

failed: channels:collect (0 * * * *) — Scheduled command 'channels:collect' exited with
status 1 (ran: '/usr/local/bin/php' '/var/www/html/myapp.php' channels:collect).
Output: PDOException: SQLSTATE[HY000] [2002] Connection refused

This matters more than it looks. Cron sends schedule:run output to /dev/null on a normal installation, so the framework's log is the last place the reason exists: anything it does not record is lost by design rather than by accident, and exited with status 1 on its own leaves nothing to do but guess. The tail rather than the whole output, because schedule.log is appended to on every tick; the tail rather than the head, because a chatty command's error is at the bottom.

A command's own output still reaches the console while it runs, so an interactive schedule:run shows it as usual — with stderr arriving on stdout, which is what makes the reason capturable at all.

Which puts a burden on every scheduled command: exit non-zero only for something a person should look at. spool:drain is the worked example, since it runs every minute:


nothing buffered	exit 0, and silent
rows written	exit 0
a row kept for the next run	exit 0 — the retry budget being spent is the spool working
a row parked as unwritable	exit 1 — data set aside, no further attempt, somebody must look

The kept case exits 0 because a retry is not something to act on: reported as a failure, it would record one every minute until the budget ran out, and three failures of one task read like three tasks failing. Keeping it at 0 is what makes the parked case a usable signal — it is the only one that fires.

The same argument makes the empty case silent rather than logging a line a minute. A log nobody reads is where the one line that mattered goes unnoticed; -v says it for somebody who is watching on purpose.

Overlap protection across containers¶

Scheduler::command('slow:job')->hourly()->withoutOverlapping();
Scheduler::command('very:slow')->daily()->withoutOverlapping('', 7200);  // wait up to 2h
Scheduler::command('shared:job')->hourly()->withoutOverlapping(ROOT . '/var');  // lock on a shared volume

The lock is a WorkerLock: it records the host beside the pid, and trusts the pid only when the host matches. A pid is a fact about the process table of whoever is asking, so where the host does not match the holder is judged by how recently it reported instead. The second argument is that threshold, for a task that legitimately runs longer than the default two minutes. Taking the lock is a single atomic create, so two schedulers a millisecond apart cannot both pass.

The lock file lives in sys_get_temp_dir() by default, which is per container. An application container and a daemon container that share only var/ do not see each other's lock unless the first argument points both at the shared directory, as in the third line above.

Replacing a framework task¶

// app/schedule.php
use Pramnos\Scheduling\FrameworkSchedule;
use Pramnos\Scheduling\Scheduler;

FrameworkSchedule::disable('spool:drain');       // or ::disableAll()
Scheduler::command('spool:drain')->everyFiveMinutes();

The application's file is loaded before the framework registers, which is what makes disable() usable at all. An application that disables these takes on running them itself.

Running a command task in process¶

Scheduler::command('reports:refresh')->everyMinute()->inProcess();

// app/schedule.php — the framework's own every-minute tasks
FrameworkSchedule::inProcess('spool:drain', 'mail:flush', 'llm:routines');

A command task starts a PHP process each time it is due: 1,440 a day for an every-minute task, each a fraction of a second and tens of megabytes. inProcess() runs the command through the console that work or schedule:run is itself, with the same exit status and output; a command that throws is a failed task, reported like a failed process.

It is opt-in, and the framework's own tasks start a process unless named. In process, the command shares the scheduler's process for as long as work runs: what it caches statically stays, memory it keeps is kept, and a fatal error or an exit() in it ends the scheduler with it. Fine for a short, well-behaved command; not for one that loads half the application. Where no console is known, outside work and schedule:run, the task starts a process as before.

Running a command task in the background¶

Scheduler::command('mail:bounces')->cron('* 1,4,8 * * *')->withoutOverlapping()->runInBackground();

A pass runs its tasks one after another, so a slow command holds every task behind it — and one due every minute of an hour that takes four minutes a run holds the worker for the whole hour, while the catch-up gives up on what it missed after fifteen minutes. A crontab ran each line in its own process, which is why moving a crontab onto the scheduler shows it.

runInBackground() starts the command detached (in its own session, through setsid where it exists) and the pass goes on at once. The detached process is schedule:background, which runs the command as the scheduler would have and does what the scheduler could not without waiting:

It holds the overlap lock until the command exits, so withoutOverlapping() still means one at a time. The scheduler skips a start while the lock is held, and records it skipped. Without withoutOverlapping() runs may overlap, as two crontab lines' would.
It finishes the run's row in the history — result, duration, exit status, the end of the output — when the command ends.
With onOneServer() the server lock decides which machine starts the command, and is let go once it has started.

Command tasks only: runInBackground() on a callable throws, since a closure belongs to the process that defined it.

The run history¶

Every run is a row in scheduleruns, whoever started it — work, schedule:run, a background process, a run asked for from the screen: the task, when it started, how long it took, ok / failed / skipped, a command's exit status and the last 4 KB of what it printed or the error it threw. A run in progress is running; one a killed worker left behind is marked interrupted by the task's next run, where the task holds an overlap lock that proves nothing else of it is alive.

Pramnos\Scheduling\RunLog reads and writes it: latestByTask(), pendingRequests(), since() (where the history begins), and request($task, $who) — a run an operator asks for, which the next pass of work or schedule:run takes, once, even with two of them on two servers. Runs older than schedule.history_days in app.php (14) are removed from that pass, at most hourly. Nothing is written until the migration has created the table, and a database error never stops a task: the history is an account of the schedule, not a condition of it.

CronExpression::nextRunAfter() and previousRunBefore() give a task's next run and the last one its expression asked for — a run missing since then is a late task — and describe() the expression in words ("At :05, :20, :35, :50 every hour").

The scheduler screen¶

/admin/Scheduler (menu: System → Scheduler) shows the schedule as schedule:list reads it, with the history beside it — the framework's own controller, so an application needs no wrapper for it:

Each task: its description, its expression in words (the expression itself on hover), its next run, its last run with when, how long and how it ended, and whether it runs without overlap, on one server, in process or in the background.
Late: the last minute the expression asked for, two minutes or more ago and after the history began, has no run of the task started since. A minute before the history began is not missed — nothing was recording then. Failed: the last run ended in an error. Running: its last run is in progress, or its overlap lock is held.
A note at the top when no work process holds its lock — the schedule is registered, not executing, unless cron runs schedule:run, which cannot be seen from the screen.
A task's page (Scheduler/show/<key>, opened in a dialog from the list) lists its runs, newest first, 25 to a page, each with its exit status, host and output.
Run now asks for one run: a POST with the session's token, written to the auth log with who asked. Nothing runs in the request — the next pass of work or schedule:run takes it, and the task shows requested until then.

The dashboard's Scheduler card counts the late, the failed and the running, and opens the screen. Both read Pramnos\Scheduling\ScheduleStatus, so they agree on what each word means.

Tasks an operator may adjust¶

A task the application marks ->adjustable() can be changed from its page on the screen, without a deploy: paused, given another frequency from a fixed list, kept to an hours window, and given a per-run limit.

// app/schedule.php
Scheduler::command('digest:send')->weekly()->at('11:15')
    ->adjustable('digest.send', 'Weekly digest', 'Sends', presets: ['daily', 'weekly'], limit: [50, 600]);
Scheduler::command('mail:flush')->everyMinute()->adjustable('mail.flush', 'Outgoing mail', 'Sends');

The id (digest.send) is the application's and must not change: the settings are kept under it, in scheduleoverrides (migration create_scheduleoverrides_table). It is not the task's key(), which follows the expression and would lose them when the code's schedule changes. The label names the task on the screen when it has no description; the group puts the adjustable tasks together under a heading, before the rest.
The frequencies are a list, never free cron: every 1, 5, 10, 15 or 30 minutes; every 1, 2, 3, 6 or 12 hours; daily at a time; weekly on a day at a time; or as in the code. presets names the ones a task offers (every:15, hours:2, daily, weekly); all of them when it names none.
The hours window is a start and an end in the application's time zone. One that ends before it starts crosses midnight (22:00–06:00). Outside it, and while paused, the task is not due: no run, and no row in scheduleruns for every minute it did not run.
The per-run limit is offered only where the task declares limit: [min, max], and is held to that range. A command then runs with --limit=N; a callable, or a job's handle(), gets N as its argument. With no limit set, the task runs as written.
Applied on the next pass. work and schedule:run read the table once a pass (ScheduleOverrides::apply()), so no restart is needed. Back to the schedule in the code deletes the row.
Not late. A paused task, or one whose window is closed, is never late, and neither is a run its expression asked for outside the window: none was meant to happen. The list says Paused since … or Outside hours · next 09:00, and the next run is the first one inside the window. ScheduleStatus decides this, so the dashboard card and anything built on it agree.
Audited. Every save is scheduler.adjusted in the audit log, every reset scheduler.reset, each with the settings before and after; the task's page says who changed it and when.

Who may. admin.scheduler opens the whole screen. scheduler.adjust (Scheduler::ADJUST_ABILITY) changes the adjustable tasks, and the screen opens for it too: an account that holds only that sees the adjustable tasks alone, with no Run now. Define it as any other ability (Gate::define('scheduler.adjust', …)); where the application defines no rule for it, admin.scheduler adjusts as well. The menu item stays admin.scheduler's, so an account that only adjusts needs a link of the application's own.

1d. The write spool — rows written out of the request path¶

Some rows are worth keeping and worth nothing individually: an access log, an audit trail, a hit counter. WriteSpool::append() puts one somewhere cheap and the scheduled spool:drain writes what has accumulated, batched, out of the request path.

\Pramnos\Database\WriteSpool::append('#PREFIX#tokenactions', [
    'tokenid' => 7, 'urlid' => 3, 'method' => 'POST', 'servertime' => time(),
]);

php pramnos spool:drain             # write everything buffered
php pramnos spool:drain --status    # how much is waiting, and where

Rows that can never be written¶

A spool is a promise to write a row later, and later can arrive after the row stopped being writable. The common case is a foreign key: tokenactions references usertokens, and a token cleaned up while its rows waited takes the key with it.

A row that fails is retried up to five times (spool_max_attempts), or twice when the error is a constraint violation such as a missing foreign-key parent — such a row will not become writable by being retried, and the second attempt exists only because a parent row may itself still be waiting in the spool. Then it is parked — appended to <table>.spool.failed with the error that stopped it, the attempts actually spent and a timestamp, removed from the spool, and never read back:

{"parked_at":"2026-08-20T04:11:07+03:00","attempts":2,
 "error":"insert or update … violates foreign key constraint","row":{"tokenid":3907,…}}

spool:drain --status reports the count; WriteSpool::parked() returns it. Parked rows are not pending — nothing will try them again — so read that file, decide, and delete it.

Tuning:

php pramnos spool:drain --max-attempts=10   # more patient
php pramnos spool:drain --max-attempts=0    # never park a transient failure; retry for ever

WriteSpool::setMaxAttempts(10);                    // in code
// or the `spool_max_attempts` application setting

A constraint violation is parked after at most two attempts whatever the setting says, 0 included; a lower setting (1) parks it sooner.

Identical failures are reported once with a count — two hundred lines a minute for one reason is how the reason stops being read.

2. `ProcessQueue` — the ready-made queue worker¶

Extend it and supply your queue model; you get the daemon loop, dashboard, heartbeat, graceful stop (SIGTERM/SIGINT/.stop), DB-reconnect and git-deploy handling for free.

use Pramnos\Console\Commands\ProcessQueue;
use Pramnos\Queue\Worker;

class MyQueue extends ProcessQueue
{
    protected function configure(): void
    {
        parent::configure();          // keeps every queue:process option
        $this->setName('my:queue');
    }

    // No getJobName() override: ProcessQueue names the lock QUEUE_PROCESSOR, or
    // QUEUE_PROCESSOR_<worker-id> with --worker-id, so several workers can run side by side.
    protected function getDashboardTitle(): string { return ' MY APP QUEUE '; }
    protected function getControllerName(): string { return 'Queueitems'; }

    protected function createWorker($controller, ?string $workerId): Worker
    {
        $w = new Worker($controller, $workerId);
        $w->registerTaskHandler('send_email', SendEmailTask::class);
        return $w;
    }
}

Run it:

php console my:queue --daemon            # loop forever
php console my:queue --daemon --runtime=55   # cron-friendly: run 55s then exit
php console my:queue --limit=100         # process up to 100 then exit
php console my:queue --force             # remove the lock, even a live holder's, and start

Common options: --daemon, --runtime=N, --sleep=N, --batch=N, --limit=N, --type=a,b, --worker-id=N, --force, --start-from=…, --reverse-order. It uses the primitives internally — you wire nothing.

3. `DaemonOrchestrator` — the supervisor¶

Extend it to declare which long-running processes should exist; it spawns them, restarts crashed ones, restarts on a stale heartbeat, and requests a graceful restart of all daemons when a new git commit is deployed.

use Pramnos\Console\DaemonOrchestrator;

final class MyDaemons extends DaemonOrchestrator
{
    protected function getDashboardTitle(): string { return ' MY DAEMONS '; }
    protected function getEntryPoint(): string     { return ROOT . '/console'; }

    protected function buildDesiredProcesses(): array
    {
        return [
            [
                'id'              => 'queue_1',
                'daemon'          => 'queue',
                'workerId'        => 'worker-1',
                'lockFile'        => ROOT . '/var/QUEUE_PROCESSOR_worker-1',
                'shellCommand'    => escapeshellarg(PHP_BINARY) . ' '
                    . escapeshellarg(ROOT . '/console') . ' my:queue --daemon --worker-id=worker-1',
                'requireLockFile' => true,   // health = a fresh heartbeat in the lock file
                'profile'         => 'email + notification queue',
            ],
        ];
    }
}

Three methods to write: buildDesiredProcesses(), getDashboardTitle() and getEntryPoint(). getJobName(), which names the orchestrator's lock under var/, is daemon_orchestrator unless a subclass says otherwise.

On SIGTERM or SIGINT it stops its daemons before it exits. Each is asked to stop after its current task — the stop sentinel a restart uses — and given STOP_GRACE_SECONDS (30); one still running then gets SIGTERM, which a CommandBase worker answers by finishing its task. So stopping the service, or Ctrl-C on daemons:start, leaves no worker behind and kills none mid-task.

Sizing the pool — `BurstPolicy`¶

The orchestrator decides which processes should exist; Console\BurstPolicy decides how many of one kind. It is deliberately pure — floor, ceiling, running count, backlog, thresholds and load in, a number out — with no queue, no database, no process table and no clock, because a scaling decision that can only be exercised by arranging a real overloaded system is one nobody dares change.

$policy = \Pramnos\Console\BurstPolicy::fromConfig([
    'floor'        => 1,      // always running
    'ceiling'      => 8,      // never more, whatever the backlog says
    'grow_above'   => 1000,   // pending items at which to add one
    'shrink_below' => 200,    // pending items at which to remove one
    'load_ceiling' => 0.75,   // load per core past which nothing is added
    'cooldown'     => 3,      // cycles to wait after a change
]);

$target = $policy->target($running, $backlog, $loadPerCore, $cyclesSinceChange);
$output->writeln($policy->explain($running, $backlog, $loadPerCore, $cyclesSinceChange));

Five rules, and each one is a failure somebody already had:

One step per cycle, never a jump to a computed target. That is the whole load protection: a policy that leaps has to be right about capacity in advance, an incremental one only about the direction. Ten seconds a step turns one worker into eight in eighty.
Two thresholds with a gap. One threshold oscillates by construction — scaling up drops the backlog under it, which scales down, which raises it again, every cycle for ever.
A cooldown. Without it the pool keeps stepping while the backlog still describes the world before the last step landed, so it overshoots and then corrects. Simulated: a load three workers can serve produced 2,3,4,5,6,7,8,8,7,6,5,4,3,2,2,2,2,3,4,4 — the ceiling, then the floor, at the reconcile interval.
Load gates growth, never shrinking. Past the ceiling nothing is added however deep the queue; but load alone never sheds a worker, because the load may not be the queue's at all and shedding would fail to fix that while making the backlog worse. Getting this backwards gives a pool that abandons its work whenever anything else on the machine is busy.
A backlog that cannot be read is no backlog — the pool holds where it is (kept within floor and ceiling), neither growing nor shrinking. Not "grow to be safe": the query most likely failed because the database is in trouble, and adding workers to it makes an outage worse; shrinking would drain the pool during a blip.

What "settled" can mean. A pool is a whole number of workers and the load it serves usually is not, so when the load needs three-point-something the pool must walk 3,4,3,2,3,4… for ever. The guarantee is a bound, not a fixed point: within one worker either side of what the load needs. That bound is what the cooldown buys — a span of two instead of six.

Log explain() beside the process count. A scaling controller needs its decision and the resulting number of processes on the same line, or a supervisor that undoes its own decisions is indistinguishable from one that never makes them — a correct policy and something else killing its workers look, from the thresholds alone, like a policy that does nothing.

And it is unsafe without the rest. Every shrink stops a worker, so queue:reclaim and the cooperative stop are prerequisites rather than companions: an autoscaler over a queue that abandons what a stopped worker held is a machine for losing work in proportion to how much it scales.

A pool that follows the backlog — `queuePool()`¶

BurstPolicy is the arithmetic; queuePool() is what puts it to work. Return it from buildDesiredProcesses() and the pool's size is decided again every cycle:

protected function buildDesiredProcesses(): array
{
    return array_merge(
        $this->queuePool('passes', [
            'types'        => ['pass_unit'],
            'floor'        => 1,
            'ceiling'      => 8,
            'grow_above'   => 50,
            'shrink_below' => 5,
            'cooldown'     => 3,
        ]),
        $this->queuePool('mail', ['types' => 'mail', 'floor' => 1, 'ceiling' => 2]),
    );
}

Key	Default
`types`	every type	a list or a comma string; becomes `queue:process --type`
`floor`, `ceiling`	1, 4
`grow_above`, `shrink_below`	1000, a fifth of it	pending tasks of the pool's types
`load_ceiling`	0.75	load per core past which nothing is added; `null` for no gate
`cooldown`	3	cycles to wait after a change
`args`	none	more `queue:process` arguments, e.g. `['--runtime', '3600']`

Each cycle it counts the pool's pending tasks (the (status, type) index makes that cheap), reads the load per core, and asks BurstPolicy::target() for one worker more, one fewer or the same. Worker n of a pool named passes is queue-passes-n, with the lock file var/QUEUE_PASSES_n.lock. A shrink drops the highest-numbered worker, which the orchestrator stops through its stop file.

A count that fails holds the pool. It does not read as "drained", which would shrink the pool during the database trouble that made the count fail.
A restarted supervisor starts from the workers it finds alive, not from the floor.
Every decision is kept in words. poolDecisions() returns explain() per pool, and a change of size is logged to the daemons log with it, so the decision and the process count are on one line.

What an operator decides: `DaemonControls`¶

The supervisor reads two tables every cycle, and the services screen writes them. They hold what a sentinel file cannot, a decision that has to last beyond the next cycle.

Table	Holds	The supervisor
`pramnos.worker_pools`	a pool defined on the screen, or the screen's changes to a `queuePool()` the code declares	applies the row's limits over the declaration; runs no workers for a pool whose `enabled` is false; supervises a screen-only pool beside the code's
`pramnos.stopped_services`	services an operator stopped	leaves them off the list, so they are stopped and not started again

Every limit in a pool row is nullable, and null means "as declared". Lowering one ceiling on the screen does not freeze the code pool's other limits at today's values. Deleting the row puts a code pool back to its declaration, and removes a screen-only pool.

$controls = new \Pramnos\Console\DaemonControls();

$controls->savePool('mail', ['types' => 'mail', 'floor' => 1, 'ceiling' => 2], 'alice');
$controls->setPoolEnabled('passes', false, 'alice');   // stop a pool
$controls->stopService('schedule', 'alice');           // keep a service stopped
$controls->startService('schedule');
$controls->deletePool('mail');

savePool() validates every value and throws InvalidArgumentException with a sentence that names the field. It checks that: - the name and the task types use a narrow alphabet, because worker ids and command arguments are built from them; - each limit is a whole number in its range; - the ceiling is not below the floor; - shrink below is less than grow above.

Without the tables nothing changes. An installation that has not run migrate, or whose database cannot answer, supervises exactly what its code declares.

Every cycle writes what it saw of each pool to var/daemon_orchestrator_pools.json (DaemonOrchestrator::poolsFilePath()). For each pool it records whether it comes from code or the screen, whether it is enabled, its size, backlog, load, and the BurstPolicy decision. The screen reads this file instead of guessing what the supervisor will do.

Worker ids are compared, not searched for¶

The orchestrator recognises its own processes by the --worker-id it passed, reading /proc/<pid>/cmdline (or ps aux where there is no /proc). It compares the argument's value. That matters more than it sounds, because the obvious implementation is a substring search and a substring search matches a prefix:

--worker-id worker-1   found inside   --worker-id worker-10

A substring search would cap every pool at nine workers per profile, silently, whatever the ceiling said: deduplicateRunningProcesses() — whose whole job is killing genuine double-spawns — would kill workers 10 and above as duplicates of worker 1.

Nine is not a special number; it is the boundary only because ids are decimal. Any id that is a prefix of another has the same hole, and no separator convention closes it while the test is a substring search. cmdline is NUL-separated, so the arguments arrive already delimited — comparing them as values removes the question rather than answering it. Both --worker-id x and --worker-id=x are understood.

Worth knowing because of where the symptom points. A pool held below its ceiling by something killing its workers looks like a scaling policy that is not working, and checking the thresholds, the load gate and the stored settings will each confirm the policy is right.

If you build a scaling policy on this, log the decision and the resulting process count on the same line. A supervisor that undoes its own decisions is otherwise indistinguishable from one that never makes them.

The schedule comes with it¶

The list above is the application's daemons. The orchestrator supervises one more without being asked — work, the framework's schedule worker (§1c) — under the id schedule, with the standard lock file at var/pramnos-work.lock:

[started] stats pid=118
[started] realtime pid=30
[started] schedule pid=141      ← not in buildDesiredProcesses()

The framework declares periodic work of its own and a schedule only happens when something runs it. A stack with an orchestrator, no crontab and no work process would run none of it: spool:drain would never fire, rows written through the WriteSpool would stay in a file, and every report reading the drained table would show "no data" — a symptom several layers away from its cause. An application answers for its own daemons; the framework's are the framework's business.

Two overrides:

// Already have a crontab line for schedule:run? Running both is safe (every
// framework task takes an overlap lock) but pointless.
protected function includeScheduler(): bool { return false; }

// Or supervise it differently — a shorter interval, a different lock file.
protected function schedulerProcess(): array
{
    return [
        'id' => 'schedule', 'daemon' => 'schedule', 'workerId' => 'schedule-1',
        'lockFile' => ROOT . '/var/pramnos-work.lock',
        'tokens'   => ['work', '--interval=30'],
    ];
}

An application that already declares work in buildDesiredProcesses() keeps its own entry — recognised by what it runs, not by the id it was given. An application whose console does not register work at all gets nothing added, rather than a supervised entry that could only ever fail to start.

requireLockFile => true — health is a fresh heartbeat (mtime) in the worker's lock file (a CommandBase/ProcessQueue worker keeps this automatically via heartbeat()).
requireLockFile => false — health is process liveness only, and the status line says (pid alive) rather than (lock active) because that is all that was checked. It also switches off the stale-heartbeat restart, which is the one restart a pid check cannot make: a wedged daemon satisfies "pid alive" indefinitely. Prefer true — a JSON lock is read fine, readWorkerPidFromLockFile() delegates to WorkerLock::pidFromFile().
Stopping a daemon = the orchestrator drops a <lockFile>.stop sentinel; the worker's shouldStop() (or a standalone script's WorkerLock::stopRequested()) sees it and exits after the current job. Constants: HEARTBEAT_STALE_SECONDS (300), GIT_CHECK_SECONDS (60).

A new commit or a migration restarts the daemons¶

Every GIT_CHECK_SECONDS the orchestrator compares two things with what it last saw: the checkout's git hash, and the schema generation (MigrationRunner::schemaGeneration(), kept in var/schema-generation). When either has moved, it asks every daemon to stop after its current job and starts them again.

On a new commit it also reloads itself, with pcntl_exec and the same arguments. That keeps its PID and the start time ps and systemd show, so "running since last week" does not mean the old code. Its log says so: [git] re-executing the orchestrator to load the new code… followed by a fresh Starting daemon orchestrator. Where pcntl is not available the log says cannot self-exec, and only a restart of the service loads new orchestrator code.

The second is for a migration that changes no commit: one run by a request after the deploy had already restarted everything, or one run on its own. A worker started before it would otherwise keep the old view of the schema — the tables hasTable() found, the column lists of its models, its prepared statements. The generation moves whenever a batch applies or rolls back anything, in whatever process ran it, and not when migrate finds nothing pending.

A worker not run by the orchestrator notices too, less thoroughly: CommandBase::shouldStop(), which every loop asks between tasks, reads the generation and forgets which tables and columns it found when it has moved.

A supervised worker watches the supervisor's lock path, and stops painting¶

Two things follow from being spawned by DaemonOrchestrator, and both are read from PRAMNOS_JOB_LOCK_FILE, which it exports:

The lock path is the supervisor's, through resolvedJobLockFilePath(). A worker that rebuilds the path itself can disagree, and disagreement here is invisible: a sentinel read where nothing writes is indistinguishable from no sentinel. A worker whose "is my lock still there" check looks at a different path from the one its supervisor named would see no lock, conclude it had been told to stop, and leave on its first pass. Always go through resolvedJobLockFilePath(), never getJobLockFilePath() directly.
No dashboard. isSupervised() is true, and the interactive dashboard is skipped: it clears the screen and repaints in place, which in a log file is thousands of escape sequences and no readable history. That log is the only account of what the worker did; the live numbers belong on /admin/Services, which reads the state file.

queue:process paints its dashboard only on a terminal, so the same holds when it is run by hand into a file or a pipe (>> var/logs/worker.log): each task is one plain line — [2026-01-02 03:04:05] #7 chat_ask completed 1.250s — answered — and a pass with nothing to do writes nothing.

A stop request has a deadline¶

Every stop — a process leaving the desired set, a redeploy, the orchestrator being disabled — records when it was asked for. A worker still running STOP_GRACE_SECONDS (30) later is sent SIGTERM and reported as its own event:

[waiting]      realtime pid=30 — gracefully stopping, will restart when done (24s before SIGTERM)
[stop-timeout] realtime pid=30 — ignored the stop sentinel for 31s, sent SIGTERM

A daemon that polls its sentinel exits long before the deadline and never sees it. One that cannot is the case the deadline exists for — and [stop-timeout] is a fact about that worker, worth finding in a log.

The sentinel applies whatever requireLockFile says: it is the orchestrator's instruction to that process, not a claim about its lock.

The time of each stop request is kept in the state file, so the deadline survives the orchestrator re-executing itself after a redeploy — otherwise a daemon that ignores its sentinel could keep serving from the old code across any number of deploys.

Health snapshot for a dashboard¶

$status = (new MyDaemons())->status();   // does NOT run a reconcile cycle
// [ 'running' => bool, 'pid' => int, 'heartbeat_age_seconds' => int, 'daemons' => [...] ]

heartbeat_age_seconds is the age of the state file, which the reconcile loop rewrites every cycle — so a fresh mtime means the supervisor is not merely alive but actively cycling. A live pid with a stale heartbeat is the third state, and it looks identical to healthy if you only read the pid: the process is there and stuck.

For code with no instance to call — a web request cannot construct the application's orchestrator subclass — the two paths are static:

DaemonOrchestrator::stateFilePath();          // ROOT/var/daemon_orchestrator_state.json
DaemonOrchestrator::orchestratorLockPath();   // ROOT/var/DAEMON_ORCHESTRATOR.lock

The services screen¶

/admin/Services (usertype 80 and up) is where an operator watches and controls what the orchestrator supervises. It is the state of the workers and nothing to configure, in reading order:

Part	Shows	Controls
The supervisor	running with its pid and last cycle, stale, or not running	—
Services	every supervised process, with its pid, status and log	Stop, Start, Restart, Logs; Restart all above the list
Worker pools	each pool, from `code` or the `screen`: its task types, workers, backlog, load, and the `BurstPolicy` decision; waiting for supervisor while a saved change has not been applied	Stop / Start, and Configure pools
Batches	batches queued in the last 24 hours, with pending, processing, done and failed	—

The numbers refresh from GET /admin/Services/status every ten seconds without reloading the page.

A pool's limits are set on /admin/Services/pools, linked from the services screen and from Settings. Each pool is a card:

Task types are boxes to tick, from every type the framework can name: those with a handler (the framework's own, queue.handlers in app.php, the queue's task classes), those already in the queue table, and those a pool names. A type not in the list goes in Other task types, comma-separated. Nothing ticked is every type — for a code pool, the types its declaration names.
Minimum and maximum workers, grow above and shrink below a number of waiting tasks, a CPU limit in per cent of the server's capacity, and a cooldown in supervisor cycles — each with a line under it saying what it does. A code pool shows its declared values as placeholders, and a field left blank keeps them.
Save, Stop / Start, and Reset to code (a code pool) or Remove (a pool added here), which asks first.

The last card adds a pool.

What each control does:

Stop records the service in pramnos.stopped_services and writes its stop file. The worker finishes its task and exits, and the supervisor leaves it off its list until Start removes the record.
Restart writes the stop file of a running service. The worker exits, and the supervisor, which still wants it, starts it again.
Restart all does that for every running service. Use it after a deploy the supervisor cannot see: it restarts by itself only when git HEAD changes, so composer update or a copied build is invisible to it.
A pool's Stop, Start, Save and Remove write pramnos.worker_pools. Stop and Start return to the screen they were pressed on; Save and Remove to the pools page.

Nothing on the screen spawns or kills a process itself. Every control is written where the supervisor reads it, and the supervisor acts on its next cycle. With no orchestrator running, only a stop file takes effect, because a daemon checks its own. Every change is a POST carrying the session's token and is recorded in the auth log with who made it.

Customised views: a project that published the services view with project:publish-views keeps its old copy of the services screen; the pools page comes from the framework until it is published too. To get both as above, run project:publish-views --group=services --force.

A pid is not enough to tell whether it is running. The supervisor's normal home is a container of its own — that is what pramnos init writes for an application with background work — so the pid in its lock file belongs to that namespace, and a web request reading it is in another. There the number matches something unrelated, or nothing at all: a working supervisor reported dead, or a dead one reported alive because pid 14 happens to be Apache.

So the reading is the pid or a heartbeat younger than 120 seconds. The state file is on the shared volume and is rewritten every reconcile cycle, so a recent mtime means actively cycling, and it cannot lie in the dangerous direction — a stopped supervisor stops touching it. A live pid with a stale heartbeat still counts as running, deliberately: that is the stuck-supervisor state, and the screen's warning for it is more useful than silence.

The screen therefore reports the supervisor's own state above the list — running with its pid and last cycle, stale if it has not cycled for two minutes, and a warning naming the consequence if it is not running at all. GET /admin/Services/status carries the same reading as orchestrator, which is what a monitor should look at first: with the supervisor gone, "0 running" is the expected number rather than an incident. The same response carries pools, batches and stopped_by_operator.

4. Standalone script worker (no `CommandBase`)¶

When a worker is a plain script (e.g. worker.php, not a Symfony Console command), reach for the primitives directly — the exact same ones CommandBase composes:

use Pramnos\Console\WorkerLock;
use Pramnos\Console\SignalStop;
use Pramnos\Console\SystemdNotifier;
use Pramnos\Console\WorkerReloader;

$lock    = new WorkerLock('worker', WorkerLock::defaultPath('worker'));
$systemd = new SystemdNotifier();
$reload  = new WorkerReloader(ROOT, ['src', 'worker.php', 'composer.lock'],
    fn () => MySettings::versionStamp());

if (!$lock->acquire($takenOverFrom)) {
    exit("another worker holds the lock\n");   // still running / wedged
}
$reload->baseline();

$running = true;
$signals = new SignalStop([], function () use (&$running): void { $running = false; });
$signals->install();
$systemd->ready();

try {
    while ($running) {
        $processed = process_one_batch();

        if (!$lock->heartbeat(['jobs_processed' => $processed])) {
            break;                          // lock taken over → stop
        }
        $systemd->watchdog();
        if ($lock->stopRequested()) break;  // supervisor's <path>.stop sentinel

        if ($reload->settingsChanged()) { rebuild_from_settings(); }
        if ($reload->codeChanged()) {
            // Under a supervisor, exit and let it restart on the new code; otherwise
            // respawn yourself (see WorkerReloader::isSupervised()).
            break;
        }
        if ($processed === 0) sleep(1);
    }
} finally {
    $systemd->stopping();
    $lock->release();
}

This is why the primitives are standalone: a script gets the same locking, heartbeat, graceful stop, systemd watchdog and reload guarantees as a CommandBase worker, without a base class.

5. The four primitives (reference)¶

`WorkerLock` — single-instance lock + heartbeat¶

A JSON file whose atomic creation is the mutex (works where advisory flock() doesn't, e.g. Docker bind mounts on macOS), doubling as the heartbeat.

Method	Description
`acquire(?string &$takenOverFrom=null): bool`	Win the lock; take over a dead/wedged holder; refuse a live+progressing one
`heartbeat(array $extra=[]): bool`	Refresh (+ record state); `false` once taken over
`release(string $status='stopped'): void`	Mark stopped, keep the file for status reads
`stopRequested(): bool`	Is there a `<path>.stop` sentinel?
`isHeldByAnother(): bool` / `holderIsWedged(): bool`	Foreign-holder liveness / alive-but-stuck
`readState(): ?array` / `heartbeatAge(?array): ?int`	State inspection (for `status`)
`static pidFromFile(string): int` / `static defaultPath(string $name, ?string $dir=null): string`	Legacy-aware pid read / `<dir>/<name>.lock`

`SignalStop` — cooperative graceful stop¶

$stop = new SignalStop();                          // SIGTERM + SIGINT
$stop->install();
while (!$stop->requested()) { /* one job */ }
// or break a blocking loop: new SignalStop([], fn () => $server->stop())

`SystemdNotifier` — sd_notify¶

$sd = new SystemdNotifier();   // reads NOTIFY_SOCKET; no-op off Type=notify
$sd->ready(); $sd->watchdog(); $sd->stopping(); $sd->status('processing');

`WorkerReloader` — code / settings reload¶

$r = new WorkerReloader(ROOT, ['src', 'worker.php', 'composer.lock'], fn () => $stamp);
$r->baseline();
if ($r->settingsChanged()) { /* rebuild snapshot objects in place */ }
if ($r->codeChanged())     { /* exit for a restart on the new code */ }
WorkerReloader::isSupervised();   // is something going to restart me if I exit?

git-HEAD vs fingerprint. The orchestrator's git-HEAD detection (§3) and WorkerReloader::codeChanged() both mean "restart on new code", but at different layers: the orchestrator (supervisor, git-deploy-granular, production) vs the worker itself (file mtime, git-less / dev --watch-files / self-respawn). They are complementary — the worker's own reload is normally off when a git+orchestrator setup owns restarts.

6. Deployment recipes¶

systemd, watchdog (recommended for a command or script worker):

[Service]
WorkingDirectory=/srv/app
ExecStart=/usr/bin/php /srv/app/console my:queue --daemon    # or /srv/app/worker.php run
Restart=always
RestartSec=5
Type=notify          # the worker's ready()/watchdog() drive this
WatchdogSec=120      # a missed watchdog ping → systemd restarts it

systemd, simplest: drop Type=notify/WatchdogSec; Restart=always still restarts a crash, and the single-instance lock prevents overlap.

cron (no supervisor): --daemon --runtime=55 (ProcessQueue) or work --max-runtime=55 polls for 55 s of each minute:

* * * * * cd /srv/app && php console my:queue --daemon --runtime=55 >> var/logs/queue.log 2>&1

Under DaemonOrchestrator: run the orchestrator itself as the single systemd service; it spawns and supervises the workers from buildDesiredProcesses().

Creating the orchestrator service — Ubuntu / Debian¶

This is the whole thing, on a stock Ubuntu or Debian box. /admin/Services links here when it finds no supervisor running.

1. Check the command exists. An application declares its own orchestrator — a DaemonOrchestrator subclass, conventionally named daemons:start:

cd /srv/app && php console list | grep daemons

If nothing comes back, there is no supervisor to run yet: write the subclass first (§3). A scaffolded application gets one in src/ConsoleCommands/Daemons.php when it enables a feature with background work — queue, messaging, broadcasting, auth or authserver.

2. The unit file — /etc/systemd/system/app-daemons.service:

[Unit]
Description=Application daemon orchestrator
# Ordering only. `After` does not wait for the database to be *ready*, which is the
# failure this file is written around — see the restart policy below.
After=network-online.target postgresql.service mysql.service
Wants=network-online.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/srv/app
ExecStart=/usr/bin/php /srv/app/console daemons:start
# `always`, not `on-failure`. This process's worst failure is a *clean* exit: a machine
# that comes back before the database accepts connections boots the framework into its
# maintenance page and returns 0. `on-failure` looks at that and correctly does nothing,
# leaving the supervisor gone with every other service up and healthy beside it.
Restart=always
RestartSec=5
# SIGTERM to the orchestrator alone: it asks every daemon to stop after its current task, waits
# up to STOP_GRACE_SECONDS (30) for them, and only then exits. systemd SIGKILLs what is left
# after TimeoutStopSec, so keep that above the grace period.
KillMode=mixed
TimeoutStopSec=60
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

3. Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now app-daemons
systemctl status app-daemons
sudo journalctl -u app-daemons -f      # what it is doing, live

4. Confirm from the application, not from the pid — /admin/Services reads the orchestrator's own state file, so it can tell a live process from one that has stopped cycling. DaemonOrchestrator::status() is the same reading for a monitor.

Two things that go wrong on a fresh box:

var/ is not writable. The locks, heartbeats and state file live in ROOT/var, and under User=www-data a directory owned by the deploying user is a supervisor that starts and immediately cannot record anything: sudo chown -R www-data:www-data /srv/app/var.
A crontab is already running the schedule. The orchestrator supervises the schedule worker itself, so both together run every scheduled job twice. Remove the schedule:run crontab line, or override includeScheduler() to false.

In Docker, the same process is a service beside the app, sharing its image and volume so it runs the code the site is running:

  daemons:
    build: { context: . }
    restart: unless-stopped          # for the reason given above
    user: www-data                   # the image maps it to the host user
    command: php /var/www/html/console.php daemons:start
    volumes:
      - .:/var/www/html
    depends_on:
      - db

pramnos init writes exactly that for an application whose features have background work — the queue, messaging, broadcasting, or the periodic jobs auth and authserver schedule.

user: www-data is the Docker form of the var/ problem above. Without it the supervisor runs as root, is usually the first process to touch var/ on a fresh clone, and leaves it root-owned — so ./dockertest stops at Unable to open lock file and the application cannot write its logs. On a project that already hit it: add the line, then docker compose exec -u root app chown -R www-data:www-data var.

7. More than one server¶

Most of the framework is already shared: the queue claims jobs with FOR UPDATE SKIP LOCKED, so two workers on two machines never take the same one; the WebSocket server has a Redis backplane in Broadcasting\Cluster; ClientIpResolver knows about proxies. What is not shared by default is everything that lives on local disk, and each of those fails quietly rather than loudly.

	Default	On two servers	What to do
Cache	file	invalidation is per node — the node that was not asked still serves the old page	`redis` / `memcached`. `health:check` reports `cache` as degraded when it fell back
PHP sessions	`files`	a visitor whose next request lands elsewhere is signed out at random	`APP_SESSION_HANDLER=redis`, or sticky sessions. `health:check` reports `session_storage`
Scheduled tasks	run on every node	a nightly email goes out once per machine	`->onOneServer()`, below
`migrate`	`WorkerLock`, which is a file	two deploys can migrate at once	run it from one node as a deploy step
Uploads	`www/uploads/`	a picture uploaded on A does not exist on B	a shared mount, or `Pramnos\Storage` with an `s3` disk — see below
`APP_URL`	inferred from the request	each node infers its own, cron infers nothing	set it. `health:check` reports `site_url`

`withoutOverlapping()` is per machine. `onOneServer()` is not¶

Scheduler::command('reports:nightly')->daily()->at('02:00')->onOneServer();
Scheduler::call(fn() => …)->hourly()->onOneServer(name: 'metrics:rollup');

withoutOverlapping() takes a WorkerLock, which is a file — and it says so: it records the holder's host and checks the holder's pid on the same host. On one machine that is right and free. On two, each has its own var/ and its own sys_get_temp_dir(), so each takes its own lock and the task runs once per node. Nothing fails; the work happens twice, which is why it can run for months unnoticed.

onOneServer() takes a Pramnos\Database\SharedLock instead — a row in pramnos.locks whose primary key is the exclusion. The database is the one shared thing every installation already has; a shared cache is the usual answer to this and is optional here, and the default cache adapter is local files, which would be a lock that excludes nothing in exactly the same way.

The two compose. withoutOverlapping() stops a slow run on this machine starting again; onOneServer() keeps the other machines out. A task that wants both says both.

The lease has to outlast the work. A holder that dies never releases, so the lock expires and the next node to ask takes it — onOneServer($ttl) defaults to an hour. A task that can run longer than its lease will overlap, which is the trade every distributed lock makes and the reason the number is an argument rather than a constant.

A closure has to be named. describeHandler() answers the constant Closure for every closure, so two unrelated closure tasks would share one lock and take turns not running. onOneServer() raises rather than letting that happen — pass name:.

Uploads on shared storage¶

A shared mount — NFS, EFS — solves this with no code at all, and for many installations that is the right answer: the web server keeps serving www/uploads/ directly, nothing is copied, and there is nothing new to configure.

What a mount cannot do is put files on a CDN's origin, survive a node being replaced, or cost nothing when nobody is reading them. For that there is Pramnos\Storage, which ships local, s3 and ftp drivers behind one interface:

// app/config/app.php
'storage' => [
    'default' => 'media',
    'disks'   => [
        'media' => [
            'driver'   => 's3',
            'bucket'   => envvar('APP_S3_BUCKET'),
            'region'   => envvar('APP_S3_REGION'),
            'key'      => envvar('APP_S3_KEY'),
            'secret'   => envvar('APP_S3_SECRET'),
            // Anything that is not AWS names its own endpoint — MinIO, R2, Spaces:
            'endpoint' => envvar('APP_S3_ENDPOINT', ''),
            'url'      => 'https://cdn.example.com',
        ],
    ],
],

Storage::put('uploads/2026/09/logo.png', $bytes);
Storage::url('uploads/2026/09/logo.png');
Storage::disk('media')->exists($path);

It is optional in the sense that matters: an installation that configures nothing gets a local disk rooted at www/ — where uploads already are — so the façade is usable without anything being imposed and without any file moving. No Storage::init() call is needed during bootstrap.

The s3 driver needs aws/aws-sdk-php, which is not a dependency of this framework: composer require aws/aws-sdk-php in the application that wants it. Without it the driver says so on construction rather than failing with a class-not-found from three frames down.

The media system publishes to a disk called `media`¶

Name a disk media and the media system uses it. Name it anything else, or none at all, and nothing changes:

'storage' => [
    'disks' => [
        'media' => ['driver' => 's3', 'bucket' => …, 'url' => 'https://cdn.example.com'],
    ],
],

A disk name rather than a new setting, because a name is discoverable — somebody reading app.php sees what it is for — and one concept is cheaper than two.

With no media disk every path is a no-op. The file stays in www/uploads/, served by the web server exactly as before, and nothing is copied. That is deliberate and not merely conservative: on one server, or on several sharing a mount, local disk is the right answer and a publish step would be a copy from a directory to itself.

The work stays local; only the result is published¶

GD writes with imagejpeg($image, $path) — a real filesystem path a bucket does not have. So the pipeline is unchanged: upload, resize and thumbnail locally, then publish what came out. The local copies stay, because they are the origin a later resize reads and, on a single server, also what is served; deleting them would make rendering a page depend on the disk being reachable.

$media->url is already the key. It is the path relative to www/ — uploads/2026/09/x.png — which is what the media system stores with or without a disk, so nothing about the database changes.

Ask for the address rather than building it¶

<img src="<?php echo htmlspecialchars($media->publicUrl('thumb')); ?>">

publicUrl() is the disk's address when there is one and the site's own when there is not. A view concatenating sURL with the stored path keeps working — and keeps pointing at the server the file may no longer be on.

Backfilling an existing library¶

republishToStorage() puts anything that is not on the disk yet and skips what is, so it is cheap to run repeatedly over a whole library — which is what you want after configuring a disk for a site that already has ten thousand pictures, and what you want again after the disk was unreachable for an hour.

Publishing is best effort: a disk that is down does not fail an upload the application has already accepted and saved. The row is the record; the disk is a copy of it.

Sessions in a shared store, without touching `php.ini`¶

// app/config/app.php
'session' => ['handler' => 'redis'],

or APP_SESSION_HANDLER=redis in .env, which wins over it.

The handler is settable at run time as long as it is set before session_start(), and that is the window Session::start() already runs in — next to use_strict_mode and for the same reason. So this needs no php.ini a deployment may not control.

With no path, the cache's host is reused: an application that has configured Redis for its cache has already said where Redis is, and a second copy of a hostname is a second thing to get wrong — silently, because a wrong session host signs people out rather than erroring. Give 'path' explicitly when the two are different servers.

It never fails the request. A handler PHP has not registered, a store that is down, a misspelling — all leave sessions on files, which is a working single server rather than a site that will not boot, and health:check says session_storage is degraded. Which store PHP has is php -i | grep "save handlers".