Skip to content

Commit 8805bb1

Browse files
committed
Maintenance mode
1 parent 8a4b7c9 commit 8805bb1

42 files changed

Lines changed: 3147 additions & 29 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,38 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
77

88
## [Unreleased](https://github.com/orisai/scheduler/compare/2.2.2...v2.x)
99

10+
### Added
11+
12+
- Maintenance mode - stop running jobs during deployments
13+
- `MaintenanceChecker` interface - implement to define when maintenance is active
14+
- `MaintenanceManager` - orchestrates maintenance checking, run tracking, status reporting and shutdown requests
15+
- `MaintenanceManager->requestShutdown()` - programmatic shutdown trigger (used by signal handlers)
16+
- `RunRegistry` interface with `FileRunRegistry` (single-server) and `LockPoolRunRegistry` (multi-server)
17+
implementations - tracks active `scheduler:run` processes
18+
- `MaintenanceStatus` - value object returned by `MaintenanceManager->getStatus()`
19+
- `ShutdownCheck` - value object carrying shutdown check and grace period for executors
20+
- `StatusCommand` (`scheduler:status`) - reports maintenance state and active runs
21+
- `ManagedScheduler`
22+
- accepts optional `MaintenanceManager` - enables maintenance mode with two-phase shutdown
23+
(graceful wait, then force-kill after configurable grace period)
24+
- `getMaintenanceManager()`
25+
- `RunCommand`
26+
- accepts optional `MaintenanceManager` - registers signal handlers for graceful shutdown
27+
- handles `SIGTERM` and `SIGINT` signals (requires `pcntl` extension, double-signal forces exit)
28+
- returns exit code `2` when run was stopped due to maintenance
29+
- `WorkerCommand`
30+
- handles `SIGTERM` and `SIGINT` signals for graceful stop (requires `pcntl` extension, double-signal forces exit)
31+
- `JobResultState::maintenance()` - for jobs skipped or terminated due to maintenance
32+
- `RunSummary->isMaintenanceActive()` - indicates whether run was affected by maintenance
33+
34+
### Changed
35+
36+
- `JobExecutor` (BC break)
37+
- `runJobs()` accepts optional `?ShutdownCheck $shutdownCheck` parameter
38+
- added `supportsJobShutdown(): bool` method
39+
- `RunSummary`
40+
- constructor accepts optional `bool $maintenanceActive` parameter
41+
1042
## [2.2.2](https://github.com/orisai/scheduler/compare/2.2.1...2.2.2) - 2026-02-12
1143

1244
### Fixed

‎docs/README.md‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,13 @@ Cron job scheduler - with locks, parallelism and more
3333
- [List command - show all jobs](#list-command)
3434
- [Worker command - run jobs periodically](#worker-command)
3535
- [Explain command - explain cron expression syntax](#explain-command)
36+
- [Maintenance mode](#maintenance-mode)
37+
- [Setup](#maintenance-setup)
38+
- [Run tracking](#run-tracking)
39+
- [Status command](#status-command)
40+
- [Signal handling](#signal-handling)
41+
- [Deploy integration](#deploy-integration)
42+
- [Exit codes](#exit-codes)
3643
- [Lazy loading](#lazy-loading)
3744
- [Integrations and extensions](#integrations-and-extensions)
3845
- [Troubleshooting guide](#troubleshooting-guide)
@@ -812,6 +819,160 @@ Options:
812819
- `--timezone=<timezone>` (or `-tz`) - the timezone time should be displayed in
813820
- `--locale=<locale>` (or `-l`) - explain in specified locale
814821

822+
## Maintenance mode
823+
824+
During deployments, you typically want to stop running cron jobs to prevent interference. The scheduler supports
825+
a two-phase shutdown: first it waits for running jobs to finish naturally (graceful), then force-kills any remaining
826+
processes after a configurable grace period.
827+
828+
Maintenance mode requires `ProcessJobExecutor`. `BasicJobExecutor` cannot terminate running jobs and will throw
829+
a `LogicException` if used with `MaintenanceManager`.
830+
831+
### Maintenance setup
832+
833+
Create a `MaintenanceChecker` implementation for your environment. The interface has a single method:
834+
835+
```php
836+
use Orisai\Scheduler\Maintenance\MaintenanceChecker;
837+
838+
class AppMaintenanceChecker implements MaintenanceChecker
839+
{
840+
841+
public function isMaintenance(): bool
842+
{
843+
// Check your app's maintenance flag (file, Redis, database, etc.)
844+
return file_exists(__DIR__ . '/../../maintenance.running');
845+
}
846+
847+
}
848+
```
849+
850+
Set up the scheduler with `MaintenanceManager`. Pass the manager to both the scheduler and the run command:
851+
852+
```php
853+
use Orisai\Scheduler\Command\RunCommand;
854+
use Orisai\Scheduler\Executor\ProcessJobExecutor;
855+
use Orisai\Scheduler\Maintenance\FileRunRegistry;
856+
use Orisai\Scheduler\Maintenance\MaintenanceManager;
857+
use Orisai\Scheduler\SimpleScheduler;
858+
859+
$checker = new AppMaintenanceChecker();
860+
$registry = new FileRunRegistry(__DIR__ . '/var/scheduler-runs');
861+
$manager = new MaintenanceManager($checker, $registry);
862+
863+
$executor = new ProcessJobExecutor();
864+
865+
$scheduler = new SimpleScheduler(
866+
null, // errorHandler
867+
null, // lockFactory
868+
$executor,
869+
null, // clock
870+
null, // logger
871+
$manager,
872+
);
873+
874+
$runCommand = new RunCommand($scheduler, null, $manager);
875+
```
876+
877+
The grace period (time before force-kill) defaults to 30 seconds. This matches the Kubernetes
878+
`terminationGracePeriodSeconds` default - long enough for most jobs to finish DB transactions, API calls
879+
and file operations, short enough to not block deploys excessively.
880+
881+
```php
882+
$manager = new MaintenanceManager($checker, $registry, 60); // 60 second grace period
883+
```
884+
885+
### Run tracking
886+
887+
`RunRegistry` tracks active `scheduler:run` processes so deploy scripts can check if it's safe to proceed.
888+
Two implementations are provided:
889+
890+
**FileRunRegistry** (default, single-server):
891+
892+
```php
893+
use Orisai\Scheduler\Maintenance\FileRunRegistry;
894+
895+
$registry = new FileRunRegistry(__DIR__ . '/var/scheduler-runs');
896+
```
897+
898+
Writes a file per active run. Automatically cleans up stale entries older than 1 hour from crashed processes.
899+
900+
**LockPoolRunRegistry** (multi-server):
901+
902+
```php
903+
use Orisai\Scheduler\Maintenance\LockPoolRunRegistry;
904+
use Symfony\Component\Lock\LockFactory;
905+
906+
$registry = new LockPoolRunRegistry($lockFactory, 10); // pool of 10 slots
907+
```
908+
909+
Uses `symfony/lock` with a fixed pool of lock keys. Works with any lock store (Redis, database, etc.).
910+
If all pool slots are taken, throws `LogicException` - increase the pool size.
911+
912+
### Status command
913+
914+
Check maintenance state and active runs:
915+
916+
```bash
917+
php bin/console scheduler:status
918+
```
919+
920+
Output:
921+
922+
```
923+
Maintenance: ACTIVE
924+
Active runs: 2
925+
- 1712345678-a3f2b1 (started 15s ago)
926+
- 1712345679-d4e5c2 (started 3s ago)
927+
Ready for shutdown: NO
928+
```
929+
930+
Also available programmatically:
931+
932+
```php
933+
$status = $manager->getStatus();
934+
$status->isMaintenance(); // bool
935+
$status->getActiveRunIds(); // list<string>
936+
$status->isReadyForShutdown(); // true when maintenance active AND no active runs
937+
```
938+
939+
### Signal handling
940+
941+
`RunCommand` and `WorkerCommand` handle `SIGTERM` and `SIGINT` signals using Symfony's `SignalRegistry`:
942+
943+
- **RunCommand**: first signal triggers graceful shutdown (same as maintenance mode), second signal forces immediate exit
944+
- **WorkerCommand**: first signal stops spawning new `scheduler:run` processes and waits for current ones to finish, second signal forces immediate exit
945+
946+
Signal handling in `RunCommand` requires `MaintenanceManager` to be passed to the command constructor.
947+
`WorkerCommand` always registers signal handlers when run through a Symfony `Application`.
948+
949+
Signal handling requires the `pcntl` extension (available on Linux/macOS CLI, not on Windows).
950+
When pcntl is not available, signals are silently skipped - the `MaintenanceChecker` polling approach still works.
951+
952+
### Deploy integration
953+
954+
Typical deploy flow:
955+
956+
1. Enable maintenance (create your maintenance flag)
957+
2. Poll `scheduler:status --fail-when-not-ready` until ready (exits with `0` when ready, `1` when not):
958+
959+
```bash
960+
while ! php bin/console scheduler:status --fail-when-not-ready; do
961+
sleep 1
962+
done
963+
```
964+
965+
3. Deploy the application
966+
4. Disable maintenance (remove your maintenance flag)
967+
968+
### Exit codes
969+
970+
The `scheduler:run` command returns:
971+
972+
- `0` - all jobs completed successfully
973+
- `1` - one or more jobs failed
974+
- `2` - maintenance shutdown (run was stopped due to maintenance)
975+
815976
## Lazy loading
816977

817978
Jobs are executed only when it is their due time. To prevent initializing potentially heavy job dependencies when they

‎src/Command/BaseRunCommand.php‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,10 @@ protected function renderJob(JobSummary $summary, int $terminalWidth, OutputInte
6161
case JobResultState::lock():
6262
$status = "<fg=yellow>$stateName</>";
6363

64+
break;
65+
case JobResultState::maintenance():
66+
$status = "<fg=yellow>$stateName</>";
67+
6468
break;
6569
}
6670

‎src/Command/RunCommand.php‎

Lines changed: 73 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
use Generator;
66
use Orisai\Clock\SystemClock;
7+
use Orisai\Scheduler\Maintenance\MaintenanceManager;
78
use Orisai\Scheduler\Scheduler;
89
use Orisai\Scheduler\Status\JobResultState;
910
use Orisai\Scheduler\Status\JobSummary;
@@ -12,19 +13,33 @@
1213
use Symfony\Component\Console\Input\InputInterface;
1314
use Symfony\Component\Console\Input\InputOption;
1415
use Symfony\Component\Console\Output\OutputInterface;
16+
use Symfony\Component\Console\SignalRegistry\SignalRegistry;
17+
use function assert;
1518
use function json_encode;
19+
use function method_exists;
1620
use const JSON_PRETTY_PRINT;
1721
use const JSON_THROW_ON_ERROR;
22+
use const SIGINT;
23+
use const SIGTERM;
1824

1925
final class RunCommand extends BaseRunCommand
2026
{
2127

28+
private const EXIT_MAINTENANCE = 2;
29+
2230
private Scheduler $scheduler;
2331

24-
public function __construct(Scheduler $scheduler, ?ClockInterface $clock = null)
32+
private ?MaintenanceManager $maintenanceManager;
33+
34+
public function __construct(
35+
Scheduler $scheduler,
36+
?ClockInterface $clock = null,
37+
?MaintenanceManager $maintenanceManager = null
38+
)
2539
{
2640
parent::__construct($clock ?? new SystemClock());
2741
$this->scheduler = $scheduler;
42+
$this->maintenanceManager = $maintenanceManager;
2843
}
2944

3045
public static function getDefaultName(): string
@@ -44,13 +59,65 @@ protected function configure(): void
4459

4560
protected function execute(InputInterface $input, OutputInterface $output): int
4661
{
47-
$summary = $this->scheduler->runPromise();
62+
$signalRegistry = $this->registerSignalHandlers();
63+
64+
try {
65+
$generator = $this->scheduler->runPromise();
66+
67+
$success = $input->getOption('json')
68+
? $this->renderJobsAsJson($output, $generator)
69+
: $this->renderJobs($output, $generator);
70+
71+
$runSummary = $generator->getReturn();
72+
assert($runSummary instanceof RunSummary);
73+
if ($runSummary->isMaintenanceActive()) {
74+
return self::EXIT_MAINTENANCE;
75+
}
76+
77+
return $success ? self::SUCCESS : self::FAILURE;
78+
} finally {
79+
if ($signalRegistry !== null && method_exists($signalRegistry, 'popPreviousHandlers')) {
80+
$signalRegistry->popPreviousHandlers();
81+
}
82+
}
83+
}
84+
85+
private function registerSignalHandlers(): ?SignalRegistry
86+
{
87+
if ($this->maintenanceManager === null) {
88+
return null;
89+
}
90+
91+
if (!SignalRegistry::isSupported()) {
92+
return null;
93+
}
94+
95+
$application = $this->getApplication();
96+
if ($application === null) {
97+
return null;
98+
}
99+
100+
$registry = $application->getSignalRegistry();
101+
if (method_exists($registry, 'pushCurrentHandlers')) {
102+
$registry->pushCurrentHandlers();
103+
}
104+
105+
$maintenanceManager = $this->maintenanceManager;
106+
$shutdownRequested = false;
107+
$handler = static function () use ($maintenanceManager, &$shutdownRequested): void {
108+
if ($shutdownRequested) {
109+
// Second signal - force exit, tested via real subprocess in SignalHandlingTest
110+
exit(1); // @codeCoverageIgnore
111+
}
112+
113+
$shutdownRequested = true;
114+
$maintenanceManager->requestShutdown();
115+
};
48116

49-
$success = $input->getOption('json')
50-
? $this->renderJobsAsJson($output, $summary)
51-
: $this->renderJobs($output, $summary);
117+
$registry->register(SIGTERM, $handler);
118+
$registry->register(SIGINT, $handler);
52119

53-
return $success ? self::SUCCESS : self::FAILURE;
120+
return $registry;
54121
}
55122

56123
/**

0 commit comments

Comments
 (0)