Systemd service failed — push на телефон.

Вариант 1: OnFailure unit

Самый чистый способ для критичных сервисов: systemd сам вызовет helper unit при failed state.

# /etc/systemd/system/[email protected]
[Unit]
Description=Nerve failure alert for %i

[Service]
Type=oneshot
Environment=NERVE_DSN=nerve://TOKEN:[email protected]
ExecStart=/bin/sh -c 'echo "SYSTEMD FAILED: %i on $(hostname)" | /usr/local/bin/nerve send --severity critical'

# /etc/systemd/system/my-app.service
[Unit]
Description=My Application
OnFailure=nerve-notify@%n.service

[Service]
ExecStart=/usr/local/bin/my-app
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl restart my-app.service

Вариант 2: timer для всех failed units

Если не хотите менять каждый unit, проверяйте общий список failed-сервисов раз в несколько минут.

# /etc/systemd/system/nerve-check-failed.service
[Unit]
Description=Check failed systemd units

[Service]
Type=oneshot
Environment=NERVE_DSN=nerve://TOKEN:[email protected]
ExecStart=/bin/sh -c 'FAILED=$(systemctl list-units --state=failed --no-legend --plain | head -10); if [ -n "$FAILED" ]; then echo "failed units on $(hostname):\n$FAILED" | nerve send --severity critical; fi'

# /etc/systemd/system/nerve-check-failed.timer
[Unit]
Description=Check failed units every 5 min

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

sudo systemctl enable --now nerve-check-failed.timer

Restart loop

SERVICE="my-app.service"
COUNT=$(journalctl -u "$SERVICE" --since "10 min ago" | grep -c "Started" || true)
if [ "$COUNT" -gt 3 ]; then
  echo "$SERVICE restart loop: $COUNT restarts in 10 min" | nerve send --severity critical
fi

Когда достаточно этого подхода

FAQ