UPS monitor/shutdown service

stoicmaverick@lemmy.world · 21 hours ago

UPS monitor/shutdown service

mbirth 🇬🇧@lemmy.ml · 19 hours ago

What was the difficulty with NUT for you? The setup is pretty straightforward, IMHO.

Should be easy to wrap in a container, too.

stoicmaverick@lemmy.world · 18 hours ago

I was hoping to setup a server and clients to shutdown the other Raspberry Pis and such, scattered around the house if the main server senses a power failure.

Also, I had an issue where it insisted on triggering a shutdown the SECOND the UPS triggered, even if it came right back. I might take another swing at it at some point, but I liked the look of that PVE-UPS things.

tychosmoose@lemmy.world · 12 hours ago

A basic NUT configuration does an immediate shutdown as soon as an ONBATT status is reached (upssched.conf has “at ONBATT execute…”, which causes the upssched-cmd to run the shutdown command immediately). But you can change that to a timer, which can be cancelled if the UPS goes back online, or only shutdown at LOWBATT.

It’s complicated, but NUT should be able to do everything you have stated as a goal. But it takes some testing.

One thing I found helpful while setting it up is the Dummy UPS driver that is included in the documentation. Rather than plug and unplug a physical UPS during testing you can make text files with various ups statuses and cp one to the dummy’s input file target to make it change to battery power or back online. Use logger to log information into the system journal from upssched-cmd to see what’s running in your script (with a unique keyword to grep on). Monitor in another terminal with journalctl -f | grep yourkeyword to view your messages. This made it much easier to get the upssched stuff sorted out.

One of my goals was unattended power-on management for when I was away from home. If power returned for an amount of time, send power commands to various smart outlets to bring everything back up in sequence. So I used a pretty big upssched-cmd, with timers for both shutdown and power up and different UPS battery levels (no power up until 30% battery, for example). It would have been pretty difficult to debug without the dummy UPS.

Anyway, NUT is really powerful.

stoicmaverick@lemmy.world · 12 hours ago

Thanks. Good info. Feel like sharing your conf file to see how it’s formatted in an example that actually works? Might make this my weeks evening project.

tychosmoose@lemmy.world · 11 hours ago

It’s a lot. But here are my most recent notes.

Note that the pipe & lock locations different by distro, as does the nut username. So these files won’t drop in and work anywhere.

The dummy driver will need to be installed and configured.

Mine also have several layers of timers on battery an online:

NUT server: In ONBATT I use sleep commands to time subsequent commands. These will always run.
NUT server: Onbatt5 is a upssched timer I set to 300s and is cancelled if the UPS returns to line power.
NUT client: Onbatt30s starts on the client when it gets an ONBATT status from the server.

Most configs would have the client side cancel their timers if power returns. In my case I don’t want that because I prefer more certainty of state so that I know it’s safe to power cycle an outlet to bring a device back up. My NUT server runs on an old pi 3b with a read-only file system. So it never needs a safe shutdown. It’s the last man standing to bring everything else back up if the UPS returns to line power before the battery is exhausted. That’s why there is no local shutdown on the NUT server config.

Each snmpset... is a truncated version of an outlet power command.

Also, the wall commands were a debug aid. Uncomment to use, but test a command first. Wall doesn’t work on some distros.

NUT Server

NUT.CONF:

	MODE=netserver

UPS.CONF:

	[ups]
	desc = My UPS
	driver = dummy-ups
	port = /opt/ups/ups.seq

UPSD.CONF:

	LISTEN 0.0.0.0 3493
	## all interfaces
	## STATEPATH /var/run/nut

UPSD.USERS:

	[upsuser]
	  password = <pw>
	  actions = set
	  upsmon master
	[upssrv]
	  password = <pw>
	  actions = set
	  upsmon slave
	[upspi]
	  password = <pw>
	  actions = set
	  upsmon slave

UPSMON.CONF:

	RUN_AS_USER nut
	#RUN_AS_USER root
	MONITOR ups@localhost 1 upsuser <pw> master
	NOTIFYMSG ONLINE "UPS %s on line power"
	NOTIFYMSG ONBATT "UPS %s on battery"
	NOTIFYMSG LOWBATT "UPS %s battery is low"
	NOTIFYMSG FSD "UPS %s: forced shutdown in progress"
	NOTIFYMSG COMMOK "Communications with UPS %s established"
	NOTIFYMSG COMMBAD "Communications with UPS %s lost"
	NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding"
	NOTIFYMSG REPLBATT "UPS %s battery needs to be replaced"
	NOTIFYMSG NOCOMM "UPS %s is unavailable"
	NOTIFYMSG NOPARENT "upsmon parent process died - shutdown impossible"
	
	NOTIFYFLAG ONLINE WALL+EXEC+SYSLOG
	NOTIFYFLAG ONBATT WALL+EXEC+SYSLOG
	NOTIFYFLAG LOWBATT SYSLOG+WALL
	NOTIFYFLAG FSD SYSLOG+WALL
	NOTIFYFLAG COMMOK SYSLOG+WALL
	NOTIFYFLAG COMMBAD SYSLOG+WALL
	NOTIFYFLAG SHUTDOWN SYSLOG+WALL
	NOTIFYFLAG REPLBATT SYSLOG+WALL
	NOTIFYFLAG NOCOMM SYSLOG+WALL
	NOTIFYFLAG NOPARENT SYSLOG+WALL
	
	MINSUPPLIES 1
	SHUTDOWNCMD "/sbin/shutdown -P +0"
	NOTIFYCMD "/usr/sbin/upssched"
	POLLFREQ 5
	POLLFREQALERT 5
	HOSTSYNC 15
	DEADTIME 15
	# POWERDOWNFLAG /var/run/killpower
	RBWARNTIME 43200
	NOCOMMWARNTIME 300
	FINALDELAY 5
	CERTVERIFY 0
	FORCESSL 0
	
	UPSSCHED.CONF:
	CMDSCRIPT /etc/nut/upssched-cmd
	PIPEFN /var/run/nut/upssched.pipe
	LOCKFN /var/run/nut/upssched.lock
	
	AT ONBATT ups@localhost CANCEL-TIMER online3
	AT ONBATT ups@localhost CANCEL-TIMER online10
	AT ONBATT ups@localhost START-TIMER onbatt5 300
	AT ONBATT ups@localhost EXECUTE onbattnotify
	
	AT ONLINE ups@localhost CANCEL-TIMER onbatt5
	AT ONLINE ups@localhost START-TIMER online3 180
	AT ONLINE ups@localhost START-TIMER online10 600
	AT ONLINE ups@localhost EXECUTE onlinenotify

UPSSCHED-CMD

	#! /bin/bash
	#
	# This script should be called by upssched via the CMDSCRIPT directive
	# The first argument passed to your CMDSCRIPT is the name of the timer
	# from your AT lines.
	logger -t upssched-cmd "Calling upssched-cmd $1"
	wall "Calling upssched-cmd $1"
	UPS="ups@localhost"
	HOST=$( uname -n)
	TIMESTAMP=$(date +%Y%m%d%H%M%S)
	STATUS=$( upsc $UPS ups.status )
	CHARGE=$( upsc $UPS battery.charge )
	CHMSG="UPS[STATUS]:$CHARGE%"
	case $1 in
			onbattnotify)
					# srv and pi are upsmon clients and will shut down on their own after a 30s delay.
					# Log the Notification.
					logger -t upssched-cmd "UPS on Battery Power. Servers will shutdown in 30 seconds. Switch shutdown in 5 minutes. $CHMSG $TIMESTAMP"
					wall "upssched-cmd onbattnotify: UPS on Battery Power. Servers will shutdown in 30 seconds. Switch shutdown in 5 minutes. $CHMSG"
	#                wall "upssched-cmd: sleeping to give servers time to poweroff"
					sleep 120s
					# Power off PDU outlets to save power.
					logger -t upssched-cmd "UPS on Battery Power. PDU outlets 4-srv-drives, 5-srv, 7-pi and 8-homeassistant power down NOW. $CHMSG $TIMESTAMP"
					snmpset ...
					snmpset ...
					snmpset ...
					snmpset ...
	#                wall "End upssched-cmd $1"
					;;
			onbatt5)
					# Shut down servers
					logger -t upssched-cmd "UPS on Battery Power. PDU outlets 3-switch and 8-homeassistant power down NOW. $CHMSG $TIMESTAMP"
					snmpset ...
					snmpset ...
	#                wall "End upssched-cmd $1"
					;;
			onlinenotify)
					# Log the notification.
					logger -t upssched-cmd "UPS back on Line Power. Bringing up network in 3 minutes, servers in 10 minutes. $CHMSG $TIMESTAMP"
	#                wall "End upssched-cmd $1"
					;;
			online3)
					# Power on PDU outlets for main switch to prep for server startup.
					logger -t upssched-cmd "UPS back on Line Power. Bringing up PDU outlet 3-switch NOW. $CHMSG $TIMESTAMP"
					snmpset ...
	#                wall "End upssched-cmd $1"
					;;
			online10)
					# Power on PDU outlets for servers now.
					logger -t upssched-cmd "UPS on Line Power. Bringing up PDU outlets 5-srv, 7-pi, 8-homeassistant NOW. $CHMSG $TIMESTAMP"
					snmpset ...
					snmpset ...
					snmpset ...
	#                wall "End upssched-cmd $1"
					;;
			upsgone)
					logger -t upssched-cmd "UPS communication has been gone awhile on $HOST. $TIMESTAMP"
					;;
			)
					logger -t upssched-cmd "Unrecognized command: $1"
					;;
	esac

NUT Client

NUT.CONF

MODE=netclient

UPS.CONF

maxretry = 3

UPSD.CONF

# empty

UPSD.USERS

# empty

UPSMON.CONF

RUN_AS_USER nut  
  
MONITOR ups@192.168.0.77 1 upspi <pw> slave  
NOTIFYCMD /usr/sbin/upssched  
SHUTDOWNCMD "sudo /sbin/shutdown -P +0"  
  
NOTIFYMSG ONLINE "UPS %s on line power"  
NOTIFYMSG ONBATT "UPS %s on battery"  
NOTIFYMSG LOWBATT "UPS %s battery is low"  
NOTIFYMSG FSD "UPS %s: forced shutdown in progress"  
NOTIFYMSG COMMOK "Communications with UPS %s established"  
NOTIFYMSG COMMBAD "Communications with UPS %s lost"  
NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding"  
NOTIFYMSG REPLBATT "UPS %s battery needs to be replaced"  
NOTIFYMSG NOCOMM "UPS %s is unavailable"  
NOTIFYMSG NOPARENT "upsmon parent process died - shutdown impossible"  
  
NOTIFYFLAG ONLINE WALL+EXEC+SYSLOG  
#+SYSLOG+WALL  
NOTIFYFLAG ONBATT WALL+EXEC+SYSLOG  
#+SYSLOG+WALL  
NOTIFYFLAG LOWBATT SYSLOG+WALL  
NOTIFYFLAG FSD SYSLOG+WALL  
NOTIFYFLAG COMMOK SYSLOG+WALL  
NOTIFYFLAG COMMBAD SYSLOG+WALL  
NOTIFYFLAG SHUTDOWN SYSLOG+WALL  
NOTIFYFLAG REPLBATT SYSLOG+WALL  
NOTIFYFLAG NOCOMM SYSLOG+WALL  
NOTIFYFLAG NOPARENT SYSLOG+WALL  

MINSUPPLIES 1  
POLLFREQ 10  
POLLFREQALERT 5  
HOSTSYNC 15  
DEADTIME 15  
RBWARNTIME 43200  
NOCOMMWARNTIME 300  
FINALDELAY 5

UPSSCHED.CONF

CMDSCRIPT /etc/nut/upssched-cmd  
PIPEFN /run/nut/upssched.pipe  
LOCKFN /run/nut/upssched.lock  
  
AT ONBATT ups@192.168.0.77 EXECUTE onbattnotify  
AT ONBATT ups@192.168.0.77 START-TIMER onbatt30s 30

# Disabled the online cancel actions in favor of having certainty on the state.
# Servers will go down on an OB event, but that's fine. When Nutpi sees it back
# OL it will bring up the servers
#AT ONLINE ups@192.168.0.77 CANCEL-TIMER onbatt30s  
#AT ONLINE ups@192.168.0.77 EXECUTE onlinenotify

UPSSCHED-CMD

#! /bin/sh  
#  
# This script should be called by upssched via the CMDSCRIPT directive.  
#  
# The first argument passed to your CMDSCRIPT is the name of the timer  
# from your AT lines.  
  
logger -i -t upssched-cmd "Calling upssched-cmd $1"  
UPS="ups@192.168.0.77"  
HOST=$( uname -n)  
STATUS=$( upsc $UPS ups.status 2>&1 | grep -v '^Init SSL' )  
CHARGE=$( upsc $UPS battery.charge 2>&1 | grep -v '^Init SSL' )  
CHMSG="$UPS[$STATUS]:$CHARGE%"  
  
case $1 in  
onbattnotify)  
# Log the Notification.  
logger -t upssched-cmd "UPS on Battery Power. Shutting down $HOST in 30 seconds. $CHMSG"  
wall "upssched-cmd: UPS on Battery Power. Shutting down $HOST in 30 seconds. $CHMSG"  
;;  
onbatt30s)  
# Shut down servers  
logger -t upssched-cmd "UPS on Battery Power. $HOST shutting down NOW. $CHMSG"  
wall "upssched-cmd: UPS on Battery Power. $HOST shutting down NOW. $CHMSG"  
sudo shutdown -P +0  
;;  
# onlinenotify)  
# # Log the notification.  
# logger -t upssched-cmd "UPS back on Line Power. $HOST is already online. $CHMSG"  
# wall "upssched-cmd: UPS back on Line Power. $HOST is already online. $CHMSG"  
# ;;  
upsgone)  
logger -t upssched-cmd "UPS communication has been gone awhile on $HOST."  
;;  
*)  
logger -t upssched-cmd "Unrecognized command: $1"  
;;  
  
esac

mbirth 🇬🇧@lemmy.ml · 17 hours ago

On clients you mostly have to set MODE=netclient in the nut.conf and modify the upsmon.conf according to the manual and detailed manual. By default, it should only shutdown systems when reaching a “low battery” situation, not immediately when switching to battery power.

neidu4@piefed.social · 16 hours ago

Note to self: Get around to wiring my UPS onto my management network for monitoring and graceful shutdown of non-essential stuff.

Ajen@sh.itjust.works · 16 hours ago

It’s also a good idea to gracefully shutdown anything essential that isn’t using a read-only rootfs. I’ve got my NAS configured to shut down after 5 minutes, but my firewall and frigate server keep running until the battery is almost dead, then safely shut down if the power still hasn’t come back yet.

_g_be@lemmy.world · 12 hours ago

Hehe I also have trouble serving NUT when I’ve had too much whiskey