The S E B transmitter has been known to lose its internet connection. Because internet. Or worse yet, the unit could lose power and shut down completely.
As a matter of fact, I know from experience that the station could “go down” due to any number of unforseen factors
- unplugging the wrong power cord
- a broken cgi-bin script deployment
- another Windows 10 Home Edition forced upgrade
- PG&E wildfire prevention blackouts
- some VirtualBox crash in the middle of night
Not a perfect science this is.
Please bear with S E B during its times of challenge.
An Anthology
The more egregious outages – 30 minutes or more – generally get documented on the S E B Facebook Page. I launched the Page back in 2007, but as I check it now, there are no Posts shown prior to 2013. Which means that the first fourteen years of the station’s life are a bit of a Dark Period.
Reviewing all the Posts has been a trip down memory lane for me. Here, let me tell you about it …
Short Notice
If I have a notion that the station will be down for a while, I’ll let you all know, with a reasonable time estimate.
Back when the transmitter was on dedicated hardware, I’d sometimes need to take full image backups. Fortunately, those regular backups allowed S E B to recover quickly after its hardware died in on 2014-12-13.
Sometimes my relay provider will have a planned outage. If I have advance warning, I’ll forward it along.
Random Networking
The Facebook feed is replete with apologies for ISP failures and other connectivity issues. Like, you know. We’ve all been there. No reason to link to them here …
Except these posts from 2013-Nov-05 and 2015-Mar-01, which document buffoonery with my Router du jour.
Power Outages
Electricity. S E B needs it. Sometimes it goes away.
It’s usually a total surprise. If my phone is charged, I can post that all-important Facebook update, like I did on 2013-Jun-05 and 2014-Jul-07.
And then, there’s living on the PG&E grid in California. They schedule blackouts during high wind periods to reduce wildfire risk. 2019 was the culmination of a particularly long drought, and there were outages on Oct-08 and Oct-27.
At other times, PG&E will just yoink the plug, such as during a windstorm on 2020-Oct-25. Also, because October.
Window 10 Home Edition
It’s lovely proprietary OS which would regularly upgrade itself, reboot, and lock me out of the transmitter.
My rantifesto on this subject warranted its own page.
No Remote Access
I’ve had a setup to RDP into the Windows XP runtime for most of S E B’s lifespan. Sometimes the Winamp Player can disconnect from the primary relay (which requires manual correction). Or I might decide to switch up the playlist; put on a bunch of Broadcast Disc 5 for a while, or switch to 100% environmental-only for 24 hours.
But if the Windows XP guest crashes, the VM needs to be relaunched … So I have RDP setup for host server access as well.
But if I can’t contact the transmitter at all – the station is down, and it stays down. Like that year when I was off at Burning Man (part of the pre-2013 Dark Period in Facebook). Or during Holiday Break on 2016-Dec-27. Or that time I was in Iowa for a wedding on 2017-Aug-26.
It looks here like the transmitter became unaccessible starting 2019-Mar-12 and took three whole days to restore. Those scale of incidents are rare & far between.
Health Checks
Observing the station’s uptime is critical to maintaining a good listing experience.
Since 2006, the Ambience for the Masses data server has tracked S E B’s Current Song. I receive a summary Prowl notification every two hours. When it says “0 TRACKS” played, then something is broke. But … I have to notice first … and after up to a two hour delay.
Clearly, 2014 was rough. I dropped apologies on Jan-23, Feb-16 and Feb-27. I remember making some data server changes, but they weren’t helping as of Dec-06.
It was always a guessing game. The weekend of 2016-Feb-29 rolls around, and the passive monitoring approach still isn’t cutting it.
It’s around that time where it occurred to me
Hmm. I could add an HTTP health check for that …
So I did. Today, each relay gets checked every five minutes, and I get an urgent Prowl notification when things go south. The first proof of improvement is from 2018-Jun-12.
And ever since the transmitter moved to Linux, it has exposed metrics about the Windows XP VM for additional checks.
As of 2021, there’s very high visibility into uptime.
Bad Deployments
Speaking of 2021 … I broke the standard Listen link while I was revamping this website. The live public link responded with HTTP 404 for about three days ending Jun-08.
It was a really dunder-headed move on my part. The listener count hasn’t fully recovered even after a month. I feel sad.
I’m thinking
Hmm. I could add an HTTP health check for that …