At work we have a datacenter, so we have things that require monitoring like generators, air handlers (industrial aircon to keep the servers cool), and uninterruptible power supplies. Our building monitoring system was originally written by a colleague that has long since left and the responsibility of maintaining it has…
I wanted an easy way to query our nagios server to see which hosts are being monitored and what checks they have assigned to them for auditing. I also wanted it to be speedy as no one wants to wait 30 seconds for a lookup, so some kind of caching…
At $work we have a client with a hadoop cluster, they wanted a nagios check that would check that all of the regions for a specified table were online/queryable. After some research it seemed like the best way was to use stargate which is the REST api for hbase.…
As an update to the previous post I'm glad to say that our puppet setup is working pretty well! The only issue I've had is that the cron generation script we were using started to break crontab on servers. This seems to be due to it creating blank entries when…
So.. Recently we have had a weird issue at $work with puppet where the puppet runs on quite a lot of servers have appeared to all bunch up and run at the same time. Like this video, but instead of metronomes you have ~1800 servers that were once staggered all…
At $work we use glorious puppet for our configuration management needs. The first version of our puppetmaster was a xen virtual machine with 2 cores, 8GB RAM and 60GB diskspace. This was fine for a while. However soon it started struggling under the load as the amount of servers managed…
As $work is a hosting provider we have lots of cPanel servers, so naturally they come under bruteforce attempts by bots/hacked sites. At first I wanted to create a script to monitor how many IPs were attempting to bruceforce login pages of sites on our shared servers, but it…
Currently it's a manual process to sign a hosts certificate request on our puppetmaster, which is fine for normal server installations but as I have been automating more of our cPanel VDS setups there ends up being a bunch of certs that need to signed, which then get forgotten as…
I keep having the issue of accidently attaching another screen session inside of an already attached screen session on my bouncebox at work (which has about 19 screen sessions running..) and because just doing screen -ls and looking for which one is Attached is too much common sense I wrote…
Everyone is aware of the heartbleed exploit for OpenSSL, since it's been all over the news/tech sites so I won't ramble on about it. I'm just going to show how I used puppet to patch our vunerable servers. We currently have over 1500 hosts on puppet, of which the…