Then I had to design a blanking panel for the grill piece and to mount the connector on. I took some basic measurements and mocked up a shape in Tinkercad, this was then 3d printed very thin in PLA so I could use it for refining the shape and hole locations against the grill.

This was drawn on with sharpie and then measurements taken and modifications made to the cad model. After several prototype prints to check the refinements I had the final design.

This was printed in PETG filament so it's suitable for outdoor use, the panel is 3mm thick. I also printed a little tag for it which was crudely superglued in place.

I offered it up to the grill part and drilled out the 6mm mounting holes. Then it was bolted into place.

Then it was just a case of fitting it onto the car. My trickle charger came with a loom that has ring terminals one end for connecting to the battery and a SAE connector at the other end. Luckily this was long enough that I was able to feed it down behind the bumper to where the trim clips in. To connect the SAE connectors together you annoyingly seem to need a middle piece else the polarity swaps over.

With the trim clipped back in place and the plug connected to the socket

Showing how easily the connector comes away

The lead goes to a secured point, so if I reverse away without disconnecting it'll just pull away like in the clip. The battery maintainer lives in a weatherproof box by the car currently and it isn't automatic so I still need to press a button to get it to charge/maintain, but connecting the car up and doing that is a lot easier than dealing with a normal extension reel and having to open the bonnet.

Parts List:

Magcode Plug/Socket/Cap: £58
SAE Leads: £9
SAE Adapters: £7
Weatherproof box: £10
Battery Maintainer: £30

Once the bot is created, you will be able to search for it and message it.

To get the messages you send to the bot, you can either poll the getUpdates endpoint or setup a web endpoint that telegram will POST some JSON to. This is the method I'm using but I will give an example of the updates API.

I'm using curl piped to jq for pretty printing of JSON on the command line. The format of the API URLs is;

https://api.telegram.org/{bot_token}/{endpoint}

For the getUpdates endpoint;

$ curl -s "https://api.telegram.org/bot1234:onDWzmiOTGB2V53qeUhXFxC1tHI0RS/getUpdates" | jq
{
  "ok": true,
  "result": [
    {
      "update_id": 123,
      "message": {
        "message_id": 2,
        "from": {
          "id": 1234567,
          "is_bot": false,
          "first_name": "Craig",
          "username": "exampleusr",
          "language_code": "en"
        },
        "chat": {
          "id": 1234567,
          "first_name": "Craig",
          "username": "exampleusr",
          "type": "private"
        },
        "date": 1613134430,
        "text": "Boop"
      }
    }
  ]
}`

As you can see it returns an array of the message. The chat id is important as you need that to send a message back to the chat, and message text is the content of the message you have sent. If you were only going to send messages to yourself from an application, you could use the sendMessage endpoint (example below) after you have found your chat id from the getUpdates endpoint. As I wanted to be able to request info as well as send messages I went with the webhook method.

To have Telegram send the messages to your webserver you need to set a webhook via the API. Your webserver requires a valid SSL certificate, and not self signed. So use Lets Encrypt.

$ curl -s "https://api.telegram.org/bot1234:onDWzmiOTGB2V53qeUhXFxC1tHI0RS/setWebhook?url=https://serverhostna.me/telegramexample/" | jq
{
  "ok": true,
  "result": true,
  "description": "Webhook was set"
}

You can then check your webhooks info with the API

$ curl -s "https://api.telegram.org/bot1234:onDWzmiOTGB2V53qeUhXFxC1tHI0RS/getWebhookInfo" | jq
{
  "ok": true,
  "result": {
    "url": "https://serverhostna.me/telegramexample/",
    "has_custom_certificate": false,
    "pending_update_count": 0,
    "max_connections": 40,
    "ip_address": "185.119.173.64"
  }
}

Now when you send a message to your bot, some JSON will get posted to your URL.

To send a message you send to the sendMessage endpoint, using the chat id retrieved from the message update/webhook, and the text field for the message should be urlencoded.

$ curl -s "https://api.telegram.org/bot1234:onDWzmiOTGB2V53qeUhXFxC1tHI0RS/sendMessage?chat_id=1234567&text=ping" | jq
{
  "ok": true,
  "result": {
    "message_id": 5,
    "from": {
      "id": 1234,
      "is_bot": true,
      "first_name": "Example",
      "username": "examplebot"
    },
    "chat": {
      "id": 1234567,
      "first_name": "Craig",
      "username": "exampleusr",
      "type": "private"
    },
    "date": 1613141321,
    "text": "ping"
  }
}

An example of my php webhook on my server;

<?php

$telegram_auth = 'bot1234:onDWzmiOTGB2V53qeUhXFxC1tHI0RS';

$input = file_get_contents("php://input");
$input = json_decode($input,true);

if ($input) {
    $user_id = $input['message']['from']['id'];
    $chat_id = $input['message']['chat']['id'];
    $request = strtolower($input['message']['text']);
    if ($user_id != '1234567') {
        exit();
    }
    if ($request == 'temps') {
        $temps = json_decode(file_get_contents("https://serverhostna.me/?temps"),true);
        foreach ($temps as $key => $value) {
            $output .= $key . ' => ' . $value . PHP_EOL;
        }
        $message = urlencode($output);
        file_get_contents("https://api.telegram.org/".$telegram_auth."/sendMessage?chat_id=".$chat_id."&text=".$message);
    }
}

As you can see, I take the input and decode it into an array, which then can be processed. I've got a check for the requesting user, if it doesn't match my user id the script will exit. I also convert the incoming message to lowercase for easier matching, for when the mobile app capitalises the first letter of a word.

My first feature is to return the output of my temperature sensors. The message is urlencoded for the request to the API.

And it in action!

The nurse guided me to a treatment bay and asked me to sit on the bed so she could clean the wound and have a look. She also commended me on my emergency dishcloth bandage for stopping the bleeding. She was like "Wow, you've certainly done a good number on this, it looks almost like a puncture wound" at this point I did start feeling faint and had to lie down. The wound didn't actually hurt that much until she started spraying saline solution into it to clean it up. She steristriped my finger back together and bandaged my hand up all neatly. When she was doing the assessment we did find out that there was some numbness on the side of my finger that pointed to nerve damage. So she phoned the hand specialists in Oxford. They wanted an X-Ray, so my nice bandage had to be cut off, and an X-Ray taken. Thankfully the X-Ray showed no damage to my bones. I was re-bandaged up, and then within 3 hours of stabbing myself I was discharged with a course of antibiotics to take for 7 days (3 of the biggest tablets I've ever had, per day), and an appointment at the hand specialists in Oxford the next day. I also missed curry with my friends.

The next day I was driven to Oxford by my dad for the appointment with the specialists. Again there was a bit of waiting around, before I my name was called. I sat in an armchair in a curtained enclosure whilst a nurse cut my bandage off, I could then see my damaged finger all steristiped together. Again I felt faint and had to have some water. The nurse also got my dad so he could come sit with me which was nice. After another period of waiting a consultant doctor came to inspect my finger. He made me move it lots and push and pull against his finger, which was fairly painful as it pulled against the cut that was starting to heal together. But whilst he was happy with my finger movement, he wasn't with the numb bit on my finger. It was nerve damage he said as had probably severed the tube the nerves are in, and that I would have to have surgery to fix it. If I didn't have surgery then there was a risk that the nerves would regrow in the wrong place and it would be very painful. So I signed the consent form, and they would call me the day before my surgery but couldn't tell me when it would be. Hopefully within a week. I also got a tetanus top-up jab.

That was Monday. On Tuesday I got a call that my operation was to be the next day and I had to be there at 8am. I also couldn't eat anything past midnight (I had a cheese and pickle sandwich around 7pm), or drink anything past 6am. Other than some water with my antibiotic tablet. Once there I checked in with the nurses, then it was some more waiting in reception. Then I was was taken through to the ward. They took my weight and gave me a robe and socks, but said I don't need to change into them until it was my time. Then it was more waiting. First the Anaesthesiologist came to see me to discuss what kind of anaesthesia I was to get. I said I don't particularly want to go to sleep with a general and he said a regional anaesthetic would be fine. Where he would just make my whole arm numb so I couldn't feel anything but I'd still be awake whilst they were operating. This was still much preferable to me than being knocked out. I've had a general before when they put a camera in my lungs as a child (at the same hospital actually) and all that I remember is being in a room with doctors etc before it, and then in a ward eating a cheese sandwich afterwards.. I don't really want to experience that again just because the whole going to sleep til it's done and not knowing weirds me out. After his visit there was more waiting, then the surgeons came around to discuss the surgery with me, and what they would be doing. They then marked my arm with a sharpie to point to what was being operated on.

So Left index finger, then a dot on the finger as well. Then it was back to waiting. A lot of waiting it turns out. They came around at about 10am, and I didn't get collected for surgery until 4pm. As I didn't sleep that well and I had to get up early to be there for 8am I spent most of the day napping. I was also very thirsty since I hadn't drunk any water since I had at my tablet around 6am..

Eventually I heard my name being discussed and the nurse came to see me, then I had to change into my robe and some compression socks. Putting socks on one handed is very hard, especially so when they are compression socks.. Turns out the socks are only needed if you are having general, but as my consent form listen general as well as regional I needed them. I was then wheeled to theatre, or rather the pre surgery room where there either numb you up or put you to sleep. They did put a cannula into my hand and give me a sedative but I didn't feel too much different so I guess it just took the edge off. The Anaesthesiologist then whipped out the biggest needle I've ever seen (it must have been like a foot long) and started injecting my armpit (with the help of an assistant) with drugs to numb up my arm. He was using what looked like an ultrasound to see where my nerves were then was injecting the drugs near them. I didn't look too closely even though I still had my glasses on. I just stared at the ceiling. A couple of times he poked too close and it made my arm spasm. It wasn't too painful, just a very weird feeling. This took around 20 minutes and once it was done and my arm was suitably numb I was wheeled into the theatre.  

Once I was in, I had to shuffle myself onto the operating table. This is where it got a bit surreal for me. The Surgeons started to prepare for surgery, they covered my arm with a sheet and put it up against my face, which is fine since I was definitely not looking in that direction anyway. I was looking at the wall where there was a clock and a bunch of other medical stuff. I could also see the Anaesthesiologist who would be monitoring me, as well as the heart rate/blood oxygen meters etc. He did speak to me a fair bit to put me at ease, his assistant also talked to me about my hobbies and what food I liked to cook to take my mind of it. The surgery took about 40 minutes. I didn't feel any pain, I only felt some movement. The surreal bits where mostly when they were looking for my nerve tube to sew it back together "Where is it? I've found the bad boy!" and that they were just shooting the shit about life and what they were currently watching on Netflix to each other just like I would with my colleagues. Except they were poking around inside my finger instead of fixing some janky webserver issue. Once they had sewed my nerve tube back together and sewed my finger back up/bandaged it up a lot, I was wheeled to the recovery room. Here I finally got to have some water, as well as just relaxing and lying down whilst they took my vitals until I reached a stage where they where happy to wheel me back to the ward.

It was 2 hours between leaving the ward and returning, so I spent nearly an hour in the recovery ward being monitored. Though it didn't feel like that at all. My arm was very much still numb. The nurse also removed the cannula which was the most painful part of the whole ordeal as it took a bunch of hand hair with it.  I had been told that my arm would be numb for a while, I would be able to move it at the elbow around 9pm at night, but I wouldn't be able to move my fingers until around 10am. Getting dressed with a completely numb arm is interesting. Before I was allowed to leave the hospital I had to drink something, eat something, and go to to the toilet. I had a cup of tea, a cheese and pickle sandwich (clearly cheese sandwiches are linked to me having anaesthetic..) and a pee.  Turns out buttoning up jeans with one working hand is very hard.

Having a fully numb arm from the shoulder down is weird. It's also very heavy and you have to move it around with your other hand. As described, around 9pm at night I could move my arm at my elbow, but when lying down I discovered that I could move my arm towards me but I couldn't move it back as it was too "heavy" but if I was sat up then gravity helped out. I woke up at 9:30am but still couldn't move my fingers. I slept a bit more and then about 11am I could wiggle them bit. In all it took about 30 hours for the numbness to completely wear off.

Then it was just more life with a hand in a bandage, like showering with a gallon ziplock bag over it. I did buy a cheap foam sling on Amazon and it made it more comfy for things like car journeys. It's very hard to do things one handed. Getting dressed is a chore, putting on socks especially so. I had to have my dad tie my shoelaces for me, and I bought my first pair of jogging trousers simply as they don't have buttons. I also had some interesting nerve developments, like when I brushed my arm against a hot radiator in the bathroom and it made my arm spasm like I had been shocked.

After 7 days I had another appointment to get the bandage removed. The nurse cut it off and I took one look at the damage and felt all faint and had to have some water and lie down again. I'm not good with injuries. Normally I would have had some hand therapy to help me heal from it, but due to the cononavirus issue I was given a leaflet, and told probably wouldn't be able to use my hand properly for a few weeks. It's now exactly a month from my surgery and my stitches have all dissolved/come out, I'm just left with a pretty gnarly scar.

I have about 80% movement in my finger compared to before the accident, but the side of it from the cut to the tip is still numb. It will take months for the feeling to come back completely, if it does at all. The nerves regrow at about 1mm per day. For a few weeks after the surgery I would get pins and needles feelings in my finger if I extended my arm too much, and if I poked the top of my palm below my finger, I could feel it in my finger tip.. but the weird nerve shit has mostly subsided now.

If I reach for something at an odd angle, or over exert my finger I get some weird sensations. I also can't yet use my finger like before, It's also very sensitive, which should reduce over time helped by inducing sensations onto it as per the leaflet I was given. Holding things is also interesting with the numb bit as you have lack of feedback. On the plus side we have been in lockdown so I've not needed to drive to do too much.. I can type with more than one hand now though which is handy for work! From what I've read online it can take from 6 to 8 weeks from surgery for it to be healed enough there is no pain, and can do stuff more normally. So that's 2 weeks to a month away for me. I'm very thankful for the great service the NHS provided, they fixed me up in no time from A&E to the specialist to surgery, all within 4 days. I'm definitely more aware of what I'm doing now with sharp things, and my friends are too, as they don't want to "Craig" themselves.

A sneak peak into the inner workings of the mainbox.

I also took the series on a run recently.

Firstly the pinch bolt that holds the selector fork to the shaft for hi/low needs to be removed, I had to tap the spanner with the mallet to free it off before it came undone easily, after that I just had to remove all the nuts holding the housing to the case. They are all whitworth, like everything on the gearbox so far. I should also say, get a set of whitworth spanners and sockets. As it's all you will use.

Once they are all removed (in my case most of the studs came out as well), the housing is just gently prised away from the transfer casing

You can then pull it completely apart, and remove the dog gear for 2wd/4wd selection.

The next step is to remove the ring from the end of the selector shaft that the red levers ball sits in. Though I found that this was on there tight.. so I had to put the casing back together slightly so I could get enough leverage to get it to loosen off. Next time I will make sure I loosen it off whilst the whole thing is still together..  

Once the nut is removed the ring bit just pulls off, you can then remove the cover for the selector rods. The rusty gunk in here takes me back to school and the smell of the trombone I had to learn to play.. I also removed the pivot arm for the 4wd locking pin, it's held in with one bolt.

The ends of the selector shafts are bit rusty. I believe this is a common issue where they rust up due to lack of use and then either get stuck or cause issues trying to change between high/low. I think this is the issue my own box has as sometimes it won't go back into high very well.

They will now mostly pull out of the housing, I did have to tap the end of the threaded one with the mallet slightly to get it to shift. Here they are in all their glory, along with the dog gear for 4wd. After playing with them on the bench I think I get how they work (though I could be wrong..).

When the red lever is forward, the dog is pushed rearward and is disengaged from the teeth on the front shaft (so it's in high range, 2wd). If you press down on the 4wd yellow lever, it pivots allowing the locking pin to come out of the shaft which then moves backwards under tension from the springs, this causes the dog gear to go backwards locking the shafts together. Then when the red lever is moved to low range, if 4wd isn't already engaged it would engage the dog, else it just moves the low range gear into selection in the transfer case. Then when the red lever is returned to the high range location, to moves the low range gear out of selection and locks it against the high gear (to hold high gear to the shaft, as otherwise it can spin freely), and it moves the dog gear forwards to disengage 4wd. As the locking pin is under tension of the spring on it's shaft, when the hole slides under the pin it allows it to drop into it. The two output shafts can spin independently when not locked together due to a bushing in the end of the front housing shaft that the transfer cases output shaft slots into. It all makes more sense when it's in front of you.

Next up is removing the front drive shaft from the housing, this is pretty easy and just requires removing the castle nut from the end of the shaft (I pre loosened this when it was all together), the front drive flange will then slide off and the shaft can be pulled out. The oil seal retainer can also be unbolted to reveal the bearing.

Then I needed to remove the remaining studs from the front of the housing so that it could sit flat for the bearing being pressed out. This was pretty easy with the double nut method. You just need to screw one nut on as far as it will go, then tighten it up slightly, then thread another nut on until it butts up against the lower one. Tighten it up slightly, at the point you should see the stud move. Then undo the stud using a spanner on the bottom nut.

To remove the nuts from the stud, I put it in a vice and then undo them with a spanner. The vice has aluminum sheet in front of the jaws to protect the threads.

The last bit to remove is the speedo housing, this is also fairly simple and just requires removing the nuts and studs like above. I did have a couple that wouldn't budge with the spanner and required using the breaker bar.

With the housing removed, you just need to slide the worm gear off the shaft and also remove the shims that are used to set the preload for the gears.

The last part of the disassembly (that wasn't the main casing) was to press out the seals and bearing. When I bought the press I also bought a set of pressing disks, they vary in size from about 18mm up to 70 something, they worked great for spreading the load of the press to the thing to be pushed out.

First up was the front output oil seal.

Then the rear output oil seal from the speedo housing. I had to use pressing disks in decreasing size to raise the height to where the press would reach it, worked well though.

Lastly was the bearing for the front output shaft. I used a pressing disk and an extension from my socket set for the press to push on to reach it.

The bearing on it's way out

Bearing freedom! It does feel pretty gritty when you turn it by hand, so probably good it's being changed.

Onwards to the main casing, firstly remove the massive circlip that retains the front bearing. This is where buying massive circlip pliers comes in handy.

The book then says to drive the shaft towards the rear until the bearing race is pushed out, I did try with the mallet but after a few hits it wouldn't budge so I took it up the the press in the shed. Pushed it out really easily.

Once the bearing race has fallen out, the case was then flipped over and the shaft pushed as far as it would go the other way. Until the gear butts up against the case.

Then it was back to the garage to prepare to remove the final bearing race. The book says to use a packing piece between the roller bearing and the race. They suggest using an old bearing race cut to size, which isn't useful if you are doing it for the first time. What I've found works (from doing the previous case) is to split the bearing away from the shaft. To do this place a chisel between the bearing inner race and the thrust washer and gently tap with a mallet, this will allow you to then pry the bearing away from the washer.

You can then fit a M10 nut in between the race and the washer. I found it works ok with one, but two would be best if you manage to get them to not fall out. Push the inner race up against the outer and tap the rear shaft with a mallet to wedge the nut tightly between the pieces. Then press the race out.

With the race removed the inner race can be pried off the shaft and then the next challenger is revealed. The circlip that holds the thrust washer in place. Safety squints engaged. You might be lucky and it has both eyes and can be easily removed. I have been unlucky on both occasions. On the first casing both eyes were missing and it took lots of trying to lever it up and away, and it snapping back in place before I was able to finally get it off. This time one eye was missing, so I was able to lever the bit with the eye out of the grove with smaller circlip pliers and then the rest of it using a couple of flat head screwdrivers, it only snapped back in place twice.

With the circlip removed, the thrust washer can be removed and then the shaft can be removed rearwards through the gears.

The last thing to do is to press the rear inner bearing race off the shaft, this is what caused me to buy the press originally. The low gear is used as a block to push the bearing off. You have to hold the shaft as you press, to catch it when the bearing comes off, else it will just drop. It's heavier than you are expecting at the time.

The complete innards reassembled on the bench, as how they would be inside the casings.

And everything packed away ready for the next step, which will probably be degrotting of all the parts.

We are using modpoll as a master for polling the clients (generators), so my next task was to figure out how you get the register address number needed to send to the generator so it could reply with another number that was either the actual value of thing you queried (like fuel level or battery voltage), or a status code (for things like alarms/control state). It turns out that it requires maths, I guess that does make sense since it's for logic boards.. What you have to do is get the "page number" from the list of register tables (which is the more the table number than anything), then times it by 256 and add the register offset from the table.

So to find the control mode register address using the registers above we do the following:

(3 x 256) + 4 = 772

Then when polled it will return one of the mode numbers below

Fairly easy once you get your head around it.

# modpoll -c 1 -m tcp 192.168.0.2 -1 -0 -r 772
modpoll 3.4 - FieldTalk(tm) Modbus(R) Master Simulator
Copyright (c) 2002-2013 proconX Pty Ltd
Visit http://www.modbusdriver.com for Modbus libraries and tools.

Protocol configuration: MODBUS/TCP
Slave configuration...: address = 1, start reference = 772 (PDU), count = 1
Communication.........: 192.168.0.2, port 502, t/o 1.00 s, poll rate 1000 ms
Data type.............: 16-bit register, output (holding) register table

-- Polling slave...
[772]: 1

Next up was trying to find a register for if the engine was running. I did find one for engine status, but even when we started a generator up it would return an invalid mode. I contacted DSE support to query this and they said that since that register is an ECU one, you're not able to query it correctly. What we could do was query for the engine RPM and if it's under a value of something like 600 it's safe to assume that it's not running. This worked really well on the tests we did. I also have to say that DSE technical support is outstanding, considering we haven't paid for anything other than the original panels with the gensets. They replied very promptly to queries, and assisted with troubleshooting issues without complaints!

Next up was the issue with the generators deciding not to talk back to us after a while. The original monitoring scripts were set to query each generator for about 15 different registers every minute, they also did one poll per register. So my guess is that the poor little control panels were effectively being denial of serviced with 15 individual queries to handle every minute. The procedure to reboot the panel so it will talk again is fairly involved as well, you have to open up the control box the panel lives in and unscrew the terminal block screw allowing you to remove the 12v positive wire to the control board, then waiting a few seconds and putting it back in/tightening up the screw. I also worked out that with modpoll you can query a register range, so if you are querying the same registers page you can pull out a number of queries at the same time. This meant that we could cut the 15 queries down to 3; for control mode, engine status, and alarms. We also only really need to check if the engines are running every minute, everything else can be checked every 10 minutes.

# modpoll -c 6 -m tcp 192.168.0.2 -1 -0 -r 48897
modpoll 3.4 - FieldTalk(tm) Modbus(R) Master Simulator
Copyright (c) 2002-2013 proconX Pty Ltd
Visit http://www.modbusdriver.com for Modbus libraries and tools.

Protocol configuration: MODBUS/TCP
Slave configuration...: address = 1, start reference = 48897 (PDU), count = 6
Communication.........: 192.168.0.2, port 502, t/o 1.00 s, poll rate 1000 ms
Data type.............: 16-bit register, output (holding) register table

-- Polling slave...
[48897]: 0
[48898]: 0
[48899]: 0
[48900]: 0
[48901]: 0
[48902]: 0

The scripts themselves were written in PHP which is good as that's my jam. So I was able to easily update how they worked to incorporate the new changes. The main changes were the splitting of the queries for different timed crons, creating a modpoll function to handle it returning more than one value, and to update the existing queries to use the new functions output.

function modpoll_dse($ip,$register_start,$count) {

    $output = shell_exec("modpoll -c " . $count . " -m tcp " . $ip . " -1 -0 -r " . $register_start);

    preg_match_all('/\[[0-9]+\].*/', $output, $matches);

    foreach ($matches[0] as $output) {

        $output = explode(':', $output);
        preg_match_all('/\d+/', trim($output[0]),$register);
        $register = $register[0][0];

        $value = trim($output[1]);

        $register_values[$register] = $value;

    }

    return $register_values;

}

Then using the same values as the modpoll example above;

$modpoll_dse = modpoll_dse($ip,'48897','6');

Array
(
    [48897] => 0
    [48898] => 0
    [48899] => 0
    [48900] => 0
    [48901] => 0
    [48902] => 0
)

The output is stored in a database, which used to get sent to nagios via a passive check but I changed this to push to a dashboard we could put up on a big TV in the office that updates every minute. That way people can see if there is an issue with a simple glance at it for colour changes. The dashboard is using the smashing framework. It's really cool, you can just curl your data to the endpoint you create and it updates. I modified the health widget to handle our alerts, if something goes over a threshold, or a status changes the box for that unit goes red otherwise it's green. There is one health widget for each of the units that is monitored, and a separate update script sets the colours based on what text is pushed to the dashboard. Critical (red) alerts/recoveries are also pushed to a slack channel for the infra team on shift, as well as direct messages for the datacenter manager/escalation points.

Since changing the query timing the generators have behaved themselves and haven't stopped talking to us, the slack notifications have also worked very well for letting people know what's going on.

• 1 large onion diced
• 4-5 large potatoes (or equivalent) cut into chunks
• 3 tbsp paprika
• 1 tsp caraway seeds
• meat or veggie hotdogs chopped (optional)
• salt/pepper
• olive oil
• water
  

Method:

1. Heat a large pan and add a generous amount of  oil
2. Add the caraway seeds and fry for a minute
3. When the seeds are toasty add the onions and fry with the lid on until they are translucent
4. Add the potatoes and fry for a few minutes
5. Remove the pan from the heat and sprinkle the paprika over, then stir in (if you don't remove the pan you risk burning the paprika and ruining the dish)
6. Put it back on the heat, stirring for a minute or two
7. Add enough water to just cover the potatoes
8. Bring to the boil, put the lid back on and simmer covered for 30 minutes
9. Take the lid off and continue to simmer for 20 minutes or until suitably reduced (it should almost be like a chunky soup)
10. If you are adding hotdogs, put them in about 5 minutes before the end to heat up.
11. Season with salt/pepper before serving

Photos:

I'd mentioned possibly buying some cheap car to tide me over to my dad so I didn't have to keep using my mothers car. His friend owns a garage, which had a polo for sale. After a short test drive Tank was mine for the pricely sum of £550. He had some issues, but who cares for that price? The rear brakes were shot and needed replacing/the handbrake reach was high, and the rear wiper/washer didn't work, and he needed front tyres. I got the brakes/handbrake sorted by the garage. That cost me around £200. The front tyres were replaced for around £70. I asked the tyre guy for the cheapest ones as he was just a temp car at the time. I took Tank camping on a farm with my farmer friend, he got muddy.

He was surprisingly good offroad, on the fields and on the bits between fields on which I would be hesitant to drive the skoda on. He even crossed them with the back full of logs and camping stuff, basically on the bump stops. At Christmas camping this year we loaded the back with all the logs and I reversed him down the field, he makes a good work horse.

Photo credit @thewatchingeye0099

With all the commuting to work and other driving I've been getting 55mpg from him which is way more than any of my other cars. The Skoda is around 33mpg and the series... maybe 15 at best? His rear wiper not working cost £15 for a second hand motor from ebay.

The washer was a bit more involved, and required removing about half the engine bay to get to where the pipe had become disconnected due to being crushed and had caused the connector to pop off the pump connection as well, but I managed it in the end. I think I'm about £1200 all in, with initial cost + insurance + tax + repairs, which isn't too bad considering. Though the water temperature gauge has started to play up.. so that's the next thing. Sensor or thermostat..? We will see.. He is 16 years old, teenager issues.

I'm really enjoying driving him so far though, he's definitely a keeper. When the Skoda is eventually fixed I'm going to get rid of it and keep him. Future things to get done are the cambelt/aux belt as I don't know when they were last done. Then whichever issues crop up..

I had been looking into getting the gearbox/transfer case rebuilt by a company but that also runs into issues. The place that people seem to recommend the most isn't currently taking orders and were fairly expensive, and the others are either humongously expensive or have mixed reviews. So I started to look into what it would take to rebuild it myself, and it doesn't look too terible. It's just bearings, seals, and endfloat. How hard can it be? I have the workshop manual and there are some good blogs/videos (1/2) on it from people who have done theirs.

My plan was to buy a second hand box off ebay to rebuild, then swap it with my box once complete. I can then marvel in the delights/horrors that lie within mine when it's on the bench, and still have a working landrover thoughout the whole process. So with this I started to look at ebay every so often for a decent box that was either close or had delivery. Sadly most of the ones listed seem to either be early series 2/2a boxes that don't have full syncromesh or were collection only and the opposite side of the country. Eventually one did come up that was a complete series 3 box and was fairly close for collection. It was up for £270 but I put an offer in for £200 that was accepted.

My friend was super awesome and helped me collect it, he has a defender so perfectly fitting for collecting landrover parts. Here it is strapped down in the back.

Landrover series gearbox/transfer cases are heavy. I knew that it would be fairly hefty but figured it would be manageable with two people, not a struggle. Once home it was placed in the back of the series. I did a lookover of it and discovered that the guy I bought it from did not drain the oil from the boxes as he said he would. This presented an issue as I had to remove the oil before I could split the gearbox from the transfer case but I couldn't lift it up to gain access to the drain plugs as our drive is gravel and sloping, no good for a crane. I managed to tip most of the oil out of the main box from the fill plug. For the transfer case a cheap oil/fluid transfer pump was sourced from Amazon, it's the finest chinesium but doesn't seem too bad for the price. The inspection plates were removed and the hose poked in. The pump was connected up to an old car battery that was lurking in the garage, and slowly but surely it sucked all the oil out.

Once that was done, it was onto the splitting of the boxes. The fairly simple looking instructions..

The first thing to do was to remove all the levers so I could turn the box upside down. This is where I ran into my first fully whitworth fastening of the day (and first of many to come). Fully as in nothing metric or imperial was close to fitting it. It was 1/4 and the bolt/nut was the one holding the red high/low lever in place. In the picture it's in the middle closest to the floor (looking from above).

Also in the picture at the top is the clevis pin that holds the yellow knob lever to the lever on the transfer box. This came undone quite easily with a screwdriver (dispite the rusty look) once the tiny split pin on the other side was removed.

Once the bolt was removed, it was just a case of awkwardly removing the lever downwards, with some tipping of the box require due to it being in the back of the series. I imagine it's much easier if the box is still in place. Making sure I didn't lose the little spring clip that locates the ball into the pivot hole.

The main gear lever was also unbolted.

All small bits were placed into labeled bags.

The next stage was to remove the transmission brake drum, so I could get to the nuts that hold the drum/brake plate to the transfer case. I spent a bit of time trying to figure this out as I removed all the small nuts that surrounded the front of the drum and it still wouldn't come off, at this point I decided to consult the manual and it turns out you only need to remove the large 1" castellated nut (with split pin) on the output shaft.. (doh!).

In the picture you can see part of the brake actuator mechanism sitting on the bottom brake shoe instead of in it's rightful place. In the manual it says to remove the hand brake linkage however it was rusted up solid so the backing plate was unbolted with it still attached and all fastenings and small parts were placed into a labeled bag. The big parts were placed into a handy cardboard box for later overhauling.

What we are left with

Next up was to remove the outer fixings that hold the transfer case to the main box. There are five and they are all whitworth. The top most two are 1/4 and the rest 5/16. The two top ones were very rusty and not very well lubricated so I had to use the breaker bar with a spanner braced against the casing to undo the nuts.

Of the remaning three, two were captive studs and one was a bolt and nut. They came undone with relative ease due to being covered with gearbox oil/oily grot for years, I put the spanner on them and tapped it with the rubber mallet to free them off, then undid them with the socket and ratchet. There is a small bolt sitting in a recess that I'm not entirely sure what it's for, but it's removal was not needed. I'm sure I will find out it's purpose in due course (some kind of stop for the gear selectors?). To the right of it you can see one of the 5/16 nuts to remove.

It was now time to flip the box upside down so I could access the transfer case cover. I decided to put some cardboard down instead of the bubble wrap it had been sitting on before (to stop grot getting on the carpet too much). It was at this point I discovered that when you turn the box upside down any remaining oil in the gearbox will flow out of the holes that the gear change lever bracket bolts into.

Now that the box was upside down and my rear tow ball had been given some some extra rust prevention, it was time to remove the transfer case cover in preparation of removing the intermediate gear. The nuts are 7/16 and I had encounted them before on my own orginal box when I replaced the leaky cover. I found that most of the studs came out instead of the nuts coming off. There was only one where the stud remained. I will have to see if these should be loctited in on reassembly.

The gears look in pretty good nick, and all spin nicely. Though I will still replace the bearings when I get a overhaul kit. Next up is removing the rear output shaft bearing cover (to the right in the above picture), this just involves removing the nuts (or studs if they come undone) and then gently levering the cover away from the case. I accidently neglected to take pictures of this.

To remove the shaft that the intermediate gear sits on you need to remove another 5/16W locking nut and then use "Service Tool 605862" to remove the shaft. I don't have a service tool but I do have a crowbar that serves the same purpose. No great force is required to remove it, just lever it fowards then it will come out by hand. Making sure you support the gear with your other hand.

The gear can then be rolled out of the case and it turns out there are two needle roller bearings inside it, that the shaft goes though. As they fell out when I lifted it out.

With the gear removed it's just a case of removing the last 3 nuts that hold the cases together. I put a socket on them and then gently tapped the ratchet with the rubber mallet to free them off, then used just the socket in my fingers to spin the nuts off as they are recessed. Thankfully as they live in an oil bath they were very easy to undo.

Lastly it's just a case of gently tapping the cases apart with the rubber mallet to get them moving. Then carefully pulling the gearbox output gear though the hole in the transfer case.

Once separated, the transfer case is fairly easy to move. I was able to lift/carry it to the garage. The main gearbox is an absolute unit in comparison (small but dense) and will require assistance to move.

My next step is to clean up the parts removed that are in the cardboard box, so they can be stored. Then I will teardown the transfer case and clean the casing. The mainbox will also be moved to the garage.

Overview

My daily driver is a Peugeot 1007. It's pretty nifty and unique which is one of the reasons I love it so much, However it's illumination is somewhat lacking, even with the full beams on and with decent bulbs. So I came up with the idea of adding some driving lights (mostly for countrylanes). Initially it was just a pipe dream, but recently I was browsing eBay and I bought a cheap china LED bar.. which made the project a reality.

Plan

My overall idea was for the light to only be able to be used with the full beams on, and then only if it was enabled by a switch. This way it's legal and can't accidently be turned on with the normal dipped headlights. After discussing the plan with my friend he kindly made a wiring diagram for me (as this is my first big electrics project).

With the headlights on I used a multimeter to determine which of the cables on the bulb socket was for the high beams. This is the cable that would be tapped to power the coil side of the relay. I also decided that the light bar will be fixed to the roof with some hench magnets, this is less destructive than modifying the front bumper/grill for a mount and will allow for it to be easily removed if it needs to be.

With the diagram and the plan, it was time to assemble the components.

Bits and bobs used.

Links where available.

Parts

• LED Light bar
• Relay
• Spade connectors
• Fuse piggyback connector
• Positaps
• Switch
• 2mm wire (5m red/5m black)
• Various sized heat shrink tubing.
• Misc crimp connectors
• Car roof magnets (102mm x 127mm x 12mm)

Tools

• Wire stripper/crimper
• Pliers
• T20 Torx screwdriver
• Multimeter
• Jet lighter for the heat shrink
• 10mm spanner
• Drill with 10mm bit

Pictures of the install

Constructing the negative earth cable.

Negative earth cable secured to engine bolt (10mm spanner)

Relay connected up

Piggyback fuse connector in place for the power feed for the light bar

Covers back on

High beam wire tapped

Feeding wires through the firewall for the switch in the cabin

Wires cabin side

Drilling hole in the dash panel for the switch

I discovered that the 10mm hole was too small for the switch thread, so I had to ream the hole out with a 12mm cold chisel.

A piece of pugly had to be sacrificed to allow the switch to fit.

Feeding the switch wires through.

Spade connectors added and wires heatshrinked.

Switch wired up.

Dash panel refitted.

Light bar fitted to roof, it's virtually impossible to move the bar with the magnets. They are very strong.

Wire ran down side of windscreen (under the body panel).

Front view of the completed install.

Video

Video of it in action.

Todos

• Better insulate relay connections, to prevent shorts. At the moment I have just added some loose heatshrink tubing.

The generate script first parses the localhost.cfg into a PHP array with a sub array for each host like so:

It then loops through the array of hosts to get the service checks specified for each host from the nagios config. The results are then constructed into a new array for each host which is then json_encoded() and set in redis with the hosts IP address as the key.

Full script is:

The generate script is ran via cron every so often, however I have also modified our in-house nagios editor so when a host is added/edited it updates the array stored in redis for the IP, and if you delete a host it removes it from redis.

The lookup endpoint is then just a simple PHP script (it sits behind a firewall for security) that either returns a json array of all IPs in nagios, or if an IP is specified the details for that host in a json array.

The way the check works is that it gets a list of all the regions in the table and then the start/end key, the name, and the location of each region. It then queries the API with '$table . "/" . $key' to see if it returns a result, if it doesn't it adds to to an array for nagios. Once it has finished checking all the regions the script looks at the nagios array, if it's not empty it will echo out all the missing regions and exit 2 (Critical). Else it will say everything is OK and exit with 0.

As the check takes a while to run for the table being checked the script uses file locking to ensure it only runs one at a time. If nagios tries to run it while it's running it will return message saying it's already running and exit 0.

The full script is:

I was worried that it wouldn't balance the load as well as the generation script, but it appears to do it perfectly fine. The network graphs for the puppet masters are nice and steady, no spikes!

This can been be seen in the network graph for one of the puppetmasters.

Every 30 minutes you have around 900 servers (half the load) requesting its catalog. That many servers requesting at once made for very unhappy masters, and the clients were getting 503s for high load. So we had servers failing runs (which is never awesome).

Despite puppet being used by lots of large enterprises, I couldn't find many writeups about dealing with your puppet agents bunching together and how to restagger them, the majority of posts seem to be related to scaling the masters.

However I did stumble across this handy post from 2012 about staggering the puppet runs with cron - http://www.krzywanski.net/archives/968

I decided to give it a whirl and after updating the script to work with the new CA command I have to say it seems to be working pretty fabulously, though the fix has only been implemented for a few hours so far.

You can see the different in network usage after all the agents were changed to cron.

And in the 48 hour graph you can see that the load is much more balanced now.

After adding the scripts from the previous link to the masters (with puppet of course!), I created the following mini class to apply the changes to the agents:

After testing this was added to the top level class for all our nodes, and it rolled out without any major issues. The only issue I encountered was due to facter returning nil for $::fqdn if the node didn't have a domain set as well as a hostname. This was fixed by editing the fqdn.rb on the node to return the hostname instead of nil, rather hacky.. but it works without needing to change the hostname of a server in production! (This issue appears to be fixed in the later version of facter)

Future plans/ideas/thoughts:

Look at random cron timings with fqdn_rand() at somepoint instead of the node list script, as the number of nodes managed by puppet increases steadily each week and the list is currently regenerated everytime a node is added.

Even though the load has been balanced the masters do still sit with a small amount of load, so adding another master to the haproxy pool would help a lot.

We also started using puppetdb for fact collection (and foreman for reporting, but that is on another server). This setup worked fine for just over a year (system_birth => Thu Jan 23 13:15:23 2014), but again it started struggling under the load of managed hosts (well over 2000 currently and increasing at a steady rate). The load would spike and we would get batches of failed runs due to timeouts.

A plan was made to investigate and setup a loadbalaced multimaster environment for better load management. Two Dell R630's with 32 cores, 128GB RAM and 2x480GB SSDs in RAID1 were provisioned. Plenty of performance for our puppets. The plan was to setup a dedicated haproxy box for the loadbalancer and to use puppet to update the server setting on each node to point to this as the master to use.

However that's not what happened, instead we had what I will affectionately call "puppet failure friday". I'm not entirely sure what happened, since I was asleep at the time.. but it looks like the server swapped, apache hit max clients and then couldn't keep up with the resulting load which climbed rapidly and resulted in puppet agents failing runs due to timeouts or runs stagnating. This is shown in the below diagram.

This therefore meant the multimaster plan was expedited somewhat, as I couldn't get the current master to handle the load at all. Having your puppet master sit at a load average of 200 isn't particularly great for having it do anything useful!

As I couldn't use puppet to set the server on each agent to a new one, I used the existing puppetmaster as the load balancer. We run puppet under apache so I changed the vhost on the existing master to listen on 8150 instead of 8140, haproxy now listens on 8140. The new masters were setup as per the puppetlabs documentation (which is generally awesome, as documentation goes), with alt_dns_names configured.

My haproxy config is:

The newer servers have a higher weight than the existing master, as they are way more awesome in terms of specs. The existing master also runs puppetdb.

The haproxy stats looks like:

The existing master is also the ca master, so the other masters are set to forward any certificate requests to it (on port 8150).

The new masters have also been configured to talk to puppetdb and send reports to theforeman. As we use svn for our manifests and modules, there is currently a cron to rsync changes to the two other masters when the existing master is updated. They will be set to do a svn up everytime their puppet agent runs (like the existing master) and added to our chatops ability to run svn up (modules|manifests) on the master(s) from our internal chatserver.

The setup is handling the current load very well (as one would hope!) and manages fine if you remove a server from the haproxy pool. The only issue is that if you restart httpd on any of the servers (or haproxy itself) you get a handful of servers failing runs due to timouts in the ~10 seconds it takes it to restart!

Future plans: Add another server to the pool to take over the master duties of the existing master (leaving it to just do puppetdb/ca). Possibly move haproxy to it's own server, though that would be another adventure.