As an update to the previous post I'm glad to say that our puppet setup is working pretty well! The only issue I've had is that the cron generation script we were using started to break crontab on servers. This seems to be due to it creating blank entries when generating the cron times for a FQDN, which wasn't super awesome.. I decided to look into using the fqdn_rand() function that is built into puppet to generate random cron timings based on the FQDN of the server. The new cron class looks like:

I was worried that it wouldn't balance the load as well as the generation script, but it appears to do it perfectly fine. The network graphs for the puppet masters are nice and steady, no spikes!