Puppet 3.8.x on CentOS 8

by Craig Parker

Yes, I know that puppet 3.8 belongs in a museum, but it's the version we are still running at $work, mostly out of tech debt but also y'know it just works good and it's stable in our multi-master setup with thousands of clients. It does create an issue every so often though, like when we want to start experimenting with CentOS 8 for internal system servers and we can't install our puppet officially because it's too old. We are moving towards updating to latest but we want to be able to manage the new servers in the meantime. It also means that we can work on making sure all our manifests work with versions greater than CentOS 7.

For installing the actual puppet packages we had to build custom RPMs, my colleague did that bit. He is the self titled bash wizard, but in this case I think RPM building wizard also works. For wrangling old stuff to work on new..  We did have a bit of back and forth on the building, puppet and related packages (ruby, facter, etc) all worked fine, but when trying to run puppet for the first time it threw SSL errors, even though puppet had been built against the latest openssl 1.0.2 package.

[root@example]# rstpuppet 
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for example
Info: Applying configuration version '1592485375'
Error: /Stage[main]/monitoring::Plugins/File[/usr/lib64/nagios/plugins]: Failed to generate additional resources using 'eval_generate': SSL_read:: shutdown while in init
Notice: Finished catalog run in 10.01 seconds
Error: Could not send report: SSL_write:: shutdown while in init

When investigating I found this, which points to the issue being introduced in the the 1.0.2f version. The puppet openssl package was rebuilt for version 1.0.2e, and then we were in business. Mostly. Installing required disabling all the repos and only enabling our own puppet one, and then adding the additional openssl package to the yum/dnf exclude.

yum -y install --disablerepo=* --enablerepo=puppet compat-openssl10 puppet

So it was installed, and could connect to the masters okay but it was having issues installing packages, as well as making sure the latest versions of packages were installed. It was also failing on service ensures as it was trying to use chkconfig instead of systemctl.

For the service ensures, I had a dig through the providers and interestingly found that the systemd one actually has this in it;

defaultfor :osfamily => :redhat, :operatingsystemmajrelease => "7"

but it seems to loop though the providers alphabetically and therefore hits chkconfig first, which defaults for redhat. I fixed this issue with a symlink.

ln -s /usr/lib/ruby/1.8/puppet/provider/service/systemd.rb /usr/lib/ruby/1.8/puppet/provider/service/a-systemd.rb

Making systemd alphabetically first..

The next issue was with package installs, it was trying to use pip instead of yum/dnf, I tracked this down to the fact that it couldn't find a python version to use (even though dnf/yum worked fine, so was going to the next thing it thought it should use). This was fixed by first installing the base python3 package, and then as it still doesn't have just python in the path that puppet wants.. more symlinking.

yum install python3
ln -s /usr/bin/python3 /usr/bin/python

So that leaves the remaining issue, that it can't ensure latest packages, as it can't list the versions. In version 3.8 of puppet, it uses a script called yumhelper.py to check for updates for packages. I believe they have changed this in the later versions, but this obviously doesn't help us now. I modified the script to work with Python3, instead of 2 and removed the bits that wouldn't work as it was trying to import the yum python library but this seems to no longer exist in CentOS8, so it uses shell commands instead.

The modified script can be found here. My favourite part of the existing comments on the script is the bit that says;

# Try to use the yum libraries by default, but shell out to the yum executable
# if they are not present (i.e. yum <= 2.0). This is only required for RHEL3
# and earlier that do not support later versions of Yum. Once RHEL3 is EOL,
# shell_out() and related code can be removed.

Good thing they didn't remove it once RHEL3 was end of life... as it sure was handy for RHEL8.

And finally a happy server that could complete it's manifests without errors. 

[root@example ~]# cat /etc/redhat-release 
CentOS Linux release 8.2.2004 (Core) 
[root@example ~]# puppet --version
3.8.7
[root@example ~]# rstpuppet 
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for example
Info: Applying configuration version '1594945834'
Notice: Finished catalog run in 13.02 seconds