Dynamic but versioned hiera data with git

Dynamic but versioned hiera data with git For some organizations, having dynamic hiera data can be a real time saver. Add a small change to your hiera data; there is no need for a Puppet redeploys, and off you go. Although this is fast, it has some potential downsides too. You cannot see who did this change and why and when it was done. Fortunately, there is a way around some of these downsides.

There is a module for that

On the puppetforge there is a module called hiera_http that can help us implement this. This module implements a hiera backend that can connect to any http endpoint. It can read yaml and json data. You can tell the hiera backend for what hiera keys this backend is used.

Our use case

Our use case for this solution is the definition of available patches. These patches are currently defined in the module hiera data of our module ora_profile. So whenever a new patch level comes available, we update the hiera data in this module and publish a new version of the module for our customers to use. However, some customers don’t want to update this module so often. A new version of a module might not only contain updated data, but it might also contain incompatible Puppet code and at least require organizations to do some testing. So for these customers, we would like to allow them to only use the updated patch-levels in the hiera data but still keep on using the version of the module they currently have. They could, of course, copy the definitions of the patches to their own hiera files. But this means some extra maintenance burden on your internal team. The solution we have in mind is to allow customers to use the latest version of the hiera data without the need to install the newest version.

How to make it work

Let’s first make sure that the module and all its prerequisites are installed. As the documentation states, the lookup_http gem must be installed. Here is the command to do this for your puppet server.

puppetserver gem install lookup_http

When you are using the puppet lookup feature, you’ll also need to install this gem for the ruby context outside of the puppetserver.

/opt/puppetlabs/puppet/bin/gem install lookup_http

Next up is the installation of the module itself. You can do this manual with this command:

puppet module install crayfishx/hiera_http

A better way is to use a Puppetfile and add this line to it:

mod 'crayfishx-hiera_http', 'x.x.x'

and then use the puppet deployments to ensure

What data to use

Now the essential tasks of installation are done, let’s focus on the configuration. We need to tell the hiera backend that it must use a github url to fetch the data. Here is part of the hiera.yaml that defines the hiera search levels.

hierarchy:
 …. 
 - name: "Dynamic patches lookup"
    lookup_key: hiera_http
   uris:
     - https://raw.githubusercontent.com/enterprisemodules/ora_profile/master/data/defaults.yaml
   options:
     output: yaml
     use_ssl: true
     confine_to_keys: 
        - ora_profile::database::patch_levels
 

Let me take you through some of the options:

  • uris: This is the url where the backend will fetch the data. You can get this url by going to the repository in github and open the file you want to use. Then select the raw button, and github will take you to a raw version of this data file. You can copy the url and put it in your hiera.yaml. For our use case, we use the main branch. But you can also select a specific branch, tag, or even commit.
  • output: In our use-case, the file we selected is a yaml file, so we want the backend to interpret the data as yaml data.
  • use_ssl: Although the backend should be able to detect itself if this is a https or http url, we need to set the value to true when using a https url,
  • confine_to_keys: We only want to enable the patch level through this mechanism in our use-case. So we specify only one key, namely the key ora_profile::database::patch_levels. All other hiera keys will be resolved through the other hiera levels and, in this case, be resolved from the module data of the current version of the installed module. You can, however, add multiple keys, and you can also use regular expressions like for example : "application.*"

test it

Now everything should be set up correctly, and we are ready to test. You can, of course, just run the Puppet agent and see if it works, but it can be difficult to distinguish between already available hiera-data in the module and the dynamic data from git. Therefore I’d like to use the puppet lookup --explain command. This command does the lookup and explains what actions hiera takes on each hiera level.

$ puppet lookup ora_profile::database::patch_levels  --explain
/etc/puppetlabs/code/environments/production/modules/hiera_http/lib/puppet/functions/hiera_http.rb:105: warning: URI.escape is obsolete
Searching for "lookup_options"
  Global Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
...[some output deleted]
       Hierarchy entry "Dynamic patches lookup"
          URI "https://raw.githubusercontent.com/enterprisemodules/ora_profile/master/data/defaults.yaml"
            Original uri: "https://raw.githubusercontent.com/enterprisemodules/ora_profile/master/data/defaults.yaml"
            Found key: "ora_profile::database::patch_levels" value: {
...[some output deleted]
...

As you can see, the Hierarchy entry "Dynamic patches lookup" returned the data, so it works.

Gotchas

In this use case, we only fetch reference data. This means that any change to this data doesn’t directly change the Puppet catalog. It allows you to use a different patch level in your manifest. Only after you change the patch level in your Puppet manifest for this node, the patch level will be applied. We like this because it means your system can only change after a Puppet deploy, and not all of a sudden show changes. For some use cases, however, this is precisely what people like and need.

Your hiera data can contain lookups in its data. For example:

my_key: "%{lookup('other_key'}"

When your hiera data contains this, beware that the other_key resolves from the same source version. Else you can get very nasty and brutal to find errors.

Conclusion

In this blog post, we have shown you how to add dynamic hiera data to your Puppet environment. Dynamic but still versioned, so you are still in control. Your use case might be slightly different, so if you need some help, don’t hesitate to contact us at info@enterprisemodules.com or by phone: +31 (0)30-601 6000 for some consultancy.

About us

Enterprise modules is the leading developer of enterprise-ready puppet modules for Oracle databases,Oracle WebLogic, and IBM MQ software. Our puppet modules help sysadmins and DBAs to automate the installation, configuration, and management of their databases and application server systems. These modules allow them to make managed, consistent, repeatable, and fast changes to their infrastructure and automatically enforce the consistency. We are a proud member of the Conclusion family.

Conclusion is thé multidisciplinary service provider in the field of Business Transformation and IT Services. Our tagline? Business Done Differently. Our 1250 specialists and professionals live up to that every day by truly combining our IT knowledge with business and domain know-how. With dedication, creativity, and flexibility, we take responsibility for the social and mission-critical business processes and systems of our customers and enable organizations to digitally transform their business model. Our primary focus is on the Dutch market and more specifically on the domains of Public Transport, Healthcare, Finance, and Industry. Conclusion. Business Done Differently