A solution for Puppet hiera yaml sprawl

A solution for Puppet hiera yaml sprawl hiera is a very powerful way to separate code and data in Puppet code. But sometimes, you need more than the standard functionality for levels that hiera provides. One way to keep configuration data from being duplicated over multiple levels would be to include other yaml files. Unfortunately, yaml doesn’t have any means to do this. That is why we built an extended yaml hiera backend that allows you to do this.

A use case for yaml include

One of the use cases that we run into regularly with our clients, is doing version upgrades of for example Oracle software. We have one role yaml that contains all of the hiera data needed to ensure a correct Oracle database is built and configured by Puppet. Here is an excerpt of such a yaml file:

ora_profile::database::dbname:                  MYDB
ora_profile::database::version:                 19.0.0.0
ora_profile::database::db_patches::patch_level: JUL2021RU
ora_profile::database::oracle_home:             /u01/app/oracle/product/19.0.0.0/db_home1
ora_profile::database::db_software::file_name:  LINUX.X64_193000_db_home
ora_profile::database::db_software::dirs:
- /u01/app/oracle/product
- /u01/app/oracle/product/DB19"

As you can see, this yaml file contains some references to the version used. In this case, Oracle 19.

When we want to start using Oracle 21, however, we want to do this gradually. So first on one development machine and then on more or even on all development machines and then through testing and acceptance finally to production. One of the ways to do this is to add the updated hiera data to the node-specific hiera levels. This allows you to start rolling out Oracle 21 controlled on a node-by-node basis. This, however leads to a sprawl of hiera entries throughout your yaml files. Wouldn’t it be great if you could include a file containing all entries for a specific version? Yes, you can!!

How does it look?

Here is how this looks in a yaml file:

---
some::other::data:  yes
...
# @include '../includes/oracle_21.yaml'
#
other::entry::in::your:yaml: 	just_some value
...

This little snippet will include the file oracle_21.yaml from the directory includes below the directory of the current yaml file into your current yaml file.

The include process itself replaces the include line with the content of the included files and then passes it to the Ruby yaml processor as one entity.

But why is it a comment (e.g. behind a #)? The reson for this, is that many Puppet CI tools check the validity of yaml files. If we would have put the include as a “normal” stratement, all of the CI checks for yaml will fail. That is not what we want.

How do we enable this?

The hiera backend is called easy_type::yaml_with_include and as the name implies, it is part of the easy_type module. The enterprisemodules-easy_type module is a collection of functions, custom types and other Puppet goodies we use for all of our commercial modules. You are free to install and use it. Check the documentation on the Puppet forge on how to install this module into your Puppet code base.

After the installation, we have to configure hiera to use it. The easiest way to do this is to replace the defaults in your hiera.yaml file. We need to change the value for the data_hash key from yaml_data to easy_type::yaml_with_include. Here is an example of how your hiera.yaml could look.

---
version: 5
defaults:  # Used for any hierarchy level that omits these keys.
  datadir: hieradata                          # This path is relative to hiera.yaml's directory.
  data_hash: easy_type::yaml_with_include

Let’s try it.

Our role yaml now contains the include statement # @include '../includes/oracle_21.yaml'. Our oracle_21.yaml` file looks like this:

ora_profile::database::version:                  21.0.0.0
ora_profile::database::db_patches::patch_level:  JUL2021RU
ora_profile::database::oracle_home:              /u01/app/oracle/product/21.0.0.0/db_home1
ora_profile::database::db_software::file_name:   LINUX.X64_213000_db_home
ora_profile::database::db_software::dirs:
- /u01/app/oracle/product
- /u01/app/oracle/product/DB21"

The best way to check if it works and also to debug lookups, is to use the puppet lookup utility. Let’s use this utility to lookup the version of Oracle.

$ puppet lookup ora_profile::database::version
--- 21.0.0.0

And it works. But it would be nice to get a bit more details. Fortunately, the puppet lookup utility has a way of explaining what it does. Let’s try again and use the --explain option:

$ puppet lookup ora_profile::database::version --explain

...

  Environment Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
    Merge strategy hash
      Hierarchy entry "Per-node data"
        Merge strategy hash
          Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.example.com.yaml"
            Original path: "nodes/%{trusted.certname}.yaml"
            No such key: "lookup_options"
            Including file '/etc/puppetlabs/code/environments/production/hieradata/includes/oracle_21.yaml' into contents of '/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.example.com.yaml'
          Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.yaml"
            Original path: "nodes/%{hostname}.yaml"
            Path not found

...

Searching for "ora_profile::database::version"
  Global Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
    No such key: "ora_profile::database::version"
  Environment Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
    Hierarchy entry "Per-node data"
      Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.example.com.yaml"
        Original path: "nodes/%{trusted.certname}.yaml"
        Found key: "ora_profile::database::version" value: "21.0.0.0"

As you can see by the line /etc/puppetlabs/code/environments/production/hieradata/includes/oracle_21.yaml' into contents of '/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.example.com.yaml' hiera tells you it is including the file. It only notifies you once. Because for performance reasons, it only reads the files one time. Every additional hiera lookup will use the cached data and not know anything about the inclusion of the file.

Conclusion

When sprawl of hiera data over your many hiera levels is an issue for you, we have a way to help you. The yaml_with_include hiera backend allows you to use include statements in your hiera yaml data. We hope this will help you. If you could use a hand, we are here to help. Making good Puppet code, is our bread and butter at Enterprise Modules. But besides developing our own modules, we are also helping customers build the best possible Puppet code. Do you think you could need some assistance? Don’t hesitate to contact us at info@enterprisemodules.com or by phone: +31 (0)30-601 6000 for some consultancy.

About us

Enterprise modules is the leading developer of enterprise-ready puppet modules for Oracle databases,Oracle WebLogic, and IBM MQ software. Our puppet modules help sysadmins and DBAs to automate the installation, configuration, and management of their databases and application server systems. These modules allow them to make managed, consistent, repeatable, and fast changes to their infrastructure and automatically enforce the consistency.