Add Steroids to your Hiera

We love Hiera. When writing infrastructure as code, Hiera allows us to separate the what from the who. Separate the settings from the way the settings are applied. The default language for Hiera is YAML. YAML is a simple way to describe the data. But when your configuration grows, you start to notice some troubles. Your YAML files start to become bigger and bigger and slowly but surely become incomprehensible. You would like to reference other values. YAML supports this….. but not between different files. And the * and & syntax YAML uses for defines and references, becomes a hassle when you use it much. In this blog post we are going to show you Connect. Connect add’s steroids to your Hiera and allows you much more flexibility than YAML.

Usefulness of Hiera

In May, the Puppetlabs blog contained a story by Reid Vandewiele, Separation of Concerns: How Hiera Helps. In this article, Reid makes a case for using Hiera:

When you start small, this is nice, but when you have a large Puppet configuration, reusing the code and only changing the data, becomes a requirement. So you start adding data to your YAML files.

Then Problems Arise

But when your infrastructure grows, your YAML files also start to grow. And slowly but surely, you see some problems:

• Your YAML files become big. So big that it is difficult to keep seeing the big picture.
• You would like to define a value once and reference it. YAML has a mechanism for that, but when your files grow, this becomes verbose and cumbersome.
• You would like to reference a value defined at another Hiera level (e.g. an other YAML file), but YAML referencing is not possible between files.
• You use Hiera standard interpolation. But in large files repeating the lookup or hiera commands in your files becomes very noisy.

Connect helps out.

Connect is a hiera backend. This means, at the Puppet level, everything stays the same. Connect only adds features at the level where we are specifying the files. It is a full replacement for the YAML files. Let’s see how Connect can help you solve these problems. First let’s look at some basics. Let’s assume we have this Puppet code:

class ntp (
  $server,
  $config',
  $driftfile,
  $autoupdate,
) {

  ...

}

This Puppet class has four parameters: $server, $config, $driftfile and $autoupdate. In Connect you would specify the values as followed:

ntp::server     = '8.8.8.8'
ntp::config     = '/etc/ntp.conf'
ntp::driftfile  = '/var/lib/ntp/drift'
ntp::autoupdate = true

Easy. But not that much better than YAML. Let’s see if we can make it make it a bit more concise:

with ntp:: do
  server      = '8.8.8.8'
  config      = '/etc/ntp.conf'
  driftfile   = '/var/lib/ntp/drift'
  autoupdate  = true
end

That is already a bit better. For this Puppet class, it doesn’t make too much of a difference, but you can imagine what a difference this makes when you are using classes with a lot of parameters.

Connecting variables

Let’s say for example that we have one server that does the DNS, NTP and LDAP serving. It would be nice if we could set the value of the server once and use it in all definitions. You can:

server =  '8.8.8.8'

with ntp:: do
  server = server
  ...
end

with dns:: do
  server = server
  ...
end

with ldap:: do
  server = server
  ...
end

In this example, we set the value of server once to 8.8.8.8 and then reference this variable for all types of servers.

Including files

If we would continue like this, very soon we would get big files, Just like when we were using YAML. Fortunately, Connect supports the include statement.

server =  '8.8.8.8'

include 'settings/ntp'
include 'settings/dns'
include 'settings/ldap'

The included files stay the same. So you can still reference the server variable in the included file.

String interpolation

Connect can interpolate strings in double quotes. Strings using single quotes are not interpolated. The Connect interpolator knows about its own variables, and it knows about Puppet variables. Let’s first interpolate some Connect variables.

my_name   = 'John'
greetings = "Hello ${my_name}"   # Will be "Hello John"
strange   = 'Hello ${my_name}'   # Will be "Hello ${my_name}"

Connect also knows how to interpolate Puppet variables:

welcome_text = "Welcome on host %{::hostname}"
                              # Will be: "Welcome on host host1"

Data types

Until now we used simple strings. But Connect supports all data types available in Ruby. Let’s see some examples:

To construct an array in Connect, use the [ and ].

my_integer_array = [1,2,3,4,5]
my_string_array  = ['a','b','c']

Because underlying type system is based on Ruby, you can mix data types in an array.

my_mixed_array = [1, 'a', 2, 'b']

You can use a trailing , in arrays. You can also write the assignment over multiple lines.

array_with_trailing_comma = [
  1,
  2,
]

This makes adding entries much less error prone. Just always add a comma at the end. Arrays can also contain references:

value = 10
array_with_connections = [
  value,
  20,
] # will result in [10,20]

To construct a Hash in Connect, you can use either { and }, or do and end. Hash entries must be separated with a ,.

# A Hash using { and }
my_hash = {
  a: 10,
  b: 'a string'
} # This translates in to Ruby {'a' => 10, 'b' => 'a string'}

# A Hash using do and end
my_hash = do
  a: 10,
  b: 20
end # This translates in to Ruby {'a' => 10, 'b' => 'a string'}

Within a hash pair, you can use colon’s (e.g. :) or the traditional ruby hash rocket (e.g. =>). You kan choose whatever suits you best.

# A Hash using hash rockets
my_hash = {
  a => 10,
  b => 'a string'
} # This translates in to Ruby {'a' => 10, 'b' => 'a string'}

Just like in Arrays, you can use trailing , if you like.

my_hash_with_trailing_comma = {
  a => 10,
  b => 'a string',
}

Just like in array’s, hashes can reference other variables.

my_value = 10
my_hash_with_reference = {
  a => my_value,
  b => 'a string',
}

Conclusion

Connect has a lot of constructs to make your Puppet data more concise and better structured. That is why we, especially when building large infrastructures, prefer Connect over YAML any day.

In our reference inplementatation you can see how we use it together with our Puppet Oracle modules and with our Puppet WebLogic modules, to install and configure Oracle and WebLogic infrastructure.

But what you have seen about Connect isn’t all. In our next blog post we are going to discuss Connect Objects and imports. Connect objects together with the Puppet create_resources function make a very strong couple. And import allows connect to interact with all of the other available data provides in your infrastructure. Data providers like for example PuppetDb and Consul. These features of Connect, allow you even more flexibility and are described in an other blog post.

If you would like to have more information about the Connect language, checkout the Connect Language, in a Nutshell.