Even more Steroids for your Hiera

Even more Steroids for your Hiera The Puppet create_resources function allows you to easy create a lot of objects. It is often used together with hiera lookups. But filling your YAML files, with resources, makes them big and hard to manage. In the previous blog post, we introduced Connect and showed you how Connect makes it easy to manage your hiera configuration. In this blog post we are going to show you even more powerful Connect stuff.

Objects

The Puppet create_resources function allows you to easy create a lot of objects. Connect makes this even easier. In Connect you can name these resources and reference them. Let’s take a look at an example:

myhost('oradb.example.com')  { ip: '10.10.10.20', host_aliases: ['oradb'] }
myhost('wls1.example.com')   { ip: '10.10.10.30', host_aliases: ['wls1'] }
myhost('wls2.example.com')   { ip: '10.10.10.31', host_aliases: ['wls2'] }

In here we define three myhosts objects. Every myhost has a name, an ip address and a set of host_aliases. These are all properties the host type recognizes. In this example, myhost is the type of the object. You can use any name you like.

Now we assign these myhosts to a variable profile::base::hosts::list. This becomes a big Hash with all hosts in it.

profile::base::hosts::list = {
  myhost('oradb.example.com').to_resource('host'),
  myhost('wls1.example.com').to_resource('host'),
  myhost('wls2.example.com').to_resource('host'),
}

We can use this in Puppet:

profile::base::hosts( $list)
{
  create_resources('host', $list)
...
}

Selecting parts of an object

After you have defined an object you can reuse it’s values in all sorts of ways. Let’s say we need the IP address of the database server for the definition of a connection:

weblogic::datasource::database_ip = myhost('oradb.example.com').ip

Using the . and [x] syntax we can select part of an object. This selecting also works on all other data types. Let’s see a contrived example:

server = all_domains[0].databases[1].ip

This would look into the array all_domains and get the first element. Of the first domain, it would then select the second database and return its ip address. The combination of file inclusion, objects and selectors, gives you great power. You can, for instance create a file containing all objects in your multinode configuration, and includ this in all config files for a specific node. To connect the values of the objects to multiple resource definitions, you can use selectors.

More about selectors

Selectors are passed to the underlying ruby system. So you can use any method ruby supports on the specified data type. Connect allows you to write selectors like this:

array  = [1,2,3,4,5]
string = array.join(',')   # "1,2,3,4,5" 
hostname = 'DMACHINE1'     # Development machine 1
type     = hostname[0,1]   # type is 'O'
host     = hostname[1..-1] # host is MACHINE1

You can also use selectors when interpolating strings.

presidents = ['Clinton', 'Bush', 'Obama']
last_president = "The last President of the USA was #{presidents.last}"

Because interpolation only works on Connect variables, using selectors is limited to interpolating Connect variables. Thus,

last_president = "The last President of the USA was %{presidents.last}"

Doesn’t work. Even if the array presidents is defined in Puppet.

Special selectors

The standard Array, Hash and String functions in ruby are already quite powerful. But sometimes you need some extra help. Connect defines the following special helper selectors.

extract

The extract helper allows you to extract an array of values from an array of objects. An example clarifies this:

all_nodes = [
  host('node1.domein.com'){
    ip : '10.0.0.1'
  },
  host('node2.domein.com'){
    ip : '10.0.0.2'
  }
  host('node3.domein.com'){
    ip : '10.0.0.3'
  }
]
ip_adresses = all_nodes.extract('ip')  # will be ['10.0.0.1','10.0.0.2','10.0.0.3']

to_resource

Sometimes your Connect object contains values, the original puppet type doesn’t support. To filter out all nonsupported attributes, you can use the to_resource selector on an object. The selector must be called with the type as a parameter.

my_raw_host = host('db.domain.com') {
  ip: '10.0.0.100',
  just_a_random_attribute: 10,
}  # my_raw_host cannot be use for create_resource call's because if the invalid attribute

my_host = my_raw_host.to_resource('host') # can be used as a parameter for create_resource

to_resource unfortunately only works on native types. So for defined types we need something else. For defined types, we have slice. The slice selector must be called with the attributes as values.

my_raw_host = host('db.domain.com') {
  ip: '10.0.0.100',
  just_a_random_attribute: 10,
}  # Contains more then an ip. I need just a hash with the name and an ip.
my_host = my_raw_host.slice('ip') # can be used just get the ip into the hash.
#
# It returns 
# {'db.domain.com' => { 'ip' => '10.0.0.100'}}
#

slice can be used on objects and hashes.

Multiple Objects with iterator

Sometimes you want to define a set of objects. Like, for example, a set of DNS servers. Besides some specific attributes, these object definitions are very similar. It would be a waste if we had to define them all. Connect has a solution for this. It’s called iterators.

With an iterator, you can define similar objects and replace specific values. That ‘s a little bit abstract. Let show an example:

dnsserver('dnsserver%d.mydomain.org') iterate ip from 1 to 10 do
  ip: '10.0.0.%{ip}',
  aliases: ['dnsserver%{ip}'],
end

This little snippet of Connect code, defines 10 dns servers: dnsserver1.mydomain.org with ip address 10.0.0.1 and alias dnsserver1 up to dnsserver10.mydomain.org with ip address 10.0.0.11 and alias dnsserver10. This concept is extremely convenient when defining a set of resources.

You can use integers like in the example above, but you can also use strings:

users('user%{postfix}') iterate postfix from 'aa' to 'bb' do
  username: 'user%{postfix}',
  home: '/users/user%{postfix}'
end

You can use references in the definition of the iterator.

start  = 'aa'
finish = 'bb'
users('user%{postfix}') iterate postfix from start to finish do
  username: 'user%{postfix}',
  home: '/users/user%{postfix}'
end

And last but not least, you can use multiple iterators:

route('%{ipaddress}')
  iterate ipaddress from '10.0.0.1' to '10.0.0.9'
  iterate adapter from 'eth0' to 'eth2'
  do
    ip:     %{ipaddress}',
    device: '%{adapter}'
end

This will create the following objects:

route('10.0.0.1') {ip: '10.0.0.1', device:'/eth0'}
route('10.0.0.2') {ip: '10.0.0.2', device:'/eth1'}
route('10.0.0.3') {ip: '10.0.0.3', device:'/eth2'}
route('10.0.0.4') {ip: '10.0.0.4', device:'/eth0'}
route('10.0.0.5') {ip: '10.0.0.5', device:'/eth1'}
route('10.0.0.6') {ip: '10.0.0.6', device:'/eth2'}
route('10.0.0.7') {ip: '10.0.0.7', device:'/eth0'}
route('10.0.0.8') {ip: '10.0.0.8', device:'/eth1'}
route('10.0.0.9') {ip: '10.0.0.9', device:'/eth2'}

You can stack as many iterators as you want. The largest iterator is leading for the number of objects that are generated. All other iterators will cycle their values.

External data

Puppet and Hiera are not the only systems in the world. There are a lot more possible sources of data for Puppet runs. For example:

Connect allows you to import data from any other data source. The generic syntax is:

import from datasource(param1, param2) into scope:: {
  value1 = 'lookup 1'
  value2 = 'lookup 2'
}

Check the list of available data sources to see if the data source you need, exists. Check how to make your own data source if you need to access other data.

import from puppetdb into datacenter:: {
  ntp_servers = 'Class[Ntp::Server]'  # Fetches all NTP nodes from puppetdb
                                      # into the array datacenter::ntp_servers

  dns_servers = 'Class[Dns::Server]'  # Fetches all DNS nodes from puppetdb
                                      # into the array datacenter::dns_servers
}

Check the puppetdb api for a specification of the supported query language.

Like other blocks, you can also use begin and end. If you do not specify a scope, the variables will go ito the default scope:

import from puppetdb begin
  ntp_servers = 'Class[Ntp::Server]'
  dns_servers = 'Class[Dns::Server]'
end

Alternatively, using the YAML importer:

import from yaml('/aaa/a.yaml') do
  variable1 = 'key1'
  variable2 = 'yaml::key2'
end

WARNING Not all data sources are available yet. This is only to show the syntax.

Conclusion

Using objects and imports, makes Connect an even better solution to big configurations. The object syntax allows you to very concise describe a lot of objects. When you combine it with the powerful selectors, your options are limitless.

The import facilities allow you to make a great mix between Puppet code, hiera lookups and other external data sources like Consul or PuppetDB. At this point in time, I know of no other tools that allow you to do this.

Like we said on the previous blog post: We, especially when building large infrastructures, prefer Connect over YAML any day.

In our reference implementation, you can see how we use it together with our Puppet Oracle modules and with our Puppet WebLogic modules, to install and configure Oracle and WebLogic infrastructure.

If you would like to have more information about the Connect language, checkout the Connect Language, in a Nutshell.