Enterprise-grade Puppet modules

Why Now is the Time to Switch Your Oracle and WebLogic Puppet® Modules

Tue, 13 Aug 2024 00:00:00 +0200

If you’re relying on the Biemond orawls and oradb puppet modules for managing Oracle and WebLogic in your Puppet® environment, there are some pressing issues you need to be aware of. These modules, once popular on Puppet® Forge, are on the brink of expiration. They haven’t been updated in over four years, and their GitHub repositories have been eerily quiet for just as long. Even worse, pull requests are gathering dust, unmerged, with no signs of life from the maintainers. But here’s the truth: Puppet® for Oracle and WebLogic is still very much alive and kicking.

The Reality of the Situation

Stale Modules: The biemond-orawls and biemond-oradb modules are outdated and on the verge of expiring. No updates, no new features, and a complete lack of support. Unmaintained Repositories: The GitHub repos are frozen in time. No activity, no responsiveness to pull requests. Essentially, you’re stuck with what you’ve got.

But All is Not Lost

Enterprise Modules is leading the way with their actively maintained Puppet modules for Oracle and WebLogic. Our modules don’t just keep up with the times—they’re at the cutting edge. Here’s why making the switch to Enterprise Modules is the smartest move you can make:

Full Support for the Latest Technology: Our modules are fully compatible with Puppet® 8 and the latest Oracle and WebLogic versions. You won’t be left behind using outdated software.
Continuous Updates and New Features: We’re not just maintaining these modules; we’re enhancing them. Expect regular updates that bring new features and keep your systems running smoothly.
Change doesn’t have to be hard—especially when you’ve got the right support. For a limited time, we’re offering special reduced rates and personalized assistance to help you transition from Biemond modules without a hitch.

Don’t Get Left Behind - Take Action Now

Managing Oracle and WebLogic with Puppet® is still the best choice for many enterprises. But you need tools that are reliable, up-to-date, and supported by a team that’s dedicated to your success. Don’t wait until your modules are expired and your systems are vulnerable. Contact us today to learn more about our special offer and discover how easy it is to transition to Enterprise Modules. Let us help you stay ahead of the curve. This exclusive offer won’t last forever, so act now to secure a seamless upgrade with zero stress and no disruption to your operations. Don’t miss out—make the switch today and enjoy the smoothest transition possible!

Contact us now to take advantage of our special transition offer, including reduced rates and expert support to ensure a seamless migration. Don’t wait until it’s too late—reach out today and future-proof your Oracle and WebLogic management with Enterprise Modules.

About us

Enterprise Modules is the leading developer of enterprise-ready puppet® modules for Oracle databases,Oracle WebLogic, and IBM MQ or DB2 software. Our puppet® modules help sysadmins and DBAs to automate the installation, configuration, and management of their databases and application server systems. These modules allow them to make managed, consistent, repeatable, and fast changes to their infrastructure and automatically enforce the consistency.

For more information, please visit our website: www.enterprisemodules.com or contact us at info@enterprisemodules.com.

Book review: Puppet® 8 for DevOps engineers

Wed, 12 Jul 2023 00:00:00 +0200

At times, our clients inquire about recommended books on Puppet® that are both recent and informative. Unfortunately, we were unable to provide any suggestions for some time. However, with the release of David’s new Puppet® book “Puppet 8 for DevOps Engineers”, we are now able to recommend it with confidence to those seeking to enhance their Puppet® skills and streamline their automation processes.

Scope of the book

Previously, most literature on Puppet® focused on the Puppet® language, which was extremely helpful at the time. However, Puppet® has since evolved to encompass much more. In addition to the language itself, one must also understand hiera data lookups, module creation and publication, and proper code structuring to ensure maintainability. Release management is also a crucial aspect of Puppet® use today. David’s book covers all of these topics in depth and with great clarity.

Shared Insights

David and we have both been working in the Puppet® domain for a while. It’s great to see that we share David’s view on how Puppet® users are evolving:

What we liked

We are highly regarded in the Puppet® community for our Oracle Puppet® modules. We appreciated that David used an example featuring Oracle in his explanation of roles and profiles.

Roles versus profiles versus modules

What can be confusing, at this point, is that you can end up with an Oracle Role, an Oracle profile, and an Oracle module. So, while the Oracle module configures and installs Oracle with various parameters available to it to customize the installation, the Oracle profile is about how your organization uses this module and what other modules it might add to this technology stack. You might specify that you always use Oracle with a cluster service and, therefore, your Oracle profile contains both an Oracle module and a cluster module. Alternatively, it might pass parameters to the Oracle module within your profile, which set default kernel settings for your organization’s configuration.

Summary

If you are looking to learn or improve your skills in using Puppet® for automation and configuration management, then “Puppet 8 for DevOps Engineers” by David Sandilands is the perfect book for you. This book provides a comprehensive guide to Puppet® 8, including the Puppet® language, Hiera data lookups, versioning strategies, and best practices.

What sets this book apart is the broad spectrum of subjects that David explains. The book is well-organized, with each chapter building upon the previous one, and includes hands-on exercises to reinforce learning. Overall, “Puppet 8 for DevOps Engineers” is a must-read for anyone looking to master Puppet® and take their automation skills to the next level. Although beginners at the start might be overwhelmed by the scope of the book, it is suitable for both beginners and experienced Puppet® users, and its practical approach makes it an invaluable resource for anyone working in DevOps.

If even after reading this book (or before) you think you need assistance with your Puppet® setup, we are here to assist you. We have been assisting customers getting the most out of their Puppet® setup for years. Don’t hesitate to contact us at info@enterprisemodules.com or by phone: +31 (0)653 847 326 for some consultancy.

About us

Enterprise Modules is the leading developer of enterprise-ready Puppet® modules for Oracle databases, Oracle WebLogic, and IBM MQ or DB2 software. Our Puppet® modules help sysadmins and DBAs automate the installation, configuration, and management of their databases and application server systems. These modules allow them to make managed, consistent, repeatable, and fast changes to their infrastructure and automatically enforce consistency.

For more information, please visit our website: www.enterprisemodules.com or contact us at info@enterprisemodules.com.

Customize the ordered steps Puppet® pattern with hiera values

Fri, 26 Aug 2022 00:00:00 +0200

Do you want to make your team’s Puppet® code easier to read and more concise, and better organized? While still allowing customizations to allow the code to fit every client team’s unique needs? Then using the easy_type::ordered_steps function is the way to go.

Introduction

The ordered steps Puppet® pattern we introduced in our last blog post helps you write more manageable Puppet® code. However, if the sequence of steps needs to support many different clients who all need slightly different behavior, then your streamlined code becomes quickly littered with if-statements and case statements.

This blog post introduces the easy_type::orderd_steps function that allows for customizations to your flow by setting hiera values.

What is the problem?

In our previous blog post, we used this example:

class oracle_database {
  contain systl
  contain os_users_and_groups
  contain limits
  contain packages
  contain oracle_software
  contain database
  contain database_users
  contain database_storage

  Class['systl']
  -> Class['os_users_and_groups']
  -> Class['limits']
  -> Class['packages']
  -> Class['oracle_software']
  -> Class['database']
  -> Class['database_users']
  -> Class['database_storage']
  -> Class['database ']
}

What if your infrastructure team supports many client teams that all have their own ways of doing Puppet® things? All just slightly different. Let’s say one of your customer teams has all of the OS limits in the base profile and doesn’t want any other profile modules to handle them. Another team has special requirements for creating the OS users and groups. Yet another team wants to install some extra security software before generating the database. You could use if-statements, but doing so would turn your once elegant class into a monstrosity.

What if you could leave the puppet code for this profile exactly as it is, allowing clients to modify it with hiera values? You certainly can!

Introducing the orderd_steps function

The basic change consists of this code:

easy_type::ordered_steps([
  'systl',
  'os_users_and_groups',
  'limits',
  'packages',
  'oracle_software',
  'database',
  'database_users',
  'database_storage']
)

The Puppet® code above uses the easy_type::ordered_steps function with an array of strings. Every string in the array is the name of a puppet class. All these classes will be executed in the order specified- the same as the original code.

This is a little more compact than the previous version, but it’s not much different. And what does this do for customization?

How to customize the behavior?

The ordered_steps function is quite powerful under the hood. Let’s look at some scenarios to illustrate its power.

Skipping a step

In the example above, we discussed a team that doesn’t want any profile classes to go over their security parameters. We can create a new profile class and remove the limits class from the manifest, or we could utilize an if-statement to fix this; however, the ordered_steps function allows you to specify a hiera value that will skip applying this class all together. When you add this to the hiera data for this team:

oracle_database::limits:		skip

the ordered_steps function will skip adding the limits class to your manifest. Thus implementing this use case.

Replacing a step

In the second use case, a team has unique requirements for creating OS users and groups. The current os_users_and_groups class does not align with these preferences. Instead of adding this to the company-wide Puppet® code, you would rather have the customer team manage it.

When you include this code:

oracle_database::os_users_and_groups:   customer_team_1::special_os_users_and_groups

in the hiera data for this team, instead of the standard class, the ordered_steps function will call the class supplied by the customer team. As a result, this use case is implemented.

Executing Puppet® statements before or after

The final use case is installing security software before initializing the database. We could employ an if-statement to do this, but by utilizing the ordered_steps function, we can also include additional Puppet® code.

When you include this code:

oracle_database::before_database:		customer_team_1::security_software

in the hiera data for this team, the ordered_steps function will ensure the class customer_team_1::security_software is applied before the database class is applied. Since the ordered_steps function supports both before and after classes, we could also accomplish the same using this code:

oracle_database::after_oracle_software:		customer_team_1::security_software

What do you need to use this function?

We left out the parameters you need to add to the oracle_database class in order to make things clearer, but under the hood, ordered_steps executes a lot of parameter lookups. In order for us not to run into any Puppet® warnings, those parameters have to be added to the class definition like this:

class oracle_database(
  Optional[String] $before_systl = undef,
  Optional[String] $before_os_users_and_groups = undef,
  Optional[String] $before_limits = undef,
  Optional[String] $before_packages = undef,
  Optional[String] $before_oracle_software = undef,
  Optional[String] $before_database = undef,
  Optional[String] $before_database_users = undef,
  Optional[String] $before_database_storage = undef,
  Optional[String] $systl = undef,
  Optional[String] $os_users_and_groups = undef,
  Optional[String] $limits = undef,
  Optional[String] $packages = undef,
  Optional[String] $oracle_software = undef,
  Optional[String] $database = undef,
  Optional[String] $database_users = undef,
  Optional[String] $database_storage = undef,
  Optional[String] $after_systl = undef,
  Optional[String] $after_os_users_and_groups = undef,
  Optional[String] $after_limits = undef,
  Optional[String] $after_packages = undef,
  Optional[String] $after_oracle_software = undef,
  Optional[String] $after_database = undef,
  Optional[String] $after_database_users = undef,
  Optional[String] $after_database_storage = undef,
) {
  easy_type::ordered_steps([
    'systl',
    'os_users_and_groups',
    'limits',
    'packages',
    'oracle_software',
    'database',
    'database_users',
    'database_storage']
  )
}

The easy_type::ordered_steps function is defined in the enterprisemodules-easy_type module. The enterprisemodules-easy_type module is a collection of functions, custom types, and other Puppet® goodies that we use for all of our commercial modules. You are free to install and use it. Check the documentation on the Puppet forge for more information. To use this function, you’ll have to add this module to your Puppetfile

Summary

The easy_type::ordered_steps function not only allows you to make your Puppet® code with the order steps pattern more concise, it also allows extensive customizations using hiera values. You can replace the current implementation with a team-specific implementation, or you can omit the step altogether. If you need to include code in the sequence, you can use the before or after option of the ordered_steps function.

If you could use a hand, we are here to help. Making good Puppet® code is our bread and butter at Enterprise Modules. But besides developing our own modules, we are also helping customers build the best possible Puppet® code. Do you think you could need some assistance? Don’t hesitate to contact us at info@enterprisemodules.com or by phone: +31 (0)653 847 326 for some consultancy.

About us

For more information, please visit our website: www.enterprisemodules.com or contact us at info@enterprisemodules.com.

How to use the ordered steps pattern to create better maintainable Puppet® code

Fri, 19 Aug 2022 00:00:00 +0200

Managing the order of execution for your Puppet® classes is important to ensure that configuration is completed in the correct order. But there are so many ways how you can do this. What is a useful and maintainable way to do this? The ordered steps pattern allows you to manage the order of execution for classes. This article will discuss how to use the ordered steps pattern and provide some tips for creating maintainable Puppet® code.

What are Puppet® patterns, and why are they useful?

Puppet® patterns help you design and organize your Puppet® code. By using patterns, you can make your code more maintainable and easier to understand. The fact that a pattern has a name also facilitates discussion about it in your team and ensures familiarity to the team members when they see it. They know how it works. If you’d have to give a definition for a Puppet® pattern, this would be one (based on the definition of a software design pattern):

It is not a finished design that can be transformed directly into Puppet® code. Rather, it is a description or template for how to solve a problem that can be used in many different situations.

There are several different Puppet® patterns, and each one is useful in different situations. One of the best-known Puppet® pattern is the roles and profiles pattern. This is a way of structuring your code so that you can easily reuse parts of it in different places. The roles and profiles pattern is especially useful for large Puppet® deployments, where you might have hundreds of different classes. The pattern allows you to make composable and reusable Puppet® building blocks. These building blocks are called profile classes. You can then pick multiple profile classes together and build Puppet® code that ensures a certain specific configuration. These classes are called role classes.

You can find more about the roles and profiles pattern hereand here.

What is the ordered steps pattern, and what are its benefits?

The ordered steps pattern is a Puppet® pattern that allows you to manage the order of execution of classes. This can be useful for classes that need to be executed in a specific order or when you want to ensure that a particular class is always executed before another task.

There are several benefits to using the ordered steps pattern:

It helps ensure that classes are executed in the correct order.
It can help improve the maintainability of your Puppet® code.
It can help reduce errors caused by incorrect ordering of classes.

This case really presents itself when writing roles and profile classes.

Let’s look at an contrived example. You need to write Puppet® code to ensure the installation and maintenance of your Oracle database on your Linux systems. When you look at the Oracle documentation, you see that these steps need to be executed before the Oracle software can be installed:

Ensure correct sysctl settings are applied
Ensure correct OS users and groups are created.
Ensure correct limits are sets
Ensure correct required packages are installed.

After the installation of the Oracle software, you want to ensure the database gets configured the correct way. You need to execute the next steps:

Create the database
Create the database users
Create the database storage setup (tablespaces)
etc.

Our first attempt.

This is how your Puppet® code could look like:

class oracle_database {
  include systl
  include os_users_and_groups
  include limits
  include packages
  include oracle_software
  include database
  include database_users
  include database_storage
}

As you can see, all of the steps are available in the Puppet® manifest and the code is pretty good readable. There are, however, some problems with this code. Although the order in which the code is written equals the requested order, Puppet® does not enforce this order. It might go correct (Puppet manifest order), But it might also execute the classed in another order. This can change depending on other Puppet® code included in your manifest.

Second attempt

Let’s fix that. To fix this, we include explicit ordering.

class oracle_database {
  include systl
  include os_users_and_groups
  include limits
  include packages
  include oracle_software
  include database
  include database_users
  include database_storage

  Class['systl']
  -> Class['os_users_and_groups']
  -> Class['limits']
  -> Class['packages']
  -> Class['oracle_software']
  -> Class['database']
  -> Class['database_users']
  -> Class['database_storage']
}

As you can see, we have used the -> syntax to describe explicit ordering between the classes. Now everything should be ok….? Yes, it might. But you might also run into ordering issues when your classes include other classes. When using include, Puppet® enforces the ordering on the specified classes but not on any included classes. This might lead to Puppet® code that executes some code included in the sysctl class very late in the execution, resulting in an error.

Final version

A fix for that is easy. Change all the include for contain.

class oracle_database {
  contain systl
  contain os_users_and_groups
  contain limits
  contain packages
  contain oracle_software
  contain database
  contain database_users
  contain database_storage

  Class['systl']
  -> Class['os_users_and_groups']
  -> Class['limits']
  -> Class['packages']
  -> Class['oracle_software']
  -> Class['database']
  -> Class['database_users']
  -> Class['database_storage']
  -> Class['database ']
}

See here for a full description on Puppet® containment. There is also a good video explaining this here. Now the explicit ordering of all requirements is ensured.

Could this be done differently?

A disadvantage of this solution is that ordering is done at a fairly high level. If an error is raised in the sysctl class, all other Puppet® code in this class is skipped. You probably could safely apply some parts of some other classes.

However, doing this would mean applying ordering on a much lower level, for example, on the individual resource level. It is our experience that this leads to:

More coupling between the classes, thus lesser reusability
greater chance of circular dependencies
More difficult to understand

These reasons are enough for us not to do this.

How to use the ordered steps pattern in Puppet® code

As explained before, you can use this pattern to ensure ordering between some (high-level) classes. Let’s see how we can use the pattern to create a role class.

class role {
  contain base_profile
  contain database_profile
  contain application_server_profile
  contain application_profile

  Class['base_profile']
  -> Class['database_profile']
  -> Class['application_server_profile']
  -> Class['application_profile']
}

This class manages a server with an application that needs both a database and an application server. A best practice is to extract the configuration required for all servers- regardless of their role -to what we call base_profile. So, base_profile contains elements like:

DNS configuration
Network configuration
Generic yum repositories
NTP configuration
timezone configuration
Common OS users and groups
etc.

We want to ensure our network configuration and DNS setup are correct before continuing. By ensuring the base_profile class finishes successfully, we can then move on and apply the classes database_profile, application_server_profile followed by application_profile. Doing things in this specific order guarantees a running application that all underlying components are working perfectly.

Examples of how to use the ordered steps pattern in practice

Most people use ordered steps in role and profile classes, but you can also find them in some component modules on the Puppet Forge. For example:

If you look around, you can probably find many more.

Summary

Puppet patterns are useful because they allow you to modularize your code, making it more maintainable. The ordered steps pattern is one of the used Puppet® patterns and allows you to ensure that certain classes are always executed in a specific order. You can use this pattern when creating role or profile classes, or when creating a component class.

When used correctly, the ordered steps pattern can help you create more maintainable and understandable Puppet® code. However, it is important to note that this pattern should only be used when necessary. Overusing this pattern can lead to more coupling between classes and So, use this pattern sparingly and only when it makes sense for your code.

We frequently utilize this Puppet® pattern in our code for Oracle databases, WebLogic, IBM DB2 and IBM MQ software. So, we created some handy functions to make it simpler to write code with ordered steps. We cover that more in-depth in our next blog post.

About us

For more information, please visit our website: www.enterprisemodules.com or contact us at info@enterprisemodules.com.

Cleanup temporary files in Puppet

Mon, 14 Feb 2022 00:00:00 +0100

In a Puppet® manifest, you sometimes need to make a temporary file. For example, a configuration file to be passed to an installer. Naturally, you want to clean up behind you. But in the end, something as simple as this turns out to be rather complicated. Unnecessary complicated. So that’s why we made a solution for that.

Why not ensure => absent?

Puppet allows you to manage a resource only once. This is to ensure consistent manifests and configuration results. Usually, that is excellent. But it hurts us for this use case. Because you can only add a file once to a manifest, you cannot use the Puppet file resource to both create and remove the file in the same manifest.

Let’s use `tidy`

At first glance, the tidy resource looks like the solution. But under the hood, tidy resource add’s file resources. To the existing manifest (generate). If your manifest already contains a file => ensure, The tidy generator will not (it can’t) create a file => absent resource to the same manifest. The annoying thing is that it doesn’t tell you about it.

A simple exec will do the job.

A simple way around these shortcomings is to use the exec resource and issue the /bin/rm -f command. One of the downsides of doing this is that it is OS-dependent. If you want your manifest to work on multiple OS-es, you might have to write a lot of code to remove the correct file.

Another downside of using an exec resource might be ordering. You will have to ensure that the file is removed only after it is used. If it is used multiple times in the manifest and in different if blocks, your ordering can become a bit complicated. At least too difficult for something simple as removing your trash.

The cleanup resource

The easy_type module contains a solution for the dilemma. The cleanup resource. As the name implies, it cleans up. Let’s see how we can use this.

file { '/data/my_temporary_file':
  ensure => 'present',
  content => 'my_data = true',
}

cleanup { '/data/my_temporary_file':}

This example shows the basic usage of the cleanup resource in combination with creating a temporary file. At the end of the Puppet® run, the cleanup resource will ensure that the file /data/my_temporary_file is removed. Because the cleanup resource is in no way connected to the Puppet® file resource, you can use them both together without a chance of duplicate resources.

No ordering?

Yes no ordering! That was not a typo. The cleanup resource removes the files after all of the Puppet® code has been applied. To show you how this works, let’s use this contrived example:

cleanup { '/a.a':
  loglevel => 'notice',
}

When we run this manifest you get this output:

Notice: Compiled catalog for oradb.example.com in environment production in 0.01 seconds
Notice: Applied catalog in 0.01 seconds
Notice: /Stage[main]/Main/Cleanup[/a.a]: Cleaning files marked for cleanup.
Notice: /Stage[main]/Main/Cleanup[/a.a]: File /a.a not found, skipping cleanup...

Two things are worth noticing. The first thing is:: the cleanup starts after the whole manifest is applied (e.g. the message Notice: Applied catalog in 0.01 seconds). There is no message during the application of the manifest. The second thing is that it will not fail when the file is not available. This means that you can be flexible with when and how often you a cleanup resource.

Multiple file cleanup

Like all Puppet® resources, you can use an array of strings as a title. See this example:

$files_to_cleanup = [
  '/tmp/my_config_file',
  '/tmp/my_tmp_dir',
  '/tmp/my_second_tmp/*.txt',
]
cleanup { $files_to_cleanup: }

Here you can see the usage of an array as a title and the use of different file specs to be used for cleanup of files.

Meaningful title

Although the cleanup resource doesn’t give any messages during normal operation (e.g., default loglevel is debug), it might still be useful to use a more meaningful text as a title. You can!

$files_to_cleanup = [
  '/tmp/my_config_file',
  '/tmp/my_tmp_dir',
  '/tmp/my_second_tmp/*.txt',
]
cleanup { 'All the temporary stuff':
  file_name => $files_to_cleanup,
  loglevel => 'notice',
}

Here is the output of this code:

Notice: Compiled catalog for oradb.example.com in environment production in 0.01 seconds
Notice: Applied catalog in 0.01 seconds
Notice: /Stage[main]/Main/Cleanup[All the temporary stuff]: Cleaning files marked for cleanup.
Notice: /Stage[main]/Main/Cleanup[All the temporary stuff]: File /tmp/my_config_file not found, skipping cleanup...
Notice: /Stage[main]/Main/Cleanup[All the temporary stuff]: File /tmp/my_tmp_dir not found, skipping cleanup...
Notice: /Stage[main]/Main/Cleanup[All the temporary stuff]: File /tmp/my_second_tmp/*.txt not found, skipping cleanup...

As you can see, the title now contains a meaningful text.

Where is the `cleanup` resource defined?

The cleanup custom types are defined in the enterprisemodules-easy_type module. The enterprisemodules-easy_type module is a collection of functions, custom types, and other Puppet® goodies we use for all of our commercial modules. You are free to install and use it. Check the documentation on the Puppet forge.

Conclusion

Something simple as cleaning up behind you was unnecessarily tricky before. But with the introduction of the cleanup resource it has become as easy as it should be.

About us

Enterprise modules is the leading developer of enterprise-ready Puppet® modules for Oracle databases,Oracle WebLogic, and IBM MQ software. Our Puppet® modules help sysadmins and DBAs to automate the installation, configuration, and management of their databases and application server systems. These modules allow them to make managed, consistent, repeatable, and fast changes to their infrastructure and automatically enforce the consistency.

Automate Puppet® fact caching management

Wed, 09 Feb 2022 00:00:00 +0100

Gathering facts about your system is an essential part of a Puppet® run. Most facts can be fetched very quickly and don’t significantly impact the speed of your Puppet® run. However, some facts can take a considerable time to resolve and have a big impact on the time it takes Puppet® to run. Since facter version 4, you can cache facts. Facts that take up a lot of time and/or are not very volatile can be cached. Caching of facts is controlled by a configuration file on your system. In this blog post, we show you how you can manage the contents of this configuration file with Puppet.

How it works

The facter.conf file is a configuration file that allows you to cache and block fact groups and facts, and manage how Facter interacts with your system. It contains segments about grouping facts, settings for caching facts, and global settings for facts. Here is the full Puppet® documentation. for the content of this file.

A use case.

Let’s say we have a fact that contains the installed patch information about your installed Oracle version. Unfortunately, it takes quite some time to gather this information. Also, this information doesn’t change very often. ( Not every Puppet® run or every day). This combination of properties makes this fact an ideal candidate for fact caching.

Enable fact caching

To enable fact caching for the fact `ora_installed_patches’ from within Puppet, you can add this code to your Puppet® codebase:

fact_config { 'ora_installed_patches':
  ttl =>  '24 hours'
}

This segment of Puppet® code will tell Puppet® to add the following line to your facter.conf file.

facts : {
  ttls : [
     { "ora_installed_patches" : 24 hours },
  ]
}

Effectively telling facter to cache the fact ora_installed_patches for 24 hours. See the documentation for this type. for more details.

Invalidating cache

So caching this information is excellent. It speeds up your Puppet® run. But If we use Puppet® to install or remove a patch, we want to tell facter to invalidate the cache. How do we do that? For caching invalidation, easy_type has the type fact_cache. Here is some code invalidating the ora_installed_patches cache after some patches are installed:

fact_cache {'ora_installed_patches':
  ensure => absent,
  refreshonly => true,
}
Ora_patch<||> ~> Fact_cache['ora_installed_patches']

Let’s look at that in more detail. The fact_cache {'ora_installed_patches': ensure => absent} part, tells Puppet® to absent(=remove) the facter cache for fact ora_installed_patches. The refreshonly => true part tells Puppet® only to do this when it is notified.

The line Ora_patch<||> ~> Fact_cache['ora_installed_patches'] ensures that every time a Ora_patch is added or removed, the fact_cache ora_installed_patches is notified. Thus completing the sequence of events needed to invalidate the cache.

Where are these facter types defined?

These Puppet® custom types are defined in the enterprisemodules-easy_type module. The enterprisemodules-easy_type module is a collection of functions, custom types, and other Puppet® goodies we use for all of our commercial modules. You are free to install and use it. Check the documentation on the Puppet forge.

Conclusion

When a fact takes a long time and doesn’t change very often, it is an ideal candidate for fact caching. With our types to manage fact caching, you can easily control this from within Puppet. If you could use a hand, we are here to help. Making good Puppet® code is our bread and butter at Enterprise Modules. But besides developing our own modules, we are also helping customers build the best possible Puppet® code. Do you think you could need some assistance? Don’t hesitate to contact us at info@enterprisemodules.com or by phone: +31 (0)653 847 326 for some consultancy.

About us

Vulnerability management with Puppet

Fri, 14 Jan 2022 00:00:00 +0100

The vulnerability in log4j has once again shown us that detecting and resolving vulnerabilities in your IT infrastructure can be a daunting and very time-consuming task. Now some tools are available to detect the log4j vulnerability. That is good, but only targeted at the log4j vulnerability. How about other vulnerabilities? You can be sure other vulnerabilities are coming. Hopefully not soon, but they will come! How can you be prepared? The best way to ensure that you can easily detect and fix these issues is to integrate them into your current way of working and your current set of tools. The Puppet module vulnerability does just that. This blog post will show you how you can install and use it.

Introducing the vulnerability module.

The vulnerabnility module from Enterprise Modules, seamlessly integrates vulnerability scanning into your Puppet® workflow. It works both with Puppet® Open Source and Puppet Enterprise®. The information about all of the found vulnerabilities (CVE’s) on your system(s) are available as a fact. This means that you can not only use Puppet® to detect the vulnerabilities but also use Puppet’s extensive configuration management features to mitigate any of the found vulnerabilities.

Let’s install the vulnerability module.

Since it is a regular Puppet® module, installing it is as easy as adding just another module to your Puppet® codebase. Add this line to your Puppetfile

mod 'enterprisemodules-vulnerability'

Or when you are using a manual installation process, use this command:

puppet module install enterprisemodules-vulnerability

For Linux systems, installing just the enterprisemodules-vulnerability module is enough. For Windows systems, there are some dependencies that you’ll have to add to the Puppetfile:

mod 'puppetlabs-powershell'
mod 'puppet-archive'
mod 'puppetlabs-pwshlib'

Activate

Besides installing the module, you will have to add one line of code to your Puppet® codebase to activate basic scanning. When using the roles and profiles pattern, and you have a profile that you activate on all of your systems, that would be a good place. You can also add it to your site.pp. Here is the required line:

include vulnerability

When you run Puppet® for the first time after you have added this line, you’ll see something like this:

Notice: /Stage[main]/Vulnerability::Install::Linux/File[/tmp/grype_0.31.1_linux_amd64.tar.gz]/ensure: defined content as '{md5}4bbbf141d3d5f0f2fce319fc31960fab'
Notice: /Stage[main]/Vulnerability::Install::Linux/Exec[extract grype]/returns: executed successfully
Notice: /Stage[main]/Vulnerability::Install::Linux/Exec[cleanup grype download]/returns: executed successfully
Notice: /Stage[main]/Vulnerability::Install::Linux/File[/usr/local/etc/grype_yaml.tpl]/ensure: defined content as '{sha256}45077b3b578c1e665b4d6a9ec94462653145bddd3532cb9c7c5582e15cbc99c9'
Notice: /Stage[main]/Vulnerability::Setup/Fact_config[cve_list]/ttl: defined 'ttl' as '24 hours'
Notice: /Stage[main]/Vulnerability::Setup/File[/usr/local/etc/grype.yaml]/ensure: defined content as '{sha256}5a1a07a3936cb21fa81ec816bfd85a3c0582cb9594b2f6792975471d25593f3a'
Info: /Stage[main]/Vulnerability::Setup/File[/usr/local/etc/grype.yaml]: Scheduling refresh of Fact_cache[cve_list]
Notice: /Stage[main]/Vulnerability::Setup/File[/usr/local/etc/vulnerability.conf]/ensure: defined content as '{sha256}0ab4e8922c550557594b8cb369c130258d1405095c206f9c148225ddc1bcd911'
Info: /Stage[main]/Vulnerability::Setup/File[/usr/local/etc/vulnerability.conf]: Scheduling refresh of Fact_cache[cve_list]
Notice: /Stage[main]/Vulnerability::Update/Exec[Update vulnerability database]/returns: executed successfully
Info: /Stage[main]/Vulnerability::Update/Exec[Update vulnerability database]: Scheduling refresh of Fact_cache[cve_list]
Notice: /Stage[main]/Vulnerability::Clear_facter_cache/Fact_cache[cve_list]: Triggered 'refresh' from 3 events

This line of Puppet® code, ensure’s that the required software is installed, the default settings are applied, and the vulnerability database is updated. Now Puppet® is all geared up to start scanning your systems.

Scanning your systems

Since scanning is fully integrated into your Puppet® workflow, there is nothing extra that you will have to do to start scanning your systems for vulnerabilities. Just run Puppet. On the first tun after installation, you’ll notice that getting the facts for your system takes longer than before. You can see this because of this line:

Info: Loading facts

Stays a little bit longer on your screen. This is when Puppet® does a full scan of vulnerabilities on your system. Because this is a time-consuming and intensive task, we have preconfigured it to run only once every 24 hours. By default, it scan’s your whole system. You can customize this by either excluding some files and/or directories or only scanning specific directories on your system. Check the documentation to see how you can customize this.

Inspecting the vulnerabilities from the command-line

You can now inspect the vulnerabilities. One way to do this is by using the command-line utility:

$ puppet vulnerability list --summary
Critical Vulnerabilities:    0 found.
High Vulnerabilities:        8 found.

If you would like to see all details, you can use this command:

$ puppet vulnerability list --summary --details
{ "CVE-2016-7545"=>
  {"artifact_name"=>"selinux",
   "version"=>2.9,
   "severity"=>"High",
   "fix_state"=>"unknown",
   "locations"=>
    ["/usr/lib64/python3.6/site-packages/selinux-2.9-py3.6.egg-info"]},
 "CVE-2021-0920"=>
  {"artifact_name"=>"python3-perf",
   "version"=>"4.18.0-348.2.1.el8_5",
   "severity"=>"High",
   "fix_state"=>"not-fixed",
   "locations"=>["/var/lib/rpm/Packages"]},
   ...
}
Critical Vulnerabilities:    0 found.
High Vulnerabilities:        8 found.

Using facts to manage the vulnerabilities

On the nodes

The integration ensures that the found Vulnerabilities (CVE’s) are available as facts on the system. You can inspect the fact like this:

$ puppet facts cve_list
{
  "cve_list": {
    "CVE-2005-2541": {
      "artifact_name": "tar",
      "fix_state": "wont-fix",
      "locations": [
        "/var/lib/rpm/Packages"
      ],
      "severity": "Medium",
      "version": "2:1.30-5.el8"
    },
    "CVE-2011-1017": {
      "artifact_name": "python3-perf",
      "fix_state": "wont-fix",
      "locations": [
        "/var/lib/rpm/Packages"
      ],
      "severity": "Medium",
      "version": "4.18.0-348.2.1.el8_5"
    },
    ...
    .
    .
    .
  }
}

Since this information is available to all puppet manifest, you can use this fact to ensure a specific puppet code is applied to your system if a certain CVE is found. This is extremely powerful.

On the Puppetserver

Because it is a regular Puppet® fact, you can also see this on the Puppetserver. Here is a screenshot of inspecting the facts of a specific node on the Puppetserver.

Guarding you systems agains vulnerabilities

Although all of these ways to inspect your systems for found vulnerabilities, checking this regularly would be taking a lot of time. You would like Puppet® to take care of it and only alert you when more vulnerabilities are found on your system than you bargained for. The vulnerability module had a guarding feature for that. By default, it is deactivated. To activate it, add this line of yaml to your hiera data:

vulnerability::guard:                     false

Next, you will have to tell Puppet® when to alarm you. Again you can use hiera to add these settings:

vulnerability::guard::critical:           0   # Alert me when 1 or more Critical vulnerabilities are found
vulnerability::guard::high:               2   # Alert me when2 or more High vulnerabilities are found
vulnerability::guard::medium:             ~   # I don't care about the medium vulnerabilities
vulnerability::guard::allow_list:
  - GHSA-gpvv-69j7-gwj8                   # I know about the GHSA-gpvv-69j7-gwj8 vulnerabiliy and I don't mind

Now on every Puppet® run, Puppet® will guard the number of found vulnerabilities and alert you when more are found on the system:

Notice: /Stage[main]/Vulnerability::Guard/Vulnerability_status[puppetserver.example.com]/high: high changed 7 to 2
Error: /Stage[main]/Vulnerability::Guard/Vulnerability_status[puppetserver.example.com]: Could not evaluate: More high vulnerabilities found than specified value 2 on puppetserver.example.com; actual found is: 7.

Remediating

Since all of the vulnerability information is available as a fact, you can easily write Puppet® code to remediate any found vulnerabilities. Here is a straightforward example:

# @summary Resolve CVE-2021-43527
#
# Resolve the CVE-2021-43527 by updating the nss-tools package to
# the latest version. 
#
class resolve::cve_2021_43527 {

  if vulnerability::detect('CVE-2021-43527') {
    package {'nss-tools':
      ensure => 'latest',
      notify => fact_cache['cve_list']
    }
  }
}

This Puppet® code check’s the cve_list fact if vulnerability CVE-2021-43527 is on your system and resolves the vulnerability by installing the latest version of the nss-tools package.

Puppet® to resolve vulnerabilities

We are in the process of publishing an open-source module containing Puppet® remediation code for many vulnerabilities. Since it is open-source, the community can help extend it so everybody can use this to quickly and safely resolve any of your vulnerabilities. Stay tuned for more information about this module.

More information

You can find more information about Puppet® vulnerability scanning here:

Some example code:

Vulnerability demo

Interested?

We hope this article has shown you the benefits of integrating vulnerability management into your Puppet® workflow and how easy it is to get it up and running. We know some of the vulnerability scanning tools out there are expensive. Besides being very efficient and easy, we have made sure that vulnerability scanning is very affordable. Contact us if you want more information.

If you want to know more, please check out the documentation of these modules here for more information about this module and all the possibilities. If you are just starting, check out our FREE trial license to get going.

About us

A solution for Puppet® hiera yaml sprawl

Wed, 20 Oct 2021 00:00:00 +0200

hiera is a very powerful way to separate code and data in Puppet® code. But sometimes, you need more than the standard functionality for levels that hiera provides. One way to keep configuration data from being duplicated over multiple levels would be to include other yaml files. Unfortunately, yaml doesn’t have any means to do this. That is why we built an extended yaml hiera backend that allows you to do this.

A use case for yaml include

One of the use cases that we run into regularly with our clients, is doing version upgrades of for example Oracle software. We have one role yaml that contains all of the hiera data needed to ensure a correct Oracle database is built and configured by Puppet. Here is an excerpt of such a yaml file:

ora_profile::database::dbname:                  MYDB
ora_profile::database::version:                 19.0.0.0
ora_profile::database::db_patches::patch_level: JUL2021RU
ora_profile::database::oracle_home:             /u01/app/oracle/product/19.0.0.0/db_home1
ora_profile::database::db_software::file_name:  LINUX.X64_193000_db_home
ora_profile::database::db_software::dirs:
- /u01/app/oracle/product
- /u01/app/oracle/product/DB19"

As you can see, this yaml file contains some references to the version used. In this case, Oracle 19.

When we want to start using Oracle 21, however, we want to do this gradually. So first on one development machine and then on more or even on all development machines and then through testing and acceptance finally to production. One of the ways to do this is to add the updated hiera data to the node-specific hiera levels. This allows you to start rolling out Oracle 21 controlled on a node-by-node basis. This, however leads to a sprawl of hiera entries throughout your yaml files. Wouldn’t it be great if you could include a file containing all entries for a specific version? Yes, you can!!

How does it look?

Here is how this looks in a yaml file:

---
some::other::data:  yes
...
# @include '../includes/oracle_21.yaml'
#
other::entry::in::your:yaml: 	just_some value
...

This little snippet will include the file oracle_21.yaml from the directory includes below the directory of the current yaml file into your current yaml file.

The include process itself replaces the include line with the content of the included files and then passes it to the Ruby yaml processor as one entity.

But why is it a comment (e.g. behind a #)? The reson for this, is that many Puppet® CI tools check the validity of yaml files. If we would have put the include as a “normal” stratement, all of the CI checks for yaml will fail. That is not what we want.

How do we enable this?

The hiera backend is called easy_type::yaml_with_include and as the name implies, it is part of the easy_type module. The enterprisemodules-easy_type module is a collection of functions, custom types and other Puppet® goodies we use for all of our commercial modules. You are free to install and use it. Check the documentation on the Puppet forge on how to install this module into your Puppet® code base.

After the installation, we have to configure hiera to use it. The easiest way to do this is to replace the defaults in your hiera.yaml file. We need to change the value for the data_hash key from yaml_data to easy_type::yaml_with_include. Here is an example of how your hiera.yaml could look.

---
version: 5
defaults:  # Used for any hierarchy level that omits these keys.
  datadir: hieradata                          # This path is relative to hiera.yaml's directory.
  data_hash: easy_type::yaml_with_include  

Let’s try it.

Our role yaml now contains the include statement # @include '../includes/oracle_21.yaml'. Our oracle_21.yaml` file looks like this:

ora_profile::database::version:                  21.0.0.0
ora_profile::database::db_patches::patch_level:  JUL2021RU
ora_profile::database::oracle_home:              /u01/app/oracle/product/21.0.0.0/db_home1
ora_profile::database::db_software::file_name:   LINUX.X64_213000_db_home
ora_profile::database::db_software::dirs:
- /u01/app/oracle/product
- /u01/app/oracle/product/DB21"

The best way to check if it works and also to debug lookups, is to use the puppet lookup utility. Let’s use this utility to lookup the version of Oracle.

$ puppet lookup ora_profile::database::version
--- 21.0.0.0

And it works. But it would be nice to get a bit more details. Fortunately, the puppet lookup utility has a way of explaining what it does. Let’s try again and use the --explain option:

$ puppet lookup ora_profile::database::version --explain

...

  Environment Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
    Merge strategy hash
      Hierarchy entry "Per-node data"
        Merge strategy hash
          Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.example.com.yaml"
            Original path: "nodes/%{trusted.certname}.yaml"
            No such key: "lookup_options"
            Including file '/etc/puppetlabs/code/environments/production/hieradata/includes/oracle_21.yaml' into contents of '/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.example.com.yaml'
          Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.yaml"
            Original path: "nodes/%{hostname}.yaml"
            Path not found

...

Searching for "ora_profile::database::version"
  Global Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
    No such key: "ora_profile::database::version"
  Environment Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
    Hierarchy entry "Per-node data"
      Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.example.com.yaml"
        Original path: "nodes/%{trusted.certname}.yaml"
        Found key: "ora_profile::database::version" value: "21.0.0.0"

As you can see by the line /etc/puppetlabs/code/environments/production/hieradata/includes/oracle_21.yaml' into contents of '/etc/puppetlabs/code/environments/production/hieradata/nodes/db190.example.com.yaml' hiera tells you it is including the file. It only notifies you once. Because for performance reasons, it only reads the files one time. Every additional hiera lookup will use the cached data and not know anything about the inclusion of the file.

Conclusion

When sprawl of hiera data over your many hiera levels is an issue for you, we have a way to help you. The yaml_with_include hiera backend allows you to use include statements in your hiera yaml data. We hope this will help you. If you could use a hand, we are here to help. Making good Puppet® code, is our bread and butter at Enterprise Modules. But besides developing our own modules, we are also helping customers build the best possible Puppet® code. Do you think you could need some assistance? Don’t hesitate to contact us at info@enterprisemodules.com or by phone: +31 (0)653 847 326 for some consultancy.

About us

Dynamic but versioned hiera data with git

Fri, 01 Oct 2021 00:00:00 +0200

For some organizations, having dynamic hiera data can be a real time saver. Add a small change to your hiera data; there is no need for a Puppet® redeploys, and off you go. Although this is fast, it has some potential downsides too. You cannot see who did this change and why and when it was done. Fortunately, there is a way around some of these downsides.

There is a module for that

On the puppetforge there is a module called hiera_http that can help us implement this. This module implements a hiera backend that can connect to any http endpoint. It can read yaml and json data. You can tell the hiera backend for what hiera keys this backend is used.

Our use case

Our use case for this solution is the definition of available patches. These patches are currently defined in the module hiera data of our module ora_profile. So whenever a new patch level comes available, we update the hiera data in this module and publish a new version of the module for our customers to use. However, some customers don’t want to update this module so often. A new version of a module might not only contain updated data, but it might also contain incompatible Puppet® code and at least require organizations to do some testing. So for these customers, we would like to allow them to only use the updated patch-levels in the hiera data but still keep on using the version of the module they currently have. They could, of course, copy the definitions of the patches to their own hiera files. But this means some extra maintenance burden on your internal team. The solution we have in mind is to allow customers to use the latest version of the hiera data without the need to install the newest version.

How to make it work

Let’s first make sure that the module and all its prerequisites are installed. As the documentation states, the lookup_http gem must be installed. Here is the command to do this for your puppet server.

puppetserver gem install lookup_http

When you are using the puppet lookup feature, you’ll also need to install this gem for the ruby context outside of the puppetserver.

/opt/puppetlabs/puppet/bin/gem install lookup_http

Next up is the installation of the module itself. You can do this manual with this command:

puppet module install crayfishx/hiera_http

A better way is to use a Puppetfile and add this line to it:

mod 'crayfishx-hiera_http', 'x.x.x'

and then use the puppet deployments to ensure

What data to use

Now the essential tasks of installation are done, let’s focus on the configuration. We need to tell the hiera backend that it must use a github url to fetch the data. Here is part of the hiera.yaml that defines the hiera search levels.

hierarchy:
 …. 
 - name: "Dynamic patches lookup"
    lookup_key: hiera_http
   uris:
     - https://raw.githubusercontent.com/enterprisemodules/ora_profile/master/data/defaults.yaml
   options:
     output: yaml
     use_ssl: true
     confine_to_keys: 
        - ora_profile::database::patch_levels
 

Let me take you through some of the options:

uris: This is the url where the backend will fetch the data. You can get this url by going to the repository in github and open the file you want to use. Then select the raw button, and github will take you to a raw version of this data file. You can copy the url and put it in your hiera.yaml. For our use case, we use the main branch. But you can also select a specific branch, tag, or even commit.
output: In our use-case, the file we selected is a yaml file, so we want the backend to interpret the data as yaml data.
use_ssl: Although the backend should be able to detect itself if this is a https or http url, we need to set the value to true when using a https url,
confine_to_keys: We only want to enable the patch level through this mechanism in our use-case. So we specify only one key, namely the key ora_profile::database::patch_levels. All other hiera keys will be resolved through the other hiera levels and, in this case, be resolved from the module data of the current version of the installed module. You can, however, add multiple keys, and you can also use regular expressions like for example : "application.*"

test it

Now everything should be set up correctly, and we are ready to test. You can, of course, just run the Puppet® agent and see if it works, but it can be difficult to distinguish between already available hiera-data in the module and the dynamic data from git. Therefore I’d like to use the puppet lookup --explain command. This command does the lookup and explains what actions hiera takes on each hiera level.

$ puppet lookup ora_profile::database::patch_levels  --explain
/etc/puppetlabs/code/environments/production/modules/hiera_http/lib/puppet/functions/hiera_http.rb:105: warning: URI.escape is obsolete
Searching for "lookup_options"
  Global Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
...[some output deleted]
       Hierarchy entry "Dynamic patches lookup"
          URI "https://raw.githubusercontent.com/enterprisemodules/ora_profile/master/data/defaults.yaml"
            Original uri: "https://raw.githubusercontent.com/enterprisemodules/ora_profile/master/data/defaults.yaml"
            Found key: "ora_profile::database::patch_levels" value: {
...[some output deleted]
...

As you can see, the Hierarchy entry "Dynamic patches lookup" returned the data, so it works.

Gotchas

In this use case, we only fetch reference data. This means that any change to this data doesn’t directly change the Puppet® catalog. It allows you to use a different patch level in your manifest. Only after you change the patch level in your Puppet® manifest for this node, the patch level will be applied. We like this because it means your system can only change after a Puppet® deploy, and not all of a sudden show changes. For some use cases, however, this is precisely what people like and need.

Your hiera data can contain lookups in its data. For example:

my_key: "%{lookup('other_key'}"

When your hiera data contains this, beware that the other_key resolves from the same source version. Else you can get very nasty and brutal to find errors.

Conclusion

In this blog post, we have shown you how to add dynamic hiera data to your Puppet® environment. Dynamic but still versioned, so you are still in control. Your use case might be slightly different, so if you need some help, don’t hesitate to contact us at info@enterprisemodules.com or by phone: +31 (0)653 847 326 for some consultancy.

About us

Gain control of your MQ versions and fixpacks

Wed, 25 Aug 2021 00:00:00 +0200

Many larger organizations use the IBM MQ as messaging middleware. Sometimes the IT infrastructure needed to run MQ can span a large number of nodes. This large number of nodes increases the issue of ensuring all systems run a correct fixpack or even a correct version of the MQ software. Having MQ running on multiple operating systems like AIX, Linux, Solaris, and Windows increases the complexity of this problem.

Using a combination of Puppet’s desired state and Puppet® tasks, you can easily (re)gain control of your MQ versions and fixpacks on a large fleet of systems using multiple operating systems.

Puppet® Desired State Language

The most known feature of Puppet® is the Puppet® desired state management. With this feature, you specify how a system should be configured using the Puppet® language. The Puppet® language allows you to specify certain external data. This is called hiera. Our Puppet IBM modules use this feature extensively.

Here is an example of the hiera data you can use to specify a particular MQ version and fixpack.

mq_install::fixpack::service_window:			   DIRECT
ibm_profile::mq_machine::software::version:  9.1.0.0
ibm_profile::mq_machine::fixpack::version:   9.1.0.7

When puppet run’s on a newly installed system, the part of the output that installs the MQ software looks like this:

Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.1.0.0]/Archive[/tmp/IBM_MQ_9.1.0.0_LINUX_X86-64.tar.gz]/ensure: download archive from /vagrant/modules/software/files/IBM_MQ_9.1.0.0_LINUX_X86-64.tar.gz to /tmp/IBM_MQ_9.1.0.0_LINUX_X86-64.tar.gz and extracted in /tmp with cleanup
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.1.0.0]/Exec[Accept MQ License Requirements]/returns: executed successfully
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.1.0.0]/Package[MQSeriesRuntime-9.1.0-0]/ensure: created
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.1.0.0]/Package[MQSeriesJRE-9.1.0-0]/ensure: created
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.1.0.0]/Package[MQSeriesJava-9.1.0-0]/ensure: created
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.1.0.0]/Package[MQSeriesServer-9.1.0-0]/ensure: created
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.1.0.0]/File[Remove temporary files]/ensure: removed
Notice: /Stage[main]/Mq_install::Fixpack/File[/tmp/mq_fixpack_9.1.0.7]/ensure: created
Notice: /Stage[main]/Mq_install::Fixpack/Archive[/tmp/9.1.0-IBM-MQ-LinuxX64-FP0007.tar.gz]/ensure: download archive from puppet:///modules/software/9.1.0-IBM-MQ-LinuxX64-FP0007.tar.gz to /tmp/9.1.0-IBM-MQ-LinuxX64-FP0007.tar.gz and extracted in /tmp/mq_fixpack_9.1.0.7 with cleanup
Notice: /Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9107-9.1.0-7]/Package[MQSeriesRuntime-U9107-9.1.0-7]/ensure: created
Notice: /Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9107-9.1.0-7]/Package[MQSeriesJRE-U9107-9.1.0-7]/ensure: created
Notice: /Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9107-9.1.0-7]/Package[MQSeriesJava-U9107-9.1.0-7]/ensure: created
Notice: /Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9107-9.1.0-7]/Package[MQSeriesServer-U9107-9.1.0-7]/ensure: created
Notice: /Stage[main]/Mq_install::Fixpack/Exec[Remove temporary files for fixpack 9.1.0.7]/returns: executed successfully

As you can see, all steps needed to fetch the MQ software and fixpack and actions required to install them are automated. Because Puppet® uses idempotency, on a second Puppet® run, Puppet® detects that the correct versions are installed, and Puppet® will do nothing.

Performing a fixpack update

If a system has been provisioned previously with the hiera data displayed above, The system will be running MQ 9.1.0.7. If we want to update to fixpack 8 (e.g., version 9.1.0.8), we only need to change the hiera data. Here is how it needs to look.

mq_install::fixpack::service_window:			   DIRECT
ibm_profile::mq_machine::software::version:  9.1.0.0
ibm_profile::mq_machine::fixpack::version:   9.1.0.8

If Puppet® now runs on this system, it determines that you have specified a newer version of the fixpack. Because the service_window parameter is set to DIRECT, it will immediately shut down MQ and apply the required fixpack. After the upgrade is done, Puppet® will automatically restart all previously running MQ queue managers.

Here is some example output:

Notice: /Stage[main]/Mq_install::Internal::Mq_stop/Exec[Stop MQ manager QM01]/returns: executed successfully
Notice: /Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9108-9.1.0-8]/Package[MQSeriesRuntime-U9108-9.1.0-8]/ensure: created
Notice: /Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9108-9.1.0-8]/Package[MQSeriesJRE-U9108-9.1.0-8]/ensure: created
Notice: /Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9108-9.1.0-8]/Package[MQSeriesJava-U9108-9.1.0-8]/ensure: created
Notice: /Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9108-9.1.0-8]/Package[MQSeriesServer-U9108-9.1.0-8]/ensure: created
Notice: /Stage[main]/Mq_install::Fixpack/Exec[Remove temporary files for fixpack 9.1.0.8]/returns: executed successfully
Notice: /Stage[main]/Ibm_profile::Mq_machine::Manager_setup/Mq_manager[QM01]/status: status changed 'Ended normally' to 'Running'
Notice: /Stage[main]/Ibm_profile::Mq_machine::Autostart/Mq_install::Autostart[QM01]/Exec[stop QM01 running outside systemd]/returns: executed successfully
Notice: /Stage[main]/Ibm_profile::Mq_machine::Autostart/Mq_install::Autostart[QM01]/Service[mq@QM01]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Ibm_profile::Mq_machine::Autostart/Mq_install::Autostart[QM01]/Service[mq@QM01]: Unscheduling refresh on Service[mq@QM01]

On the next Puppet® run, Puppet® will detect that the correct versions are installed, and Puppet® will do nothing.

Oh Oh, issues

Although you have prepared everything meticulously, sometimes things go wrong, and you have to roll back to the previous. Fortunately, Puppet® can help you here too: Just change the hiera data back and run Puppet® again.

Here is the example output.

Notice: /Stage[main]/Mq_install::Internal::Mq_stop/Exec[Stop MQ manager QM01]/returns: executed successfully
Notice: /Stage[main]/Mq_install::Fixpack/Exec[uninstall fixpack U9108]/returns: executed successfully
Notice: /Stage[main]/Ibm_profile::Mq_machine::Manager_setup/Mq_manager[QM01]/status: status changed 'Ended normally' to 'Running'
Notice: /Stage[main]/Ibm_profile::Mq_machine::Autostart/Mq_install::Autostart[QM01]/Exec[stop QM01 running outside systemd]/returns: executed successfully
Notice: /Stage[main]/Ibm_profile::Mq_machine::Autostart/Mq_install::Autostart[QM01]/Service[mq@QM01]/ensure: ensure changed 'stopped' to 'running'

As you can see, fixpack U9108 is uninstalled, and the situation is back to before the fixpack update.

Performing a major update

At this point in time, MQ version 9.2 is the latest. Let’s update the systems to this version. Here is the required hiera data.

mq_install::software::service_window:        DIRECT
ibm_profile::mq_machine::software::version:  9.2.0.0
# ibm_profile::mq_machine::fixpack::version: This needs to unset when no fixpack is installed

The process is the same as with the installation of a fixpack. Puppet® determines that you have requested a new major version. It will shut down all running MQ managers, uninstall the old software, install the new major version and start the queue managers again.

Notice: /Stage[main]/Mq_install::Internal::Mq_stop/Exec[Stop MQ manager QM01]/returns: executed successfully
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.2.0.0]/Exec[uninstall RPMS for version 9.1.0.7]/returns: executed successfully
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.2.0.0]/Archive[/tmp/IBM_MQ_9.2.0_LINUX_X86-64.tar.gz]/ensure: download archive from /vagrant/modules/software/files/IBM_MQ_9.2.0_LINUX_X86-64.tar.gz to /tmp/IBM_MQ_9.2.0_LINUX_X86-64.tar.gz and extracted in /tmp with cleanup
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.2.0.0]/Exec[Accept MQ License Requirements]/returns: executed successfully
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.2.0.0]/Package[MQSeriesRuntime-9.2.0-0]/ensure: created
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.2.0.0]/Package[MQSeriesGSKit-9.2.0-0]/ensure: created
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.2.0.0]/Package[MQSeriesJRE-9.2.0-0]/ensure: created
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.2.0.0]/Package[MQSeriesJava-9.2.0-0]/ensure: created
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.2.0.0]/Package[MQSeriesServer-9.2.0-0]/ensure: created
Notice: /Stage[main]/Mq_install::Software/Mq_install::Software::Unix[mq_installation_9.2.0.0]/File[Remove temporary files]/ensure: removed
Notice: /Stage[main]/Ibm_profile::Mq_machine::Manager_setup/Mq_manager[QM01]/status: status changed 'Ended normally' to 'Running'
Notice: /Stage[main]/Ibm_profile::Mq_machine::Autostart/Mq_install::Autostart[QM01]/Exec[stop QM01 running outside systemd]/returns: executed successfully
Notice: /Stage[main]/Ibm_profile::Mq_machine::Autostart/Mq_install::Autostart[QM01]/Service[mq@QM01]/ensure: ensure changed 'stopped' to 'running'

When to update

Before you can apply a fixpack or even install a new major version of MQ, you are required to shut down the running MQ software on your system. Obviously, you want control over when you do this. You don’t want Puppet® to shut down your MQ messaging during operating hours. One of the ways to get control over this is to specify a time frame for you service_window parameter. Let’s say you are allowed to shut down your MQ software every night between 1:00 and 2:00. You can tell Puppet® this by setting the service_window.

mq_install::fixpack::service_window:    1 - 2
mq_install::software::service_window:   1 - 2 

This means that Puppet® will skip the shutdown and installation of a new fixpack or major version every time unless it is currently running in the service window.

How about an initial install outside the service window

When you are installing a brand new system, you probably don’t need to adhere to the service window because no users are using the system yet. The MQ modules will detect this use-case and install any requested fixpack on initial install

We need more control

Although the service window gives you a lot more control, for some organizations, it is not enough. Maybe you need to contact your business and use a precise moment for a subset of systems. For this use case, we can use the Puppet® plan (or Bolt plan). In this blog post, we will show you the bolt commands.

To tell the Puppet® agent to leave the MQ versions like they are, we need to specify the next hiera data:

mq_install::fixpack::service_window:    INSTALL_ONLY
mq_install::software::service_window:   INSTALL_ONLY

This means that only on a first Puppet® run, Puppet® will install the software and the fixpack, but Puppet® will never do an update. To actually do the update, we need to run a bolt plan.

Puppet/Bolt plans

Here is the command that shows the available bolt plans.

$ bolt plan show --modulepath /etc/puppetlabs/code/environments/production/modules/ | grep mq_install
  mq_install::apply_fixpack           Apply the in hiera specified MQ fixpack to the specified nodes.
  mq_install::apply_upgrades          Apply the in hiera specified MQ upgrades to the specified nodes.

Let’s see some more information about the plan mq_install::apply_fixpack .

bolt plan show --modulepath /etc/puppetlabs/code/environments/production/modules/ mq_install::apply_fixpack
mq_install::apply_fixpack
  Apply the in hiera specified MQ fixpack to the specified nodes.

Usage
  bolt plan run mq_install::apply_fixpack targets=<value>

Parameters
  targets  TargetSpec
    The MQ node(s) you want to apply a software update to

Module
  /etc/puppetlabs/code/environments/production/modules/mq_install

As you can see, the bolt plan takes a parameter called targets. You can pass a comma-separated list of target names, wildcard patterns, or group names to a plan parameter of type TargetSpec. For more information on the TargetSpec type, see Writing plans in the Puppet® language. The fact that you can specify specific target nodes means that you can precisely select the nodes you want to update the MQ fixpack with a bolt plan. The plan is executed directly and concurrently on all specified target nodes.

Here is an example of this running on a node.

$ bolt plan run --modulepath /etc/puppetlabs/code/environments/production/modules/ --hiera_config /etc/puppetlabs/code/environments/production/hiera.yaml mq_install::apply_fixpack  targets=mq01.example.com
Starting: plan mq_install::apply_fixpack
Starting: Uploading Enterprise Modules run-time information... on mq01.example.com
Finished: Uploading Enterprise Modules run-time information... with 0 failures in 0.04 sec
Starting: install puppet and gather facts on mq01.example.com
Finished: install puppet and gather facts with 0 failures in 5.88 sec
Starting: apply catalog on mq01.example.com
Finished: apply catalog with 0 failures in 53.43 sec
Puppet: When a fixpack update is detected, it will take place directly without any schedule
Puppet: When a software update is detected, it will be skipped, because service window is set to INSTALL_ONLY
/Stage[main]/Mq_install::Fixpack/File[/tmp/mq_fixpack_9.1.0.8]/ensure: created
/Stage[main]/Mq_install::Fixpack/Archive[/tmp/9.1.0-IBM-MQ-LinuxX64-FP0008.tar.gz]/ensure: download archive from /vagrant/modules/software/files/9.1.0-IBM-MQ-LinuxX64-FP0008.tar.gz to /tmp/9.1.0-IBM-MQ-LinuxX64-FP0008.tar.gz and extracted in /tmp/mq_fixpack_9.1.0.8 with cleanup
/Stage[main]/Mq_install::Internal::Mq_stop/Exec[Stop MQ manager QM01]/returns: executed successfully
/Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9108-9.1.0-8]/Package[MQSeriesRuntime-U9108-9.1.0-8]/ensure: created
/Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9108-9.1.0-8]/Package[MQSeriesJRE-U9108-9.1.0-8]/ensure: created
/Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9108-9.1.0-8]/Package[MQSeriesJava-U9108-9.1.0-8]/ensure: created
/Stage[main]/Mq_install::Fixpack/Mq_install::Internal::Mq_packages[U9108-9.1.0-8]/Package[MQSeriesServer-U9108-9.1.0-8]/ensure: created
/Stage[main]/Mq_install::Fixpack/Exec[Remove temporary files for fixpack 9.1.0.8]/returns: executed successfully
/Stage[main]/Main/Mq_install::Autostart[QM01]/Service[mq@QM01]/ensure: ensure changed 'stopped' to 'running'
Puppet: Applied catalog in 41.75 seconds
Finished: plan mq_install::apply_fixpack in 1 min, 1 sec
Plan completed successfully with no result

Experience the Power of Puppet® for IBM MQ

If you want to play and experiment with Puppet® and IBM MQ, please contact us to discuss your specific use case and requirements. We can help you get started with our modules and provide guidance on implementation.

For more information about our IBM MQ solutions, please contact us.

Conclusion

In this blog post, we have shown you easy it is to (re)gain control of the MQ fixpacks and versions you are running on your fleet of systems. Puppet, in combination with the Puppet® modules for MQ make it very easy to automate this work. The Puppet® desired state language can work seamlessly with Puppet(Bolt) plans to give you full control over the timing of your updates. Because the modules support multiple operating systems, your entire MQ fleet can be managed with one single approach and set of tools.

If you would like to know more, contact us at info@enterprisemodules.com or by phone: +31 (0)653 847 326

Enterprise-grade Puppet modules

Why Now is the Time to Switch Your Oracle and WebLogic Puppet® Modules

The Reality of the Situation

But All is Not Lost

Don’t Get Left Behind - Take Action Now

About us

Book review: Puppet® 8 for DevOps engineers

Scope of the book

Shared Insights

Puppet has come a long way since its inception.

What we liked

Roles versus profiles versus modules

Summary

About us

Customize the ordered steps Puppet® pattern with hiera values

Introduction

What is the problem?

Introducing the orderd_steps function

How to customize the behavior?

Skipping a step

Replacing a step

Executing Puppet® statements before or after

What do you need to use this function?

Summary

About us

How to use the ordered steps pattern to create better maintainable Puppet® code

What are Puppet® patterns, and why are they useful?

Definition of a Puppet® Pattern

What is the ordered steps pattern, and what are its benefits?

Our first attempt.

Second attempt

Final version

Could this be done differently?

How to use the ordered steps pattern in Puppet® code

Examples of how to use the ordered steps pattern in practice

Summary

About us

Cleanup temporary files in Puppet

Why not ensure => absent?

Let’s use tidy

A simple exec will do the job.

The cleanup resource

No ordering?

Multiple file cleanup

Meaningful title

Where is the cleanup resource defined?

Conclusion

About us

Automate Puppet® fact caching management

How it works

A use case.

Enable fact caching

Invalidating cache

Where are these facter types defined?

Conclusion

About us

Vulnerability management with Puppet

Introducing the vulnerability module.

Let’s install the vulnerability module.

Activate

Scanning your systems

Inspecting the vulnerabilities from the command-line

Using facts to manage the vulnerabilities

On the nodes

On the Puppetserver

Guarding you systems agains vulnerabilities

Remediating

Puppet® to resolve vulnerabilities

More information

Interested?

About us

A solution for Puppet® hiera yaml sprawl

A use case for yaml include

How does it look?

How do we enable this?

Let’s try it.

Conclusion

About us

Dynamic but versioned hiera data with git

There is a module for that

Let’s use `tidy`

Where is the `cleanup` resource defined?