Scroll to top
© 2019, DigitalOnUs

Integrating Terraform with Puppet, vRealize, and MuleSoft


DigitalOnUs - December 6, 2019 - 0 comments

Cloud computing has gained immense popularity in the last decade, as it has managed to efficiently change the way companies scale and manage databases, servers, and data. However, with the proliferation of complex computing systems, which often need to perform in high-traffic situations, the need for cloud infrastructure has been on the rise. Infrastructure as Code (IaC) addresses this concern and provides businesses with a way to manage databases, networks, and other key infrastructure through code, rather than manually and by hand. There are numerous IaC tools available in the market for the diverse needs of businesses. Puppet and Terraform are two of the more popular ones.

Puppet is a configuration management system that allows users to define and deploy system configurations to manage diverse and large-scale systems quickly and effectively. It allows seamless integration with other existing servers, making integrated automated testing and compliance validation to the development lifecycle.

Terraform is an IaC provisioning tool developed by HashiCorp, which makes it possible to manage infrastructure at large scale, from storage to networking, to virtual machines and most everything in between. Terraform makes both immutable and mutable infrastructure approaches possible: the user can define infrastructure that is never modified or infrastructure that is modified when necessary.

Terraform mandates a declarative approach to infrastructure via HashiCorp Declarative Language (HCL), which allows the user to define the entire desired state of the infrastructure. It is not necessary for the user to know the full record of changes that have taken place since its origin. Terraform supports a variety of platforms like AWS, Alibaba Cloud, Vault, Google Cloud and more. Terraform is open source and is easily extensible to provide support for new platforms and cloud providers. It can deploy multiple resources in one safe, single workflow, and can isolate groups of resources from one another.

Users typically require multiple tools to build a robust infrastructure and this is where cloud management tools like vRealize and data integration platforms like MuleSoft factor in. vRealize is a hybrid cloud management platform that assists in the personalization of IT infrastructure and configuration of resources. MuleSoft, on the other hand, is a systematized data integration platform that enables users to connect data and applications in cloud environments.

Integration of Terraform, vRealize, and Puppet

Terraform can be used with a configuration management tool like Puppet with remarkable ease. All necessary cloud artifacts like AMIs and Docker images can be provisioned by Puppet using build systems such as Packer. Terraform can be used to define and manage core infrastructures such as route tables, load balancers, and servers. Puppet can then be used to configure these systems, and to deploy applications to these systems.

When using a provider like Amazon Web Services (AWS), an Amazon Machine Image (AMI) needs to be used to start and run virtual machines, which are called EC2 Instances. Existing free AMIs can be used or new ones can be created using a tool called Packer. Each instance is allotted basic resources like memory, network capacity, and disk space.

Terraform has the ability to work with VMware vRealize to deploy existing blueprints. To configure VMware vRA7 Provider, arguments like username, password, tenant, and host need to be configured. Additionally, if there exists on-premises infrastructure such as offices and data-centers, VMware has a variety of solutions to help businesses manage or replace them.

MuleSoft can be used to connect data sources, perform analytics, and can be deployed at any point in the infrastructure lifecycle. Triggers can also be used to deploy updated MuleSoft binaries, and significant automation can be realized with continuous integration (CI) and continuous deployment (CD) systems.

Understanding Different VM Lifecycle Phases

At least four different lifecycle phases exist for machines:

  1. AMI Creation Time
  2. Infrastructure Provisioning Time
  3. System Boot/Cloud-init Time
  4. Runtime

Each of these lifecycle phases has its own use-cases, which will now be described.

AMI Creation Time

Puppet can be used in conjunction with Packer to create machine images with certain preset configurations. Any virtual machine booting this machine image will have all of the packages, services, and files pre-installed in it that it will require on machine boot. All instances using this machine image will start in the same state.

Infrastructure Provisioning Time

Declarative code is written in HCL for Terraform and stored in a Git repository. Any Terraform code modifications in the Git repository will activate an automation service such as Jenkins, which can execute tests and deploy new/changed infrastructure.

System Boot/Cloud-Init Time

Cloud-init is a system service that executes upon a machine’s first boot. This provides an opportunity for individual machine configuration to diverge on a per-host basis. If machines are dependent on Puppet configuration, Puppet can be run during the cloud-init execution.

Machine Runtime

After booting the machine and executing cloud-init, configuration management tools like Puppet and Ansible can be run on-demand or at intervals to apply configuration changes to machines.

These four machine lifestyle phases are used in combination to build and maintain diverse systems for large organizations.

Infrastructure Mutability Patterns

There are generally around four infrastructure mutability patterns, each best-suited to particular use-cases:

  • Fully Dynamic
  • Mostly Dynamic
  • Mostly Static
  • Fully Static

Each of these patterns has advantages and disadvantages, so choosing the correct pattern for your use-case is of utmost importance.

Fully Dynamic

Description

Here, an upstream AMI is used directly instead of building a custom AMI. Terraform creates the virtual machines and specifies cloud-init configuration which is used to configure the entire machine from scratch. As a result, all configuration is performed during cloud-init execution. Each machine starts, executes cloud-init, which installs a specified version of Puppet, and then executes Puppet in order to configure itself.

Pros

No AMI build process is involved or required.

Cons

The machine is fully dependent on cloud-init to launch, so if there is any kind of fault encountered during cloud-init execution, the machine will fail to provision itself properly.

Since all configuration happens after machine boot, however long cloud-init and Puppet take to execute will be the minimum amount of time required after a machine boot before a machine is ready to serve traffic.

Mostly Dynamic (Typical Puppet Workflow)

Description

In this strategy, an AMI is created with a specific Puppet version installed and provided with a fairly generic configuration applicable to most or to all hosts in the organization.

Cloud-init configuration is defined in Terraform source code, then cloud-init is executed upon system boot.

During cloud-init execution on each machine, Puppet is run using the supplied cloud-init configuration so that the machine can configure itself.

Pros

Having to ship AMI updates is not something that is done frequently. Puppet is largely the only thing that needs to be tested, and configuration changes can be rolled out without any infrastructure changes in Terraform or otherwise.

Cons

Instances are stateful. Before deploying changes to existing infrastructure, all changes in Puppet need to be tested both from scratch on new machines in addition to testing on machines with previous versions of configuration.

Since Puppet runs take time, there is a delay between when the instance has booted and when it is available to start serving traffic. Additionally, there is a chance that machine provisioning can fail for a variety of reasons, in turn creating hosts that cannot serve traffic and which need to be replaced.

Mostly Static

Description

In this strategy, an AMI is created with all dependencies required by the host. Since application code and system configuration is baked into the image, the instance boots with all the prerequisites present.

As machines boot, cloud-init is executed and does a per-host configuration. There is no runtime configuration beyond what happens in the cloud-init stage for the initial configuration. Since this has an immutable infrastructure model, servers are never updated or modified after they are deployed.

Pros

Since instances are mostly stateless, only AMI and cloud-init modifications result in deploying new infrastructure. The AMI does not need to be changed if changes are made in cloud-init, and vice-versa.

Cons

As in the fully static strategy’s case, each time configuration or code changes need to be done,, new instances need to be deployed. If either the AMI or the cloud-init configuration needs to change, new instances likewise need to be deployed.

Fully Static

Description

The AMI created has everything needed at runtime. Terraform manages less configuration than other approaches. No cloud-init or runtime configuration is utilized.

Pros

Instances are completely stateless and when they come online, they can serve traffic immediately, with no time spent on per-machine configuration. Puppet can be used to build the image with all batteries included at the AMI creation time itself.

Cons

Changing any part of the code or infrastructure requires an AMI bake and a replacement of all infrastructure using that particular AMI. Each AMI bake operation usually takes not less than ten minutes, but large-scale server replacement can be rolled out relatively quickly.

Conclusion

Since there are numerous tools with many capabilities available, it is prudent that the right tool is selected for each part of the job. Terraform was built and designed to manage infrastructure, while tools like Puppet, Chef, Salt, and Ansible are best utilized to manage AMI creation, OS configuration, and application deployments.

Related posts