Restructure codebase to separate bolt plans/tasks by distro

README.md

@@ -117,12 +117,12 @@ Once the `xorriso` command is run successfully, the `ubuntu-22.04-autoinstall.is
 
 ## Creating the VM using Bolt
 
-Update the parameters provided to the below plan run command as needed.
+Update the parameters provided to the below plan run command as needed. Note that you should always run plans and tasks out of the `bolt` directory.
 
 ```bash
-cd bolt_vm_automation
-bolt plan run bolt_vm_automation::create_ubuntu \
-  target=vortex \
+cd bolt
+bolt plan run ubuntu::create_vm \
+  target_host=vortex \
   vm_name=moeny-bank01 \
   hostname=moeny-bank01
   ip_with_cidr=100.40.223.189/24 \
@@ -130,12 +130,12 @@ bolt plan run bolt_vm_automation::create_ubuntu \
 
 ## Alpine VMs
 
-There are now separate plans for generating a VM using Alpine and Ubuntu. [create_alpine](bolt_vm_automation/plans/create_alpine.yaml) should be run for Alpine and [create_ubuntu](bolt_vm_automation/plans/create_ubuntu.yaml) should be run for Ubuntu. These plans each run tasks tailored for the appropriate distribution.
+There are now separate plans for generating a VM using Alpine and Ubuntu. [alpine::create_vm](bolt/vm_automation/alpine/plans/create_vm.yaml) should be run for Alpine and [ubuntu::create_vm](bolt/vm_automation/ubuntu/plans/create_vm.yaml) should be run for Ubuntu. These plans each run tasks tailored for the appropriate distribution.
 
 Below is a sample command to run the Alpine bolt plan.
 
 ```bash
-bolt plan run bolt_vm_automation::create_alpine \
+bolt plan run alpine::create_vm \
   vm_name=moeny-service \
   hostname=moeny-service \
   ip_with_cidr=100.40.223.189/24 \
@@ -143,29 +143,30 @@ bolt plan run bolt_vm_automation::create_alpine \
   dns_hostname=service
 ```
 
-Note that `add_a_record_bool` must be set to `true` if you would like an A record for the VM to be added to the DNS server zone file, as it is `false` by default. If using this functionality, `dns_hostname` should also be provided and optionally `dns_ttl` if you do not want the default of `3600`. The ability to interact with the DNS server depends on having set up a TSIG key on your DNS server for dynamic updates and storing a copy of your `tsig.key` file in a directory called `keys` at the root of this project. If either of these conditions have not been met, do not attempt to use this functionality. For more information on setting up dynamic DNS with a TSIG key, see our [bind9](https://gitea.moeny.ai/moeny/bind9) repo.
+Note that `add_a_record_bool` must be set to `true` if you would like an A record for the VM to be added to the DNS server zone file, as it is `false` by default. If using this functionality, `dns_hostname` should also be provided and optionally `dns_ttl` if you do not want the default of `3600`. The ability to interact with the DNS server depends on having set up a TSIG key on your DNS server for dynamic updates and storing a copy of your `tsig.key` file in a directory called `keys` at the root of the bolt project, alongside `bolt-project.yaml`. If either of these conditions have not been met, do not attempt to use this functionality. For more information on setting up dynamic DNS with a TSIG key, see our [bind9](https://gitea.moeny.ai/moeny/bind9) repo.
 
 Similarly, `install_docker_bool` can be set to `false` if you do not want docker to be installed on the VM. It is true by default.
 
 For more detailed logging on the `bolt plan run` add the `-v` flag at the end of the command.
 
-If you want to delete an A record that you have added, you can use the [`delete_dns_a_record`](bolt_vm_automation/tasks/delete_dns_a_record.sh) task. You'll just need to provide it with the dns_hostname you set. Here's a sample command.
+If you want to delete an A record that you have added, you can use the [`delete_dns_a_record`](bolt/vm_automation/common/tasks/delete_dns_a_record.sh) task. You'll just need to provide it with the dns_hostname you set. Here's a sample command.
 
 ```bash
-bolt task run bolt_vm_automation::delete_dns_a_record dns_hostname=service --targets localhost
+bolt task run common::delete_dns_a_record dns_hostname=service --targets localhost
 ```
 
-Lastly, even though it is designed to be run with the `create_alpine` plan, you can also run the [`add_dns_a_record`](bolt_vm_automation/tasks/add_dns_a_record.sh) task on its own. You'll just need to provide it a few parameters. Here's a sample command.
+Lastly, even though it is designed to be run with the `create_alpine` plan, you can also run the [`add_dns_a_record`](bolt/vm_automation/common/tasks/add_dns_a_record.sh) task on its own. You'll just need to provide it a few parameters. Here's a sample command.
 
 ```bash
-bolt task run bolt_vm_automation::add_dns_a_record add_a_record_bool=true ip_with_cidr=100.40.223.189/24 dns_hostname=service dns_ttl=3600 --targets localhost
+bolt task run common::add_dns_a_record add_a_record_bool=true ip_with_cidr=100.40.223.189/24 dns_hostname=service dns_ttl=3600 --targets localhost
 ```
 
 Alternatively, to update DNS with the `nsupdate` command directly from the terminal, run something like the following with the path to your `tsig.key`:
 
 ```bash
-nsupdate -k ../../keys/tsig.key << EOF
+nsupdate -k ./keys/tsig.key << EOF
 server ns1.moeny.ai
+debug yes
 zone moeny.ai
 update add service.moeny.ai 3600 A 6.5.2.5
 send
@@ -179,7 +180,7 @@ ssh moeny@ns1.moeny.ai "sudo rndc sync moeny.ai"
 In order to spin up VMs on an internal network, you will need to generate an Alpine iso compatible with the internal IPs you are using and specify its path. You will also want to declare the staging IP and gateway IP parameters accordingly. Here is a sample command to run the Alpine bolt plan.
 
 ```bash
-bolt plan run bolt_vm_automation::create_alpine \
+bolt plan run alpine::create_vm \
   vm_name=moeny-service-alpine \
   hostname=moeny-service-alpine \
   network=internal-moeny \
@@ -192,12 +193,12 @@ bolt plan run bolt_vm_automation::create_alpine \
 Similarly, a new Ubuntu iso will need to be generated that is compatible with the internal IPs. This can be done by simply updating the `user-data` file from Step 6 to have the proper network configuration, as in [`user-data-internal`](user-data-internal.yaml). Here is a sample command to run the Ubuntu bolt plan.
 
 ```bash
-bolt plan run bolt_vm_automation::create_ubuntu \
+bolt plan run ubuntu::create_vm \
   vm_name=moeny-service-ubuntu \
   hostname=moeny-service-ubuntu \
   network=internal-moeny \
   ip_with_cidr=10.44.0.20/24 \
   gateway_ip=10.44.0.1 \
   iso_path=/mnt/nfs/kvm-isos/iso-build/ubuntu-22.04-autoinstall-internal_moeny.iso \
-  staging_ip=10.44.0.250 -v
+  staging_ip=internal -v
 ```

bolt/bolt-project.yaml

@@ -0,0 +1,4 @@
+---
+name: vm_automation
+modulepath:
+  - vm_automation

bolt/vm_automation/alpine/plans/create_vm.yaml

@@ -1,21 +1,67 @@
 ---
+# Plan to Create an Alpine VM (alpine::create_vm)
 parameters:
   target_host:
     type: String
     description: "Target host to create the VM on"
     default: "vortex"
-  target_vm:
-    type: String
-    description: "Target VM for post-installation tasks"
-    default: $vm_name
-  iso_path:
-    type: String
-    description: "Path to the ISO file"
-    default: "/mnt/nfs/kvm-isos/iso-build/alpine-autoinstall-wan_verizon.iso"
+  ## Main Configurations
   vm_name:
     type: String
     description: "Name of the VM"
     default: "vm-template-staging"
+  # Network Configuration
+  hostname:
+    type: String
+    description: "Hostname of the VM"
+    default: "vm-template-staging"
+  network:
+    type: String
+    description: "Network to connect the VM to"
+    default: "wan-verizon"
+  ip_with_cidr:
+    type: String
+    description: "Public IP of the VM"
+    default: "100.40.223.190/24"
+  gateway_ip:
+    type: String
+    description: "Gateway IP for the VM"
+    default: "100.40.223.1"
+  # Define Based on Whether Public or Internal VM
+  iso_path:
+    type: String
+    description: "Path to the ISO file"
+    default: "/mnt/nfs/kvm-isos/iso-build/alpine-autoinstall-wan_verizon.iso"
+  staging_ip:
+    type: String
+    description: "Staging IP"
+    default: "100.40.223.190"
+  ## Optional Configurations
+  # Zabbix
+  install_zabbix_bool:
+    type: Boolean
+    description: "Whether to install Zabbix on the VM"
+    default: true
+  # Docker
+  install_docker_bool:
+    type: Boolean
+    description: "Whether to install Docker on the VM"
+    default: true
+  # DNS
+  add_a_record_bool:
+    type: Boolean
+    description: "Whether to add a DNS A record for the VM"
+    default: false
+  dns_hostname:
+    type: String
+    description: "Hostname for the DNS A record"
+    default: "vm-template-staging"
+  dns_ttl:
+    type: Integer
+    description: "TTL for the DNS A record"
+    default: 3600
+  ## Rarely Changed Configurations
+  # VM Specifications
   ram:
     type: Integer
     description: "Amount of RAM in MB"
@@ -32,33 +78,15 @@ parameters:
     type: String
     description: "Base path for disk images"
     default: "/mnt/nfs/moeny-images"
-  network:
-    type: String
-    description: "Network to connect the VM to"
-    default: "wan-verizon"
   os_variant:
     type: String
     description: "OS variant for the VM"
     default: "alpinelinux3.20"
-  ip_with_cidr:
-    type: String
-    description: "Public IP of the VM"
-  staging_ip:
-    type: String
-    description: "Staging IP"
-    default: "100.40.223.190"
-  hostname:
-    type: String
-    description: "Hostname of the VM"
-    default: "vm-template-staging"
+  # Rarely Changed Network Configuration
   dhcp:
     type: Boolean
     description: "Enable DHCP on the VM"
     default: false
-  gateway_ip:
-    type: String
-    description: "Gateway IP for the VM"
-    default: "100.40.223.1"
   nameserver1:
     type: String
     description: "Primary nameserver for the VM"
@@ -71,27 +99,17 @@ parameters:
     type: String
     description: "Tertiary nameserver for the VM"
     default: "1.1.1.1"
-  install_docker_bool:
-    type: Boolean
-    description: "Whether to install Docker on the VM"
-    default: true
-  # DNS Variables
-  add_a_record_bool:
-    type: Boolean
-    description: "Whether to add a DNS A record for the VM"
-    default: false
-  dns_hostname:
-    type: String
-    description: "Hostname for the DNS A record"
-    default: "vm-template-staging"
-  dns_ttl:
-    type: Integer
-    description: "TTL for the DNS A record"
-    default: 3600
 
 steps:
+  - name: check_ip_availability
+    description: Check if the target IP is already in use
+    task: common::check_ip_availability
+    targets: localhost
+    parameters:
+      network: $network
+
   - name: create_vm
-    task: bolt_vm_automation::create_alpine
+    task: alpine::create_vm
     targets: $target_host
     parameters:
       iso_path: $iso_path
@@ -105,7 +123,7 @@ steps:
 
   - name: install_alpine
     description: Install Alpine OS on the VM
-    task: bolt_vm_automation::install_alpine
+    task: alpine::install_alpine
     targets: localhost
     parameters:
       vm_name: $vm_name
@@ -115,28 +133,29 @@ steps:
   
   - name: install_packages
     description: Install Packages on the VM
-    task: bolt_vm_automation::install_packages_alpine
+    task: alpine::install_packages
     targets: localhost
     parameters:
       staging_ip: $staging_ip
 
   - name: install_zabbix
     description: Install Zabbix on the VM
-    task: bolt_vm_automation::install_zabbix_alpine
+    task: alpine::install_zabbix
     targets: localhost
     parameters:
+      install_zabbix_bool: $install_zabbix_bool
       staging_ip: $staging_ip
 
   - name: install_docker
     description: Install Docker on the VM
-    task: bolt_vm_automation::install_docker_alpine
+    task: alpine::install_docker
     targets: localhost
     parameters:
       install_docker_bool: $install_docker_bool
       staging_ip: $staging_ip
 
   - name: system_setup
-    task: bolt_vm_automation::system_setup_alpine
+    task: alpine::system_setup
     targets: localhost
     parameters:
       ip_with_cidr: $ip_with_cidr
@@ -150,7 +169,7 @@ steps:
 
   - name: add_dns_a_record
     description: Add a DNS A record for the VM
-    task: bolt_vm_automation::add_dns_a_record
+    task: common::add_dns_a_record
     targets: localhost
     parameters:
       add_a_record_bool: $add_a_record_bool

bolt/vm_automation/alpine/tasks/create_vm.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+# Task to Create an Alpine VM (alpine::create_vm)
 
 # Input Variables
 ISO_PATH=$PT_iso_path

bolt/vm_automation/alpine/tasks/install_alpine.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+# Task to Install Alpine on a VM (alpine::install_alpine)
 
 # Input Variables
 VM_NAME="${PT_vm_name}"

bolt/vm_automation/alpine/tasks/install_docker.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+# Task to Install Docker on an Alpine VM (alpine::install_docker)
 
 # Input Variables
 INSTALL_DOCKER="${PT_install_docker_bool}"

bolt/vm_automation/alpine/tasks/install_packages.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+# Task to Install Packages on an Alpine VM (alpine::install_packages)
 
 # Input Variables
 STAGING_IP="${PT_staging_ip}"

bolt/vm_automation/alpine/tasks/install_zabbix.sh

@@ -1,8 +1,16 @@
 #!/bin/bash
+# Task to Install Zabbix on an Alpine VM (alpine::install_zabbix)
 
 # Input Variables
+INSTALL_ZABBIX="${PT_install_zabbix_bool}"
 STAGING_IP="${PT_staging_ip}"
 
+# Check if Zabbix installation is requested
+if [ "$INSTALL_ZABBIX" != "true" ]; then
+    echo '{"status": "skipped", "message": "Zabbix installation not requested, skipping..."}'
+    exit 0
+fi
+
 # Install zabbix-agent2
 ssh -o StrictHostKeyChecking=no root@${STAGING_IP} "apk add zabbix-agent2"
 

bolt/vm_automation/alpine/tasks/system_setup.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+# Task to Configure the System on Alpine (alpine::system_setup)
 
 # Using Bolt's environment variables
 IP="${PT_ip_with_cidr}"

bolt/vm_automation/common/tasks/add_dns_a_record.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+# This script adds a DNS A record to the DNS server zone file (common::add_dns_a_record)
 
 # Bolt environment variables
 ADD_A_RECORD="${PT_add_a_record_bool}"
@@ -20,8 +21,9 @@ fi
 
 # Create DNS A record
 IP_ADDRESS=$(echo ${IP} | cut -d'/' -f1)
-nsupdate -k ../../keys/tsig.key << EOF
+nsupdate -k "./keys/tsig.key" << EOF
 server ns1.moeny.ai
+debug yes
 zone moeny.ai
 update add ${HOSTNAME}.moeny.ai ${TTL} A ${IP_ADDRESS}
 send

bolt/vm_automation/common/tasks/check_ip_availability.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+# This script checks the availability of an IP address (common::check_ip_availability)
 
 # Extract parameters
 network="$PT_network"

bolt/vm_automation/common/tasks/delete_dns_a_record.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-
+# This script deletes a DNS A record from the DNS server zone file (common::delete_dns_a_record)
 # Bolt environment variables
 HOSTNAME="${PT_dns_hostname}"
 
@@ -10,8 +10,9 @@ if [ -z "$HOSTNAME" ]; then
 fi
 
 # Delete DNS A record
-nsupdate -k ../../keys/tsig.key << EOF
+nsupdate -k "./keys/tsig.key" << EOF
 server ns1.moeny.ai
+debug yes
 zone moeny.ai
 update delete ${HOSTNAME}.moeny.ai A
 send

bolt/vm_automation/ubuntu/plans/create_vm.yaml

@@ -1,21 +1,67 @@
 ---
+# Plan to Create an Ubuntu VM (ubuntu::create_vm)
 parameters:
   target_host:
     type: String
     description: "Target host to create the VM on"
     default: "vortex"
-  staging_ip:
-    type: String
-    description: "IP address of the target VM for post-installation tasks"
-    default: "public"
-  iso_path:
-    type: String
-    description: "Path to the ISO file"
-    default: "/mnt/nfs/kvm-isos/iso-build/ubuntu-22.04-autoinstall-wan_verizon.iso"
+  ## Main Configurations
   vm_name:
     type: String
     description: "Name of the VM"
     default: "vm-template-staging"
+  # Network Configuration
+  hostname:
+    type: String
+    description: "Hostname of the VM"
+    default: "vm-template-staging"
+  network:
+    type: String
+    description: "Network to connect the VM to"
+    default: "wan-verizon"
+  ip_with_cidr:
+    type: String
+    description: "Public IP of the VM"
+    default: "100.40.223.190/24"
+  gateway_ip:
+    type: String
+    description: "Gateway IP for the VM"
+    default: "100.40.223.1"
+  # Define Based on Whether Public or Internal VM
+  iso_path:
+    type: String
+    description: "Path to the ISO file"
+    default: "/mnt/nfs/kvm-isos/iso-build/ubuntu-22.04-autoinstall-wan_verizon.iso"
+  staging_ip:
+    type: String
+    description: "Target VM for post-installation tasks as either public or internal"
+    default: "public"
+  ## Optional Configurations
+  # Zabbix
+  install_zabbix_bool:
+    type: Boolean
+    description: "Whether to install Zabbix on the VM"
+    default: true
+  # Docker
+  install_docker_bool:
+    type: Boolean
+    description: "Whether to install Docker on the VM"
+    default: true
+  # DNS
+  add_a_record_bool:
+    type: Boolean
+    description: "Whether to add a DNS A record for the VM"
+    default: false
+  dns_hostname:
+    type: String
+    description: "Hostname for the DNS A record"
+    default: "vm-template-staging"
+  dns_ttl:
+    type: Integer
+    description: "TTL for the DNS A record"
+    default: 3600
+  ## Rarely Changed Configurations
+  # VM Specifications
   ram:
     type: Integer
     description: "Amount of RAM in MB"
@@ -32,30 +78,15 @@ parameters:
     type: String
     description: "Base path for disk images"
     default: "/mnt/nfs/moeny-images"
-  network:
-    type: String
-    description: "Network to connect the VM to"
-    default: "wan-verizon"
   os_variant:
     type: String
     description: "OS variant for the VM"
     default: "ubuntu22.04"
-  ip_with_cidr:
-    type: String
-    description: "Public IP of the VM"
-    default: "100.40.223.190/24"
-  hostname:
-    type: String
-    description: "Hostname of the VM"
-    default: "vm-template-staging"
+  # Rarely Changed Network Configuration
   dhcp:
     type: Boolean
     description: "Enable DHCP on the VM"
     default: false
-  gateway_ip:
-    type: String
-    description: "Gateway IP for the VM"
-    default: "100.40.223.1"
   nameserver1:
     type: String
     description: "Primary nameserver for the VM"
@@ -72,13 +103,13 @@ parameters:
 steps:
   - name: check_ip_availability
     description: Check if the target IP is already in use
-    task: bolt_vm_automation::check_ip_availability
+    task: common::check_ip_availability
     targets: localhost
     parameters:
       network: $network
 
   - name: create_vm
-    task: bolt_vm_automation::create_ubuntu
+    task: ubuntu::create_vm
     targets: $target_host
     parameters:
       iso_path: $iso_path
@@ -92,16 +123,20 @@ steps:
 
   - name: install_zabbix
     description: Install Zabbix on the VM
-    task: bolt_vm_automation::install_zabbix_ubuntu
+    task: ubuntu::install_zabbix
     targets: $staging_ip
-    
+    parameters:
+      install_zabbix_bool: $install_zabbix_bool
+
   - name: install_docker
     description: Install Docker on the VM
-    task: bolt_vm_automation::install_docker_ubuntu
+    task: ubuntu::install_docker
     targets: $staging_ip
+    parameters:
+      install_docker_bool: $install_docker_bool
 
   - name: system_setup
-    task: bolt_vm_automation::system_setup_ubuntu
+    task: ubuntu::system_setup
     targets: $staging_ip
     parameters:
       ip_with_cidr: $ip_with_cidr
@@ -112,5 +147,15 @@ steps:
       nameserver2: $nameserver2
       nameserver3: $nameserver3
 
+  - name: add_dns_a_record
+    description: Add a DNS A record for the VM
+    task: common::add_dns_a_record
+    targets: localhost
+    parameters:
+      add_a_record_bool: $add_a_record_bool
+      ip_with_cidr: $ip_with_cidr
+      dns_hostname: $dns_hostname
+      dns_ttl: $dns_ttl
+
 return:
   message: "VM ${vm_name} created and updated successfully!"

bolt/vm_automation/ubuntu/tasks/create_vm.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+# Task to Create an Ubuntu VM (ubuntu::create_vm)
 
 # Input Variables
 ISO_PATH=$PT_iso_path

bolt/vm_automation/ubuntu/tasks/install_docker.sh

@@ -1,4 +1,15 @@
 #!/bin/bash
+# Task to Install Docker on Ubuntu (ubuntu::install_docker)
+
+# Input Variables
+INSTALL_DOCKER="${PT_install_docker_bool}"
+
+# Check if Docker installation is requested
+if [ "$INSTALL_DOCKER" != "true" ]; then
+    # Output JSON that Bolt will understand
+    echo '{"status": "skipped", "message": "Docker installation not requested, skipping..."}'
+    exit 0
+fi
 
 # Update package list and install prerequisites
 sudo apt-get update

bolt/vm_automation/ubuntu/tasks/install_zabbix.sh

@@ -1,4 +1,14 @@
 #!/bin/bash
+# Task to Install Zabbix on Ubuntu (ubuntu::install_zabbix)
+
+# Input Variables
+INSTALL_ZABBIX="${PT_install_zabbix_bool}"
+
+# Check if Zabbix installation is requested
+if [ "$INSTALL_ZABBIX" != "true" ]; then
+    echo '{"status": "skipped", "message": "Zabbix installation not requested, skipping..."}'
+    exit 0
+fi
 
 # Download the Zabbix release package
 sudo wget -O /tmp/zabbix-release.deb https://repo.zabbix.com/zabbix/7.2/release/ubuntu/pool/main/z/zabbix-release/zabbix-release_latest_7.2+ubuntu22.04_all.deb

bolt/vm_automation/ubuntu/tasks/system_setup.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+# Task to Configure the System on Ubuntu (ubuntu::system_setup)
 
 # Using Bolt's environment variables
 IP="${PT_ip_with_cidr}"

bolt_vm_automation/bolt-project.yaml

@@ -1,2 +0,0 @@
----
-name: bolt_vm_automation