Custom OS Configuration in BOSH

Others by whlee

Overview

BOSH is a powerful tools in provisioning and deploying software over VMs consistently. BOSH will perform monitoring, failure recovery and software updates with zero-to-minimal downtime. With BOSH, application platform like VMware Tanzu Application Service (formerly Pivotal Platform Application Service) and Pivotal Container Service (PKS) can be deployed in a consistent manner.

However, sometimes there might be custom OS configuration required. BOSH os-conf release is developed to enable configuration of Linux OS, for example:

  • Customize login banner text
  • Add UNIX users to VM
  • Add system wide CA Certificate
  • Configure resolv.conf search domain
  • Change TCP keepalive kernel args
  • Apply arbitrary sysctls

In this example, we will create a custom SSH login user for Master and Worker nodes in BOSH deployed PKS cluster. There are various use case that can utilize os-conf BOSH release including creating temporary service account for vulnerable scanning (VA scanning) or anti-virus agent and applying custom login banner to OS.

Steps to Enable Custom SSH Login User

Steps for creating a custom SSH login for BOSH-deployed VMs is defined as below:-

  1. Download os-conf BOSH release into jumphost (Pivotal Operation Manager in this guide)
  2. Create os-conf BOSH release tarball
  3. Upload BOSH release tarball to BOSH director
  4. Create SSH key pair for VMs access
  5. Create runtime-config that fixed to BOSH deployment
  6. Run update runtime-config on BOSH

Download os-conf BOSH release

Release and source code for os-conf BOSH release can be obtained at cloudfoundry/os-conf-release. First, in jumphost (Pivotal Operation Manager in this example), we will clone os-conf source code into OpsMan directory and navigate to the directory of os-conf-release.

git clone https://github.com/cloudfoundry/os-conf-release.git
cd os-conf-release

Create os-conf BOSH release tarball

Next, we will package and create the os-conf BOSH release tarball in jumphost based on the required version of release (version 21.0.0 in this example).

While creating BOSH release, some additional packages is needed to be downloaded from internet. Do run the bosh create-release command on jumphost that has internet access. Result of the command line will be os-conf release tarball created in .tgz format. 

bosh create-release releases/os-conf/os-conf-<VERSION>.yml --tarball os-conf-<VERSION>.tgz
Sample output screenshot of bosh create-release command

Upload Tarball Release to BOSH Director

Tarball for os-conf release created in previous step will then be uploaded to BOSH director using bosh upload-release command. Wait for BOSH director to extract, verify and create new release. Verify that os-conf release has been uploaded to BOSH director using bosh releases command. Ensure the correct version uploaded to BOSH director.

bosh -e <BOSH_ENV> upload-release os-conf-<VERSION>.tgz
bosh -e <BOSH_ENV> releases
Sample output screenshot of bosh upload-release command
Sample output screenshot of bosh releases command

Generate SSH Key Pair

Adding OS user to BOSH-deployed VMs required credentials to be created in the first place. In os-conf BOSH release, credentials can be added to user is two ways:-

  • Username with Password
  • Username with SSH private and public key pair

In this example, we will be adding user to BOSH-deployed VMs using username with SSH private and public key pair which is a more secure way of passing OS credentials. We will create the SSH private and public key pair in this step by using ssh-keygen command. Note that other method of generating SSH key like PuTTYgen will also work. 

ssh-keygen
Sample output screenshot of ssh-keygen command
Verify SSH public and private key pair created in desired directory

Create Runtime Config YAML File

Runtime config is a way that BOSH used to update or define IaaS agnostic configuration that applies to deployments. Based on created runtime config YAML file, BOSH director will apply the changes to specified deployment during the next bosh deploy cycle.

Addons block in runtime config can be used to specify placement rules for required BOSH-deployed VMs. Placement rules is defined using include (inclusion placement rules) or exclude (exclusion placement rules). Available rules for the addons directives are:

  • stemcell
    • os – Based on stemcell’s operating system
  • deployments – Based on deployment names
  • jobs
    • name – Based on job names
    • release – Based on job release names
  • instance_groups – Based on instance group names
  • lifecycle – Based on lifecycle type, either service or errand
  • networks – Based on network names
  • teams – Based on team names

Reference: Placement rules for BOSH runtime-config

releases:
- name: os-conf                         # Release name
  version: 21.0.0                       # Release version

addons:
- name: os-conf
  jobs:
  - name: user_add
    release: os-conf
    properties:
      persistent_homes: true            # If set to true, user home directories are created on persistent storage in /var/vcap/store/home
      users:
      - name: va-scan
        public_key: "<PUBLIC_SSH_KEY>"  # Add in Public SSH key created
        shell: /bin/bash                # OPTIONAL: Defaults to `/bin/bash`
        sudo: true                      # OPTIONAL: Defaults to `true`
  include:
    deployments: [<DEPLOYMENT_ID>]      # Add in Deployment ID obtain using bosh deployments command

Update Runtime Config in BOSH Director

After compiling of the runtime config YAML file with parameters suited to use case (applied to one PKS cluster deployment in this example), run bosh update-runtime-config command to apply runtime config to BOSH director. Run bosh configs to verify if runtime config updated in BOSH director.

bosh -e <BOSH_ENV> update-runtime-config --name=create_ssh_user ./runtime-config.yml
bosh -e <BOSH_ENV> configs
Sample output screenshot of bosh update-runtime-config command
Sample output screenshot of bosh configs command

Once verified runtime config updated in BOSH director, apply changes to deployment by running apply changes in Pivotal Operation Manager (OpsMan) or trigger update of PKS cluster using pks update-cluster command. On OpsMan, run bosh tasks to check status of rolling update on deployment and run bosh vms to check on status of deployment once rolling update of VMs completed by BOSH director.

pks login -a <PKS_API_FQDN> -u <USERNAME> -p <PASSWORD>
pks update-cluster <CLUSTER_NAME> --num-nodes <NUMBER_OF_NODES>

bosh -e <BOSH_ENV> task <TASK_ID>
bosh -e <BOSH_ENV> -d <DEPLOYMENY_ID> vms

After rolling update on BOSH-deployed VMs (which are PKS nodes in this example) completed, BOSH-deployed VMs can be access using SSH username and SSH private key created in previous step.

ssh -i <PRIVATE_KEY_FILE_NAME> <USER_NAME>@<VM_IP>
Sample output screenshot of ssh login successful to PKS Master node

Conclusion

With this, we have successfully apply os-conf BOSH release to the deployment. External user will now be able to SSH access to BOSH-deployed VMs using SSH private key and/or password created and defined.

Reference

Leave A Comment