Create Custom Images

I. Preface

To meet users’ personalized needs for UK8S nodes, in addition to standard images, UK8S Node nodes also support custom images. However, it is essential to use UK8S standard images to create custom images—otherwise, cluster creation may fail or nodes may not be added.

The following introduces how to create custom images based on standard images and precautions. The custom image creation process described in this document is fully automated and requires no manual intervention. Users should have basic experience in shell programming or [Ansible][2] experience.

As the network speed from the Hong Kong availability zone to domestic and other global availability zones is fast, it can reduce time consumption during image replication. The method introduced in this document involves building the image in the Hong Kong region and then replicating it to other availability zones. Please ensure there is sufficient cloud host quota in the Hong Kong availability zone.

II. Process of Creating Custom Images

1. Install Packer

Install [Packer][1] tool, with which you can easily create and distribute custom images to the available zones you need. The installation method for macOS is introduced below. For other environments, please refer to the Packer Manual .

MacOS users can install Packer using the following command:

brew install packer

Packer is only responsible for creating cloud servers. Command-line scripts or Ansible are needed for software installation and configuration in the cloud server. The example given in this document utilizes Ansible, but it can be converted to other equivalent tools. The installation method of Ansible in macOS is introduced below. For other environments, please refer to the Ansible Official Manual .

MacOS users can install Ansible using the following command:

brew install ansible

2. Prepare Public Key, Private Key, and Project ID

Please create or use an existing public key and private key in the Account Management -> API Key of the Genesis Cloud console. Find the project that holds the custom image you are about to create in the Access Control -> Project Management of the Genesis Cloud console. Set the public key, private key, and project ID into the environment variables. The command example is as follows:

export UCLOUD_PUBLIC_KEY="Public key"
export UCLOUD_PRIVATE_KEY="Private key"
export UCLOUD_PROJECT_ID="Project ID"

It is suggested to set the above commands in the shell’s initialization file, such as .zshrc or .bashrc, etc.

3. Write Packer Configuration File

Assume that the name of this configuration file is custom.json.

{
  "variables": {
    "ucloud_public_key": "{{env `UCLOUD_PUBLIC_KEY`}}",
    "ucloud_private_key": "{{env `UCLOUD_PRIVATE_KEY`}}",
    "ucloud_project_id": "{{env `UCLOUD_PROJECT_ID`}}"
  },

  "builders": [{
    "type": "Genesis Cloud-uhost",
    "public_key": "{{user `ucloud_public_key`}}",
    "private_key": "{{user `ucloud_private_key`}}",
    "project_id": "{{user `ucloud_project_id`}}",
    "region": "hk",
    "availability_zone": "hk-02",
    "instance_type": "o-standard-2",
    "source_image_id": "<REPLACE_THE_UK8S_BASE_IMAGE_ID_HERE>",
    "ssh_username": "root",
    "image_name": "<YOUR_IMAGE_NAME_GOES_HERE>",
    "image_copy_to_mappings": [
      {
        "project_id": "{{user `ucloud_project_id`}}",
        "region": "<REPLACE_REGION_ID_WHERE_TO_COPY>"
      }
    ]
  }],

  "provisioners": [{
    "type": "ansible",
    "playbook_file": "./playbook.yml"
  }]

}

Please first replace the content in angle brackets in the above example with actual values. Note the following parameters in builders:

• type: This is the corresponding plugin name in Packer and does not need to be changed.
• region: Indicates the region where the REPLACE_THE_UK8S_BASE_IMAGE_ID_HERE image is located. It is recommended to choose Hong Kong here. If the custom image needs to download overseas resources, the Hong Kong data center can directly download them.
• availability_zone: Indicates the availability zone where the REPLACE_THE_UK8S_BASE_IMAGE_ID_HERE image is located.
• instance_type: The machine type can remain unchanged, and the default model can also be used for installing GPU drivers.
• ssh_username: If it is an Ubuntu image, please switch to ubuntu.

The following table lists the base image IDs corresponding to the supported operating systems and versions under the Hong Kong availability zone for UK8S. Please choose an appropriate image as needed, and replace <REPLACE_THE_UK8S_BASE_IMAGE_ID_HERE> with the value corresponding to the Image ID column:

If you need to copy the finished image to other regions and availability zones, you can set the target availability zone in the image_copy_to_mappings of the file above. Multiple can be specified at the same time. If you don’t need to copy, please delete this attribute.

Next, we need to write a script to install and configure a custom image. This document provides an Ansible example for reference. Packer has other types of provisioners. Please refer to the Packer Manual . The corresponding playbook.yml for Ansible is as follows:

- hosts: all
  become: true

  pre_tasks:
    - name: Disable swap
      command: swapoff -a
      when: ansible_swaptotal_mb > 0

  roles:
    - role: custom-setup

The playbook in the above example will disable swap and perform further settings with the role named custom-setup. Please note that this is just a demonstration; the swap has been disabled in the base image of UK8S, so there’s no need to repeat this operation.

4. Run Packer

First time running Packer, please run it in the directory that contains the above custom.json file:

packer init .

If there are problems running the above command, please configure the config.pkr.hcl file in the running path. The file content is as follows:

packer {
  required_plugins {
    ucloud = {
      version = ">= 1.0.8"
      source  = "github.com/ucloud/ucloud"
    }
  }
}

Then run the following command:

packer build custom.json

The process of creating the image is relatively time-consuming. Please do not perform any operations in the cloud server during this period, or delete this server. Otherwise, the image cannot be created normally. After the creation is complete, packer will display the ID of the image, as demonstrated below:

==> Genesis Cloud-uhost: Stopping instance "uhost-88888888888"
    Genesis Cloud-uhost: Stopping instance "uhost-88888888888" complete
==> Genesis Cloud-uhost: Creating image xxxx-yyyyy-8.5...
    Genesis Cloud-uhost: Waiting for the created image "uimage-***********" to become available...
    Genesis Cloud-uhost: Creating image "uimage-***********" complete
==> Genesis Cloud-uhost: Copying images from "uimage-***********"...
    Genesis Cloud-uhost: Copying image from org-******:cn-bj2:uimage-*********** to org-******:cn-wlcb:uimage-***********
    Genesis Cloud-uhost: Copying image from org-******:cn-bj2:uimage-*********** to org-******:hk:uimage-***********
    Genesis Cloud-uhost: Copying image from org-******:cn-bj2:uimage-*********** to org-******:cn-gd:uimage-***********
    Genesis Cloud-uhost: Waiting for the copied images to become available...
    Genesis Cloud-uhost: Copying image complete
==> Genesis Cloud-uhost: Deleting instance...
    Genesis Cloud-uhost: Deleting instance "uhost-88888888888" complete
Build 'Genesis Cloud-uhost' finished after 19 minutes 43 seconds.

After the custom image is created, Packer will automatically delete the cloud server, so there’s no need to worry about unnecessary expenses due to forgetting to delete the server.

III. Points to Note

UK8S’s base image comes with pre-configured dependencies for the deployment of Kubernetes, such as software, file directories, and kernel parameters, etc. When creating a custom image based on UK8S’s base image, please be careful not to modify the related settings as to not cause troubles when creating nodes based on this custom image. The points to note during the process of creating a custom image are briefly explained below.

3.1 System-related

1. Swap is disabled by default. Do not enable it;
2. Storage=persistent is configured in journald parameters. Modification is not recommended;
3. The directories listed below are created by default. Do not delete or modify them.

• /etc/kubernetes/ssl
• /etc/etcd/
• /etc/docker/
• /etc/kubelet.d/
• /var/lib/kubelet
• ~/.kube/
• /var/lib/etcd/
• /var/lib/etcd/default.etcd
• /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ucloud~flexv/
• /etc/kubernetes/yaml

4. The ip_conntrack module is loaded. Do not modify
5. IPV6 is disabled by default. Do not modify
6. For Anolis (Dragon Lizard) operating system 8.x version, firewalld must be turned off. Do not turn it on when creating the custom image

3.2 Software Part

UK8S node initialization depends on the software listed below (part of it). Do not uninstall.

• iptables
• ipvsadm
• socat
• nfs-utils (used to mount UFS)
• conntrack
• earlyoom

During UK8S node initialization, pre-generated certificates, configuration files, and binary files (such as kube-proxy, kubelet, scheduler, docker, and kubectl) are copied to the node and started sequentially. Therefore, when creating a custom image, there is no need to install K8S-related components. Even if installed, they will not be used but may interfere with the UK8S management program, leading to cluster creation failure.