The Container Service Extension 4.0 has been launched with a number of vital enhancements and extra use instances, together with Cluster API, lifecycle administration by way of a person interface, GPU assist for Kubernetes clusters, and integration with VMware Cloud Director as infrastructure. With its feature-rich person interface, clients can carry out operations corresponding to creation, scaling, and upgrading on Tanzu Kubernetes clusters. Nonetheless, some clients could search automation assist for these identical operations.
This weblog submit is meant for purchasers who need to automate the provisioning of Tanzu Kubernetes clusters on the VMware Cloud Director Tenant portal utilizing the VMware Cloud Director API. Though the VCD API is supported, the weblog submit is important as a result of the Cluster API is used to create and handle TKG clusters on VCD. The payload required to carry out operations on TKG clusters requires some work to offer the Cluster API-generated payload. The weblog submit outlines the step-by-step course of for producing the proper payload for purchasers utilizing their VCD infrastructure.
Model Help:
This API information is relevant to clusters created by CSE 4.0 and CSE 4.0.1 Tanzu Kubernetes Clusters.
The present stipulations for purchasers to create TKG clusters of their organizations additionally apply to the automation circulate. These stipulations are summarized right here and will be discovered within the official documentation to onboard Supplier and Tenant Admin customers. The next sections present an summary of the necessities for each cloud supplier directors and Tenant Admin customers.
Cloud Supplier Admin Steps
The Steps to onboard the purchasers is demonstrated on this video and documented right here. As soon as buyer group and its customers are onboarded, they will use subsequent part to make use of APIs, or eat it to create automated Cluster operations.
As a fast abstract following steps are anticipated to be carried out by cloud supplier to onboard and put together the shopper:
- Overview Interoperability Matrix to assist Container Service Extension 4.0 and 4.0.1
- Enable vital communication for CSE server
- Begin CSE server and Onboard buyer group (Reference Demo and Official Documentation)
Buyer Org Admin Steps
When the cloud supplier has onboarded the shopper onto the Container Service Extension, the group administrator should create and assign customers with the potential to create and handle TKG clusters for the shopper group. This documentation outlines the process for making a person with the “Kubernetes cluster writer” position throughout the tenant group.
It’s then assumed that the person “acmekco” has obtained the required assets and entry throughout the buyer group to execute Kubernetes cluster operations.
Generate ‘capiyaml’ payload
- Gather VCD Infrastructure and Kubernetes Cluster particulars
This Operation requires following info for VCD tenant portal. The proper column describes instance values used as reference on this weblog submit.
| Enter | Instance worth for this weblog |
| VCD_SITE | VCD Tackle (https://vcd-01a.native) |
| VCD_ORGANIZATION | Buyer Group title(ACME) |
| VCD_ORGANIZATION_VDC | Buyer OVDC title (ACME_VDC_T) |
| VCD_ORGANIZATION_VDC_NETWORK | Community title in buyer org (172.16.2.0) |
| VCD_CATALOG | CSE shared catalog title (cse) |
| Enter | Instance worth for this weblog |
| VCD_TEMPLATE_NAME | Kubernetes and TKG model of the cluster(Ubuntu 20.04 and Kubernetes v1.22.9+vmware.1) |
| VCD_CONTROL_PLANE_SIZING_POLICY | Sizing coverage of management aircraft vms(TKG small) |
| VCD_CONTROL_PLANE_STORAGE_PROFILE | Storage profile for management aircraft of the cluster (Capability) |
| VCD_CONTROL_PLANE_PLACEMENT_POLICY | Elective – Depart empty if not utilizing |
| VCD_WORKER_SIZING_POLICY | Sizing coverage of employee nodes vms(TKG small) |
| VCD_WORKER_PLACEMENT_POLICY | Elective – Depart empty if not utilizing |
| VCD_WORKER_STORAGE_PROFILE | Storage profile for management aircraft of the cluster (Capability) |
| CONTROL_PLANE_MACHINE_COUNT | 1 |
| WORKER_MACHINE_COUNT | 1 |
| VCD_REFRESH_TOKEN_B64 | “MHB1d0tXSllVb2twU2tGRjExNllCNGZnVWZqTm5UZ2U=” Ref VMware Doc to Generate token earlier than reworking it to Base64 |
- Set up required instruments to generate the
capiyaml. Consumer can use any Working System or a Digital Machine(together with Linux, Mac or Home windows) to generate the payload. - As soon as the tenant person has collected all the data, person must set up following elements corresponding to Clusterctl 1.1.3, Form(0.17.0), and Docker (20.10.21) on finish person’s machine. The next step requires above collected info, and never the entry to VCD Infrastructure to generate capiyaml payload.
- Copy TKG CRS Recordsdata domestically. Incase the TKG model is lacking from the folder, be sure to have the templates created for the specified TKG variations. The Following desk gives supported listing of and many others, coredns, tkg, tkr variations for CSE 40 and CSE 4.0.1 launch. Alternatively this script to fetch the identical values from Tanzu Kubernetes Grid assets.
| Kubernetes Model | Etcd ImageTag | CoreDNS ImageTag | Full Distinctive Model | OVA | TKG Product Model | TKr model |
| v1.22.9+vmware.1 | v3.5.4_vmware.2 | v1.8.4_vmware.9 | v1.22.9+vmware.1-tkg.1 | ubuntu-2004-kube-v1.22.9+vmware.1-tkg.1-2182cbabee08edf480ee9bc5866d6933.ova | 1.5.4 | v1.22.9—vmware.1-tkg.1 |
| v1.21.11+vmware.1 | v3.4.13_vmware.27 | v1.8.0_vmware.13 | v1.21.11+vmware.1-tkg.2 | ubuntu-2004-kube-v1.21.11+vmware.1-tkg.2-d788dbbb335710c0a0d1a28670057896.ova | 1.5.4 | v1.21.11—vmware.1-tkg.3 |
| v1.20.15+vmware.1 | v3.4.13_vmware.23 | v1.7.0_vmware.15 | v1.20.15+vmware.1-tkg.2 | ubuntu-2004-kube-v1.20.15+vmware.1-tkg.2-839faf7d1fa7fa356be22b72170ce1a8.ova | 1.5.4 | v1.20.15—vmware.1-tkg.2 |
|
|
mkdir ~/infrastructure–vcd/ cd ~/infrastructure–vcd mkdir v1.0.0 cd v1.0.0 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
|
v1.0.0% ls –lrta whole 280 drwxr–xr–x 3 bhatts workers 96 Jan 30 16:41 .. drwxr–xr–x 6 bhatts workers 192 Jan 30 16:42 crs –rw–r—r— 1 bhatts workers 9073 Jan 30 16:56 cluster–template–v1.20.8–crs.yaml –rw–r—r— 1 bhatts workers 9099 Jan 30 16:56 cluster–template–v1.20.8.yaml –rw–r—r— 1 bhatts workers 9085 Jan 30 16:57 cluster–template–v1.21.8–crs.yaml –rw–r—r— 1 bhatts workers 9023 Jan 30 16:57 cluster–template–v1.21.8.yaml –rw–r—r— 1 bhatts workers 9081 Jan 30 16:57 cluster–template–v1.22.9–crs.yaml –rw–r—r— 1 bhatts workers 9019 Jan 30 16:57 cluster–template–v1.22.9.yaml –rw–r—r— 1 bhatts workers 9469 Jan 30 16:57 cluster–template.yaml –rw–r—r— 1 bhatts workers 45546 Jan 30 16:58 infrastructure–elements.yaml –rw–r—r— 1 bhatts workers 165 Jan 30 16:58 metadata.yaml –rw–r—r— 1 bhatts workers 3355 Jan 30 18:53 clusterctl.yaml drwxr–xr–x 13 bhatts workers 416 Jan 30 18:53 .
crs % ls –lrta whole 0 drwxr–xr–x 6 bhatts workers 192 Jan 30 16:42 . drwxr–xr–x 4 bhatts workers 128 Jan 30 16:42 tanzu drwxr–xr–x 4 bhatts workers 128 Jan 30 16:51 cni drwxr–xr–x 4 bhatts workers 128 Jan 30 16:54 cpi drwxr–xr–x 6 bhatts workers 192 Jan 30 16:55 csi drwxr–xr–x 13 bhatts workers 416 Jan 30 18:53 .. |
- Copy the
~/infrastructure-vcd/v1.0.0/clusterctl.yamlto~/.cluster-api/clusterctl.yaml. - The ‘
clusterctl‘ command makes use ofclusterctl.yamlfrom~/.cluster-api/clusterctl.yamlto create the capiyaml payload. Replace the infrastructure particulars from step one on this doc. - Replace the
suppliers.urlin~/.cluster-api/clusterctl.yamlto~/infrastructure-vcd/v1.0.0/infrastructure-components.yaml.
|
|
suppliers: – title: “vcd” url: “~/infrastructure-vcd/v1.0.0/infrastructure-components.yaml” sort: “InfrastructureProvider” |
At this level, we are going to want a sort cluster to put in clusterctl to generate the payload. On this step, create Form cluster to generate capiyaml payload and initialize clusterctl as follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
|
cat > type–cluster–with–extramounts.yaml <<EOF type: Cluster apiVersion: type.x–k8s.io/v1alpha4 nodes: – position: management–aircraft extraMounts: – hostPath: /var/run/docker.sock containerPath: /var/run/docker.sock EOF
Create a native cluster on mac // This may be equally executed on selection of your working system.
type create cluster —config type–cluster–with–extramounts.yaml kubectl cluster–data —context type–type kubectl config set–context type–type kubectl get po –A –owide clusterctl init —core cluster–api:v1.1.3 –b kubeadm:v1.1.3 –c kubeadm:v1.1.3 vcd:v1.0.0 |
Replace the under tkg labels to “Form: Cluster” object and annotations.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
|
OLD Metadata:
apiVersion: cluster.x–k8s.io/v1beta1 type: Cluster metadata: labels: ccm: exterior cni: antrea csi: exterior title: api5 namespace: default
New Metadata:
apiVersion: cluster.x–k8s.io/v1beta1 type: Cluster metadata: labels: cluster–position.tkg.tanzu.vmware.com/administration: “” tanzuKubernetesRelease: v1.21.8—vmware.1–tkg.2 tkg.tanzu.vmware.com/cluster–title: api5 annotations: osInfo: ubuntu,20.04,amd64 TKGVERSION: v1.4.3 title: api5 namespace: api5–ns |
- At this level, the capiyaml is able to be consumed by VCD APIs to carry out numerous operations. For verification, be sure that cluster title, namespace values are constant. Copy the content material of capiyaml to generate jsonstring utilizing comparable software as right here.
Following part describes all supported API operations for Tanzu Kubernetes Cluster on VMware Cloud Director:
Listing Clusters
Listing all clusters within the buyer group. for CSE 4.0 launch the CAPVCD model is 1.
|
|
GET https://{{vcd}}/cloudapi/1.0.0/entities/varieties/vmware/capvcdCluster/1 |
Data Cluster
Filter Cluster by title
|
|
GET https://{{vcd}}/cloudapi/1.0.0/entities/varieties/vmware/capvcdCluster/1?filter=title==clustername |
Get cluster by ID:
|
|
GET https://{{vcd}}/cloudapi/1.0.0/entities/id |
Get Kubeconfig of the cluster:
|
|
GET https://{{vcd}}/cloudapi/1.0.0/entities/id |
The Kubeconfig will be discovered as follows at: entity.standing.capvcd.non-public.kubeconfig
Create a brand new Cluster
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
|
POST https://{{vcd}}/cloudapi/1.0.0/entityTypes/urn:vcloud:sort:vmware:capvcdCluster:1.1.0 “entityType”: “urn:vcloud:sort:vmware:capvcdCluster:1.1.0”, “title”: “demo”, “externalId”: null, “entity”: { “type”: “CAPVCDCluster”, “spec”: { “vcdKe”: { “isVCDKECluster”: true, “markForDelete”: false, “forceDelete”: false, “autoRepairOnErrors”: true }, “capiYaml”: “”apiVersion: cluster.x–k8s.io/v1beta1nkind: Clusternmetadata:n labels:n cluster–position.tkg.tanzu.vmware.com/administration: ““n tanzuKubernetesRelease: v1.22.9—vmware.1–tkg.2n tkg.tanzu.vmware.com/cluster–title: api4n title: api4n namespace: api4–nsn annotations:n osInfo: ubuntu,20.04,amd64n TKGVERSION: v1.5.4nspec:n clusterNetwork:n pods:n cidrBlocks:n – 100.96.0.0/11n serviceDomain: cluster.nativen providers:n cidrBlocks:n – 100.64.0.0/13n controlPlaneRef:n apiVersion: controlplane.cluster.x–k8s.io/v1beta1n type: KubeadmControlPlanen title: api4–management–aircraftn namespace: api4–nsn infrastructureRef:n apiVersion: infrastructure.cluster.x–k8s.io/v1beta1n type: VCDClustern title: api4n namespace: api4–nsn—napiVersion: v1ndata:n password: ““n refreshToken: WU4zdWY3b21FM1k1SFBXVVp6SERTZXZvREFSUXQzTlE=n username: dG9ueQ==nkind: Secretnmetadata:n title: capi–person–credentialsn namespace: api4–nsntype: Opaquen—napiVersion: infrastructure.cluster.x–k8s.io/v1beta1nkind: VCDClusternmetadata:n title: api4n namespace: api4–nsnspec:n loadBalancerConfigSpec:n vipSubnet: ““n org: starkn ovdc: vmware–cloudn ovdcNetwork: non-public–snatn website: https://vcd.tanzu.labn useAsManagementCluster: falsen userContext:n secretRef:n title: capi-user-credentialsn namespace: api4-nsn—napiVersion: infrastructure.cluster.x-k8s.io/v1beta1nkind: VCDMachineTemplatenmetadata:n title: api4-control-planen namespace: api4-ns nspec:n template:n spec:n catalog: CSE-Templatesn diskSize: 20Gin enableNvidiaGPU: falsen placementPolicy: nulln sizingPolicy: TKG smalln storageProfile: lab-shared-storagen template: Ubuntu 20.04 and Kubernetes v1.22.9+vmware.1n—napiVersion: controlplane.cluster.x-k8s.io/v1beta1nkind: KubeadmControlPlanenmetadata:n title: api4-control-planen namespace: api4-nsnspec:n kubeadmConfigSpec:n clusterConfiguration:n apiServer:n certSANs:n – localhostn – 127.0.0.1n controllerManager:n extraArgs:n enable-hostpath-provisioner: “true”n dns:n imageRepository: tasks.registry.vmware.com/tkgn imageTag: v1.8.4_vmware.9n etcd:n native:n imageRepository: tasks.registry.vmware.com/tkgn imageTag: v3.5.4_vmware.2n imageRepository: tasks.registry.vmware.com/tkgn initConfiguration:n nodeRegistration:n criSocket: /run/containerd/containerd.sockn kubeletExtraArgs:n cloud-provider: externaln eviction-hard: nodefs.obtainable<0%,nodefs.inodesFree<0%,imagefs.obtainable<0percentn joinConfiguration:n nodeRegistration:n criSocket: /run/containerd/containerd.sockn kubeletExtraArgs:n cloud-provider: externaln eviction-hard: nodefs.obtainable<0%,nodefs.inodesFree<0%,imagefs.obtainable<0percentn customers:n – title: rootn sshAuthorizedKeys:n – “”n machineTemplate:n infrastructureRef:n apiVersion: infrastructure.cluster.x-k8s.io/v1beta1n type: VCDMachineTemplaten title: api4-control-planen namespace: api4-nsn replicas: 1n model: v1.22.9+vmware.1n—napiVersion: infrastructure.cluster.x-k8s.io/v1beta1nkind: VCDMachineTemplatenmetadata:n title: api4-md-0n namespace: api4-nsnspec:n template:n spec:n catalog: CSE-Templatesn diskSize: 20Gin enableNvidiaGPU: falsen placementPolicy: nulln sizingPolicy: TKG smalln storageProfile: lab-shared-storagen template: Ubuntu 20.04 and Kubernetes v1.22.9+vmware.1n—napiVersion: bootstrap.cluster.x-k8s.io/v1beta1nkind: KubeadmConfigTemplatenmetadata:n title: api4-md-0n namespace: api4-nsnspec:n template:n spec:n joinConfiguration:n nodeRegistration:n criSocket: /run/containerd/containerd.sockn kubeletExtraArgs:n cloud-provider: externaln eviction-hard: nodefs.obtainable<0%,nodefs.inodesFree<0%,imagefs.obtainable<0percentn customers:n – title: rootn sshAuthorizedKeys:n – “”n—napiVersion: cluster.x-k8s.io/v1beta1nkind: MachineDeploymentnmetadata:n title: api4-md-0n namespace: api4-nsnspec:n clusterName: api4n replicas: 1n selector:n matchLabels: nulln template:n spec:n bootstrap:n configRef:n apiVersion: bootstrap.cluster.x-k8s.io/v1beta1n type: KubeadmConfigTemplaten title: api4-md-0n namespace: api4-nsn clusterName: api4n infrastructureRef:n apiVersion: infrastructure.cluster.x-k8s.io/v1beta1n type: VCDMachineTemplaten title: api4-md-0n namespace: api4-nsn model: v1.22.9+vmware.1n” }, “apiVersion”: “capvcd.vmware.com/v1.1” } } |
Resize a Cluster
|
|
GET https://{{vcd}}/cloudapi/1.0.0/entities/varieties/vmware/capvcdCluster/1?filter=title==clustername |
- Fetch the Cluster ID(
"id": "urn:vcloud:entity:vmware:capvcdCluster:<ID>) from the above API name’s output. - Copy the entire output of the API response.
- Notedown eTag Worth from API response header
- Modify “capiyaml” with following values:
- To resize Management Airplane VMs Modify
kubeadmcontrolplane.spec.replicaswith desired variety of management aircraft vms. Notice solely odd numbers of management aircraft are supported. - To resize Employee Airplane VMS Modify
MachineDeployment.spec.replicaswith desired variety of employee aircraft VMs
- To resize Management Airplane VMs Modify
- Whereas performing the
PUTAPI name, guarantee to incorporate fetched eTag worth as If-Match
|
|
PUT https://{{vcd}}/cloudapi/1.0.0/entities/{cluster-id from the GET API response} headers: Settle for: utility/json; worth=37.0 Authorization: Bearer Token {token} If–Match: {eTag worth from earlier GET name} BODY: Copy whole physique from the earlier GET name, modify capiyaml values as described in above Modify step. |
Improve a Cluster
To Improve a cluster, Supplier admin must publish desired the Tanzu Kubernetes templates to the shopper group in catalog utilized by Container Service Extension.
accumulate the GET API response for the cluster to be upgraded as follows:
|
|
GET https://{{vcd}}/cloudapi/1.0.0/entities/varieties/vmware/capvcdCluster/1?filter=title==clustername |
- Fetch the Cluster ID(
"id": "urn:vcloud:entity:vmware:capvcdCluster:<ID>) from the above API name’s output. - Copy the entire output of the API response.
- Notedown eTag Worth from API response header
- The shopper person performing cluster improve would require entry to Desk 3 info. Modify Following values matching the goal TKG model. The Following desk exhibits Improve for TKG model 1.5.4 from v1.20.15+vmware.1 to v1.22.9+vmware.1
| Management Airplane Model | Outdated Values | New Values |
| VCDMachineTemplate | ||
| VCDMachineTemplate.spec.template.spec.template | Ubuntu 20.04 and Kubernetes v1.20.15+vmware.1 | Ubuntu 20.04 and Kubernetes v1.22.9+vmware.1 |
| KubeadmControlPlane | ||
| KubeadmControlPlane.spec.model | v1.20.15+vmware.1 | v1.22.9+vmware.1 |
| KubeadmControlPlane.spec.kubeadmConfigSpec.dns | imageTag: v1.7.0_vmware.15 | v1.8.4_vmware.9 |
| KubeadmControlPlane.spec.kubeadmConfigSpec.etcd | v3.4.13_vmware.23 | v3.5.4_vmware.2 |
| KubeadmControlPlane.spec.kubeadmConfigSpec.imageRepository | imageRepository: tasks.registry.vmware.com/tkg | imageRepository: tasks.registry.vmware.com/tkg |
| Employee Node Model | ||
| VCDMachineTemplate | ||
| VCDMachineTemplate.spec.template.spec.template | Ubuntu 20.04 and Kubernetes v1.20.15+vmware.1 | Ubuntu 20.04 and Kubernetes v1.22.9+vmware.1 |
| VCDMachineTemplate.spec.template.spec | ||
| MachineDeployment | ||
| MachineDeployment.spec.model | v1.20.15+vmware.1 | v1.22.9+vmware.1 |
- Whereas performing the
PUTAPI name, guarantee to incorporate fetched eTag worth as If-Match
|
|
PUT https://{{vcd}}/cloudapi/1.0.0/entities/{cluster-id from the GET} headers: Settle for: utility/json; worth=37.0 Authorization: Bearer Token &lt;token> If–Match: &lt; eTag worth from earlier GET name> BODY: Copy whole physique from the earlier GET name, modify capiyaml values as described in above step to modify capiyaml. |
Delete a Cluster
|
|
GET https://{{vcd}}/cloudapi/1.0.0/entities/varieties/vmware/capvcdCluster/1?filter=title==clustername |
- Fetch the Cluster ID(
"id": "urn:vcloud:entity:vmware:capvcdCluster:<ID>) from the above API name’s output. - Copy the entire output of the API response.
- Notedown eTag Worth from API response header
- Add or modify the next fields to delete or forcefully delete the cluster below entity.spec.vcdke:
- “markForDelete”: true, –> Set the worth to true to delete the cluster
- “forceDelete”: true, –> Set this worth to true for Forceful deletion of a cluster
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
|
PUT https://{{vcd}}/cloudapi/1.0.0/entities/{cluster-id from the GET API response} { “entityType”: “urn:vcloud:sort:vmware:capvcdCluster:1.1.0”, “title”: “demo”, “externalId”: null, “entity”: { “type”: “CAPVCDCluster”, “spec”: { “vcdKe”: { “isVCDKECluster”: true, —Add or modify the this discipline to delete the cluster “markForDelete”: true, — Add or modify the this discipline to drive delete the cluster “forceDelete”: false, “autoRepairOnErrors”: true }, “capiYaml”: “<Your capYaml payload generated from Step 5> }, . . . . #Different payload from the GET API response . . .
“org“: { “title“: “acme“, “id“: “urn:vcloud:org:cd11f6fd–67ba–40e5–853f–c17861120184“ } } |
Advice for API Utilization throughout automation
- DO NOT hardcode API urls with RDE variations. ALWAYS parameterize RDE variations. For instance:
POST https://{{vcd}}/cloudapi/1.0.0/entityTypes/urn:vcloud:sort:vmware:capvcdCluster:1.1.0 Guarantee to declare 1.1.0 as a variable. This may guarantee straightforward API shopper upgrades to future variations of CSE.
- Make sure the API shopper code ignores any unknown/further properties whereas unmarshaling the API response
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
|
#For instance, capvcdCluster 1.1.0 API payload seems to be like under { standing: { kubernetesVersion: 1.20.8, nodePools: {} } } #Sooner or later, subsequent model of capvcdCluster 1.2.0 could add extra properties (“add-ons”) to the payload. # The previous API shopper code should guarantee it doesn’t break on seeing newer properties sooner or later payloads. { standing: { kubernetesVersion: 1.20.8, nodePools: {}, add–ons: {} // new property sooner or later model } } |
Abstract
To summarize, we checked out CRUD operations for a Tanzu Kubernetes clusters on VMware Cloud Director platform utilizing VMware Cloud Director supported APIs. Please be happy to checkout different assets for Container Service Extension as follows: