feat(rcv1p): unify cert bootstrap flow and add Windows CA refresh task#8096
Open
rchincha wants to merge 99 commits into
Open
feat(rcv1p): unify cert bootstrap flow and add Windows CA refresh task#8096rchincha wants to merge 99 commits into
rchincha wants to merge 99 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to unify the custom-cloud CA certificate bootstrap path (removing the separate “operation-requests” init scripts) and adds a Windows scheduled task to periodically refresh custom-cloud CA certificates.
Changes:
- Windows: add a scheduled task to refresh custom-cloud CA certificates; update
Get-CACertificatesto support legacy vs “rcv1p” modes keyed off location. - Linux: consolidate custom-cloud init to a single init script and update CSE command generation to set a cert-endpoint mode variable.
- Regenerate multiple custom data / generated command snapshots to reflect the new templates.
Reviewed changes
Copilot reviewed 74 out of 176 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| staging/cse/windows/kubernetesfunc.ps1 | Adds CA refresh scheduled task + updates CA retrieval logic and error behavior |
| parts/windows/kuberneteswindowssetup.ps1 | Wires Get-CACertificates -Location and registers refresh task for custom clouds |
| pkg/agent/variables.go | Always injects initAKSCustomCloud payload into cloud-init data |
| pkg/agent/const.go | Removes separate custom-cloud init script constants; keeps single init script |
| pkg/agent/baker.go | Simplifies GetTargetEnvironment; notes IsAKSCustomCloud as deprecated |
| parts/linux/cloud-init/artifacts/cse_cmd.sh | Updates CSE command to set cert endpoint mode + run custom-cloud init script |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh | Deleted (custom-cloud init consolidation) |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh | Deleted (custom-cloud init consolidation) |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh | Deleted (custom-cloud init consolidation) |
| aks-node-controller/parser/templates/cse_cmd.sh.gtpl | Mirrors CSE command template updates for aks-node-controller parser |
| aks-node-controller/parser/testdata/Compatibility+EmptyConfig/generatedCSECommand | Regenerated snapshot for new CSE cmd template |
| aks-node-controller/parser/testdata/AzureLinuxv2+Kata+DisableUnattendedUpgrades=false/generatedCSECommand | Regenerated snapshot for new CSE cmd template |
| aks-node-controller/parser/testdata/AKSUbuntu2204+SSHStatusOn/generatedCSECommand | Regenerated snapshot for new CSE cmd template |
| aks-node-controller/parser/testdata/AKSUbuntu2204+EnablePubkeyAuth/generatedCSECommand | New snapshot for new template output |
| aks-node-controller/parser/testdata/AKSUbuntu2204+DisablePubkeyAuth/generatedCSECommand | New snapshot for new template output |
| aks-node-controller/parser/testdata/AKSUbuntu2204+DefaultPubkeyAuth/generatedCSECommand | New snapshot for new template output |
| aks-node-controller/parser/testdata/AKSUbuntu2204+CustomOSConfig/generatedCSECommand | Regenerated snapshot for new CSE cmd template |
| aks-node-controller/parser/testdata/AKSUbuntu2204+CustomCloud/generatedCSECommand | Regenerated snapshot for new CSE cmd template |
| aks-node-controller/parser/testdata/AKSUbuntu2204+Containerd+MIG/generatedCSECommand | Regenerated snapshot for new CSE cmd template |
| aks-node-controller/parser/testdata/AKSUbuntu2204+CloudProviderOverrides/generatedCSECommand | New snapshot for new template output |
| aks-node-controller/parser/testdata/AKSUbuntu2204+China/generatedCSECommand | Regenerated snapshot for new CSE cmd template |
| pkg/agent/testdata/MarinerV2+Kata/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AzureLinuxV2+Kata/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AzureLinuxV3+Kata/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AKSUbuntu2204+China/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AKSUbuntu2204+SSHStatusOff/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/CustomizedImage/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/CustomizedImageKata/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData | Regenerated snapshot (custom data gzip payload changed) |
| pkg/agent/testdata/Flatcar/CustomData.inner | Regenerated snapshot (embedded gzip payload changed) |
| pkg/agent/testdata/ACL/CustomData.inner | Regenerated snapshot (embedded gzip payload changed) |
You can also share your feedback on Copilot code review. Take the survey.
rchincha
force-pushed
the
origin/rchinchani/rcv1p-2
branch
from
44ff9ee to
a0a1307
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to unify AKS custom-cloud CA certificate bootstrap behavior (legacy vs “rcv1p/operation-requests” style flows) and adds a Windows scheduled task to periodically refresh custom-cloud CA certificates.
Changes:
- Adds Windows CA refresh scheduled task registration and introduces location-based endpoint-mode selection (legacy vs rcv1p).
- Refactors Windows CA certificate retrieval to support both endpoint modes and opt-in gating for rcv1p.
- Simplifies Linux custom-cloud init script selection by consolidating onto
init-aks-custom-cloud.shand removing older variants; updates generated testdata accordingly.
Reviewed changes
Copilot reviewed 93 out of 99 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| staging/cse/windows/kubernetesfunc.ps1 | Adds CA refresh scheduled task and endpoint-mode-aware Get-CACertificates implementation. |
| pkg/agent/variables.go | Simplifies how initAKSCustomCloud is added to Linux cloud-init variables. |
| pkg/agent/testdata/MarinerV2+Kata/CustomData | Updates expected CustomData snapshot (generated content changed). |
| pkg/agent/testdata/Flatcar/CustomData.inner | Updates expected Flatcar CustomData snapshot (generated content changed). |
| pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData | Updates expected CustomData snapshot (generated content changed). |
| pkg/agent/testdata/CustomizedImageKata/CustomData | Updates expected CustomData snapshot (generated content changed). |
| pkg/agent/testdata/CustomizedImage/CustomData | Updates expected CustomData snapshot (generated content changed). |
| pkg/agent/testdata/AzureLinuxV3+Kata/CustomData | Updates expected CustomData snapshot (generated content changed). |
| pkg/agent/testdata/AzureLinuxV2+Kata/CustomData | Updates expected CustomData snapshot (generated content changed). |
| pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingNoConfig/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingDisabled/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworking/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+ootcredentialprovider/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+SecurityProfile/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+ManagedIdentity/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+KubeletServingCertificateRotation/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+KubeletClientTLSBootstrapping/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+K8S119/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+K8S119+FIPS/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+K8S119+CSI/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+K8S118/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+K8S117/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+K8S116/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+EnablePrivateClusterHostsConfigAgent/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+CustomVnet/CustomData | Updates expected Windows CustomData snapshot (calls/refresh task additions). |
| pkg/agent/testdata/AKSWindows2019+CustomCloud/CustomData | Updates expected Windows CustomData snapshot (new Get-CACertificates call form + refresh task). |
| pkg/agent/testdata/AKSWindows2019+CustomCloud+ootcredentialprovider/CustomData | Updates expected Windows CustomData snapshot (new Get-CACertificates call form + refresh task). |
| pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData | Updates expected Ubuntu CustomData snapshot (generated content changed). |
| pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData | Updates expected Ubuntu CustomData snapshot (generated content changed). |
| pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData | Updates expected Ubuntu CustomData snapshot (generated content changed). |
| pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData | Updates expected Ubuntu CustomData snapshot (generated content changed). |
| pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData | Updates expected Ubuntu CustomData snapshot (generated content changed). |
| pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData | Updates expected Ubuntu CustomData snapshot (generated content changed). |
| pkg/agent/testdata/AKSUbuntu2204+China/CustomData | Updates expected Ubuntu CustomData snapshot (generated content changed). |
| pkg/agent/testdata/ACL/CustomData.inner | Updates expected ACL CustomData snapshot (generated content changed). |
| pkg/agent/const.go | Consolidates custom-cloud init script constants to a single script. |
| parts/windows/kuberneteswindowssetup.ps1 | Updates Windows setup flow to call Get-CACertificates with location and registers CA refresh scheduled task. |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh | Removes operation-requests-specific Linux init script (consolidation). |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh | Removes Mariner/AzureLinux operation-requests init script (consolidation). |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh | Removes Mariner/AzureLinux legacy init script variant (consolidation). |
| aks-node-controller/parser/templates/cse_cmd.sh.gtpl | Adds a LOCATION shell variable in the generated CSE command template. |
| aks-node-controller/parser/helper.go | Factors out a shared getCloudLocation helper and reuses it in getCloudTargetEnv. |
You can also share your feedback on Copilot code review. Take the survey.
rchincha
changed the title
rchincha
force-pushed
the
origin/rchinchani/rcv1p-2
branch
from
2b3c1d6 to
e19a19b
Compare
rchincha
force-pushed
the
origin/rchinchani/rcv1p-2
branch
from
e19a19b to
d41856f
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR unifies the AKS custom cloud CA certificate bootstrap logic to a single flow and adds a Windows scheduled task to periodically refresh custom cloud CA certificates. It also updates Linux/customdata generation and test snapshots to reflect the new wiring.
Changes:
- Add Windows scheduled task registration for daily CA certificate refresh and introduce a location-based cert endpoint mode selector.
- Simplify Linux custom cloud init script selection by standardizing on
init-aks-custom-cloud.sh, plus add wiring/tests for refresh-mode arguments. - Update aks-node-controller template to export
LOCATION, and regenerate CustomData snapshot test artifacts.
Reviewed changes
Copilot reviewed 95 out of 101 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| staging/cse/windows/kubernetesfunc.tests.ps1 | Adds Pester coverage for cert endpoint mode selection, scheduled task registration, and CA retrieval behavior. |
| staging/cse/windows/kubernetesfunc.ps1 | Implements unified Windows CA retrieval logic with legacy/rcv1p modes and registers a daily refresh scheduled task. |
| spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh | Adds ShellSpec assertions to validate refresh-mode argument parsing/wiring in the Linux init script. |
| pkg/agent/variables.go | Changes how initAKSCustomCloud is injected into Linux cloud-init data. |
| pkg/agent/const.go | Removes per-cloud custom init script constants and standardizes on init-aks-custom-cloud.sh. |
| parts/windows/kuberneteswindowssetup.ps1 | Wires CA retrieval call and registers the Windows CA refresh scheduled task during BasePrep. |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh | Removed (operation-requests variant no longer used). |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh | Removed (operation-requests Mariner/AzureLinux variant no longer used). |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh | Removed (Mariner/AzureLinux legacy variant no longer used). |
| aks-node-controller/parser/templates/cse_cmd.sh.gtpl | Exports LOCATION into the CSE environment for downstream scripts. |
| aks-node-controller/parser/helper.go | Adds a helper to normalize location and reuses it in cloud target env detection. |
| pkg/agent/testdata/MarinerV2+Kata/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/Flatcar/CustomData.inner | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/CustomizedImageKata/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/CustomizedImage/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/AzureLinuxV3+Kata/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/AzureLinuxV2+Kata/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingNoConfig/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingDisabled/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworking/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+ootcredentialprovider/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+SecurityProfile/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+ManagedIdentity/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+KubeletServingCertificateRotation/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+KubeletClientTLSBootstrapping/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+K8S119/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+K8S119+FIPS/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+K8S119+CSI/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+K8S118/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+K8S117/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+K8S116/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+EnablePrivateClusterHostsConfigAgent/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+CustomVnet/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+CustomCloud/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSWindows2019+CustomCloud+ootcredentialprovider/CustomData | Regenerated CustomData snapshot due to Windows CA refresh task wiring. |
| pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/AKSUbuntu2204+China/CustomData | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
| pkg/agent/testdata/ACL/CustomData.inner | Regenerated CustomData snapshot due to init/custom cloud wiring changes. |
You can also share your feedback on Copilot code review. Take the survey.
rchincha
force-pushed
the
origin/rchinchani/rcv1p-2
branch
from
d41856f to
18ba549
Compare
rchincha
force-pushed
the
origin/rchinchani/rcv1p-2
branch
from
18ba549 to
e94c465
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates AKS custom cloud certificate bootstrapping to use a single unified flow and adds a Windows scheduled task for periodic custom cloud CA refresh.
Changes:
- Added Windows CA refresh task registration plus new logic to select cert retrieval mode and opt-in gating.
- Simplified Linux custom cloud init script wiring by removing legacy “operation-requests” variants and normalizing location for refresh mode.
- Added/updated tests and refreshed golden testdata outputs to reflect new custom data content.
Reviewed changes
Copilot reviewed 95 out of 101 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| staging/cse/windows/kubernetesfunc.tests.ps1 | Adds Pester coverage for endpoint-mode selection, task registration behavior, and CA retrieval failure handling. |
| staging/cse/windows/kubernetesfunc.ps1 | Implements endpoint-mode derivation, opt-in gating, CA retrieval paths, and a Windows scheduled task for refresh. |
| spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh | Adds ShellSpec checks to ensure init script wiring for ca-refresh mode and LOCATION usage. |
| pkg/agent/variables.go | Simplifies init script selection and updates how custom cloud init script is injected into cloud-init data. |
| pkg/agent/const.go | Removes now-unused custom-cloud init script constants; keeps unified init script constant. |
| parts/windows/kuberneteswindowssetup.ps1 | Updates Windows setup to call Get-CACertificates with Location and conditionally register refresh task. |
| aks-node-controller/parser/templates/cse_cmd.sh.gtpl | Adds LOCATION variable for downstream scripts during custom cloud provisioning. |
| aks-node-controller/parser/helper.go | Adds getCloudLocation helper and reuses it for cloud target env detection. |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh | Removes legacy operation-requests init script (superseded by unified script). |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh | Removes legacy Mariner operation-requests init script (superseded by unified script). |
| parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh | Removes legacy Mariner init script variant (superseded by unified script). |
| pkg/agent/testdata/MarinerV2+Kata/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/Flatcar/CustomData.inner | Updates golden ignition/customData payload for unified custom cloud init content. |
| pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/CustomizedImageKata/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/CustomizedImage/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AzureLinuxV3+Kata/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AzureLinuxV2+Kata/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingNoConfig/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingDisabled/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworking/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+ootcredentialprovider/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+SecurityProfile/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+ManagedIdentity/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+KubeletServingCertificateRotation/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+KubeletClientTLSBootstrapping/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+K8S119/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+K8S119+FIPS/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+K8S119+CSI/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+K8S118/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+K8S117/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+K8S116/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+EnablePrivateClusterHostsConfigAgent/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+CustomVnet/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+CustomCloud/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSWindows2019+CustomCloud+ootcredentialprovider/CustomData | Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating. |
| pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AKSUbuntu2204+SSHStatusOff/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/AKSUbuntu2204+China/CustomData | Updates golden customData to match unified custom cloud init content. |
| pkg/agent/testdata/ACL/CustomData.inner | Updates golden ignition/customData payload for unified custom cloud init content. |
Comments suppressed due to low confidence (7)
staging/cse/windows/kubernetesfunc.ps1:1
Get-CACertificatesused to fail fast viaSet-ExitCodeon retrieval/parse errors, but now returns$false(and logs warnings) for a wide range of failure cases. Because call sites in the generated setup scripts invokeGet-CACertificates -Location $Locationwithout checking the return value, this can silently proceed without required CA material and lead to harder-to-diagnose TLS failures later in provisioning. Consider restoring fatal behavior for “expected-to-install” scenarios (e.g., legacy mode, or rcv1p when opted-in), or have callers check the return value and invokeSet-ExitCodewhen it’s$falsein those modes.
staging/cse/windows/kubernetesfunc.ps1:1Get-CACertificatesused to fail fast viaSet-ExitCodeon retrieval/parse errors, but now returns$false(and logs warnings) for a wide range of failure cases. Because call sites in the generated setup scripts invokeGet-CACertificates -Location $Locationwithout checking the return value, this can silently proceed without required CA material and lead to harder-to-diagnose TLS failures later in provisioning. Consider restoring fatal behavior for “expected-to-install” scenarios (e.g., legacy mode, or rcv1p when opted-in), or have callers check the return value and invokeSet-ExitCodewhen it’s$falsein those modes.
pkg/agent/variables.go:1- This change removes the previous
cs.IsAKSCustomCloud()guard and injects the custom cloud init script intocloudInitDataunconditionally. That can increase customData size for all clusters (risking platform limits) and may introduce unintended side effects if any downstream template writes/executes this script outside custom cloud. Recommend reinstating the custom cloud guard (and only settinginitAKSCustomCloudwhenIsAKSCustomCloud()is true), while still using the unifiedinitAKSCustomCloudScriptfor all custom clouds.
staging/cse/windows/kubernetesfunc.ps1:1 $resourceFileNameis used directly to build a path underC:\ca. If the upstream response ever contains path separators (e.g.,..\fooor nested paths), this can write outside the intended directory. Prefer sanitizing to a basename (e.g., usingSplit-Path -Leafor[IO.Path]::GetFileName($resourceFileName)) beforeJoin-Path, and consider rejecting names containing directory traversal characters.
staging/cse/windows/kubernetesfunc.ps1:1$resourceFileNameis used directly to build a path underC:\ca. If the upstream response ever contains path separators (e.g.,..\fooor nested paths), this can write outside the intended directory. Prefer sanitizing to a basename (e.g., usingSplit-Path -Leafor[IO.Path]::GetFileName($resourceFileName)) beforeJoin-Path, and consider rejecting names containing directory traversal characters.
staging/cse/windows/kubernetesfunc.ps1:1- The new rcv1p operation-requests flow is non-trivial (multiple requests, JSON shape assumptions, per-item content downloads, and
$downloadedAnyaggregation), but the added Pester tests only cover legacy mode and the “throws returns false” path. Add tests that (1) exercise the rcv1p path end-to-end with mockedRetry-Commandreturning operation requests and cert bodies, and (2) verify behavior when operation requests are empty/invalid (ensuring the function returns$falseand logs expected warnings).
pkg/agent/variables.go:1 - The PR description still contains placeholder text (
Fixes #with no linked issue and no explanation of “what/why”). Please update the PR description to summarize the behavior change (unified bootstrap + Windows refresh task) and link the relevant issue or remove the placeholder.
rchincha
force-pushed
the
origin/rchinchani/rcv1p-2
branch
from
e94c465 to
f20d5b8
Compare
rchincha
force-pushed
the
origin/rchinchani/rcv1p-2
branch
from
f20d5b8 to
b53f240
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The task must only be registered after temp kubeconfig removal (line 595) to avoid a TLS bootstrap race. The earlier call in BasePrep was redundant and reintroduced that race condition. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously install_certs_to_trust_store failure was silently ignored, allowing provisioning to continue without proper CA configuration. Now exits with error to match the fatal-after-retries intent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The auto-injection is determined by subscription enrollment, not tenant. Removing the parenthetical avoids confusion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Daily orchestration in aks-rp already runs these tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The auto-inject behavior is controlled by the E2E config flag, not the tenant itself. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Properly unmarshal the ARM response and check properties.state with case-insensitive comparison instead of brittle string matching. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In production, AKS RP handles tag application and CSE sequencing through CRP. The deferred extension is only needed in E2E because Uniform VMSS requires BeginUpdate for instance-level tags. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Emit guest agent telemetry when restarting chrony in custom cloud environments for better observability. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mdata The initAKSCustomCloudRepos write_files entry was placed unconditionally in the non-scriptless branch, but variables.go only populates it when IsAKSCustomCloud() is true. With missingkey=zero, non-custom-cloud nodes would get a 0-byte file written unnecessarily. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The aks-rp orchestrator queues e2e-tme.yaml with --subscription-id, but the top-level pipeline did not declare a subscriptionId parameter, causing ADO to reject the queue request with a validation error. PR #8747 added the parameter to the inner template (e2e-template.yaml) but ADO validates parameters against the top-level YAML before expanding templates. Forward the parameter through to the template so orchestrated RCV1P runs can pass through the RCV1P subscription ID. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The crontab filter used 'grep -v "$scriptPath" ca-refresh' which treats $scriptPath as a BRE regex. Path metacharacters like '.' would match any character, risking removal of unrelated crontab entries. Switch to 'grep -F -v' for literal-string matching. Eliminates the regex-injection surface even if the script path ever changes to include shell/regex metacharacters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Commit 4353cd4 ("e2e: simplify RCV1P to single-subscription-per-job model") gutted the dual-subscription override branches in GetResourceGroupName and GetVMIdentityResourceID, but left Scenario.SubscriptionID and GetSubscriptionID() as vestigial code. The field is set nowhere; every call to GetSubscriptionID() already resolves to config.Config.SubscriptionID. Worse, GetVMIdentityResourceID never honored the override even when the field was active, creating a latent footgun if anyone repopulated the field: VMSS identity would resolve in the wrong subscription and ARM creation would fail. Remove the half-honored surface entirely, aligning the code with the single-subscription-per-job architecture: - Drop Scenario.SubscriptionID field and GetSubscriptionID() method - Drop GetVMIdentityResourceID() (one-line wrapper, used once) - Replace s.GetSubscriptionID() with config.Config.SubscriptionID at all 4 call sites in vmss.go and 1 in test_helpers.go - Replace s.GetVMIdentityResourceID() with config.Config.VMIdentityResourceID(s.Location) in vmss.go Behavior is identical (all call sites already resolved to the same value). go vet and go build pass clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… test runner exit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Failed gate runRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=169432101 Detective summary
Likely cause and signatureCSE cached-performance: likely budget/test-code or PR-caused CSE timing regression; PR #8096 changes relevant CSE, nodecustomdata, E2E validator/helper paths. Signature: Recommended owner/actionPR #8096 author and Node Lifecycle CSE/E2E owners should inspect the timing budget, task suffix mismatch warnings, and whether Evidence
|
…ptionId param Variable group ab-e2e-tme defines E2E_SUBSCRIPTION_ID, which ADO auto-exposes as an env var to all tasks. The previous task-level env: override did not reliably win against the auto-exposed group variable in AzureCLI@2, so the orchestrator's --subscription-id (e.g. RCV1P sub) was ignored and tests ran/skipped against the default TME sub. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ails The rcv1p opted-in path previously ignored the exit status of install_certs_to_trust_store, allowing provisioning to continue with an incomplete trust store. This now matches the legacy path's behavior of exiting non-zero on failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Failed gate runRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=169461289 Detective summary
Likely cause and signatureE2E: known HTTPSProxy PrivateDNS CSE/provisioning flake under repair Bug #38559852. Signature: Recommended owner/actionContinue the existing repair investigations; no new repair item was created and no PR-specific action is indicated from this evidence alone. Evidence
|
Contributor
Failed gate runRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=169595823 Detective summaryNo E2E scenario ran. The job repeatedly warned that Likely cause and signatureLikely PR-caused E2E pipeline variable regression from #8096 YAML/template changes. Signature: Recommended owner/actionPR #8096 owner should fix the E2E variable mapping so Evidence
|
…l variable reference
Previously the subscriptionId parameter defaulted to $(E2E_SUBSCRIPTION_ID)
and the template defined a variable E2E_SUBSCRIPTION_ID with value
${{parameters.subscriptionId}}, creating a cycle when no caller explicitly
passed subscriptionId. Use an empty-string sentinel default and only override
the pipeline variable when the parameter is non-empty.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- init-aks-custom-cloud.sh: sanitize cert_filename via basename to prevent path traversal from wireserver response (defense-in-depth) - init-aks-custom-cloud.sh: quote $certs in retrieve_legacy_certs to avoid word-splitting on JSON payload - scenario_rcv1p_win_test.go / validators.go: update misleading comments that claimed Windows cert store import; current validators only check files in C:\ca Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it
This PR implements RCV1P (Robust Certificate Validation for 1P) — the next-generation mechanism for distributing Azure root CA certificates to AKS nodes. Instead of hardcoding certificate bundles, RCV1P queries the Azure wireserver at provisioning time to download and install the latest root certificates into the OS trust store.
Reference: https://eng.ms/docs/products/onecert-certificates-key-vault-and-dsms/onecert-customer-guide/autorotationandecr/rcv1ptsg
Summary of Changes
1. Linux: Unified cert bootstrap flow (
init-aks-custom-cloud.sh)init-aks-custom-cloud-mariner.sh,init-aks-custom-cloud-operation-requests-mariner.sh, andinit-aks-custom-cloud-operation-requests.sh. All cert logic now flows through a singleinit-aks-custom-cloud.shthat detects the distro (Ubuntu, Mariner, AzureLinux, Flatcar, ACL) at runtime.init-aks-custom-cloud-repos.shto keep basecustomDatasize small for non-custom-cloud scenarios (critical for Flatcar/ACL which have tight size limits).legacy(ussec/usnat regions) andrcv1p(all other regions), selected by cloud location at runtime.2. Windows: CA cert refresh task and rcv1p support (
kubernetesfunc.ps1)Get-CACertificateswith-Locationparameter: Determines cert endpoint mode from location, uses legacy endpoint for ussec/usnat, rcv1p for all others.Register-CACertificatesRefreshTask: Registers a daily scheduled task to refresh CA certificates, with backward compatibility for older VHDs that don't accept-Location.Should-InstallCACertificatesRefreshTask: Gates refresh task registration on wireserver opt-in status.3. E2E tests (
e2e/scenario_rcv1p_test.go,e2e/scenario_rcv1p_win_test.go)C:\ca, Windows certificate store import, and scheduled task registration.Test_RCV1P_NotOptedInverifies that omitting the VM opt-in tag correctly prevents cert installation..pipelines/e2e-rcv1p.yamlruns daily at 3am PST with tag filterrcv1pcertmode=true(not yet enabled).4. E2E infrastructure: multi-subscription and VM instance tagging
RCV1P_SUBSCRIPTION_ID) with theMicrosoft.Compute/PlatformSettingsOverridefeature flag. AddedSubscriptionIDfield to scenarios andGetAzure()/GetSubscriptionID()helpers.platformsettings.host_environment.service.platform_optedin_for_rootcerts=true) on the VMSS at creation time via aVMConfigMutator. VMSS-level tags inherit to VM instances automatically.1. Cert endpoint mode is determined by cloud location, not a flag
Decision:
ussec*/usnat*→legacymode, everything else →rcv1pmode. This is determined at runtime from the node's Azure location.Why: Avoids requiring a new API contract field. The location-based approach lets us roll out rcv1p incrementally — ussec/usnat stay on the legacy endpoint that works today, while all other regions use the new rcv1p endpoint with opt-in gating.
2. Two-layer access control for rcv1p
Decision: Both conditions must be met for cert installation:
Microsoft.Compute/PlatformSettingsOverride) enables the wireserver endpointplatformsettings.host_environment.service.platform_optedin_for_rootcerts=true) grants per-VM accessWhy: Defense in depth — the subscription flag is a coarse gate, the VM tag provides per-node opt-in control. Without the tag, wireserver returns
IsOptedInForRootCerts=false.3. VM opt-in tag is set at VMSS creation time
Decision: The opt-in tag (
platformsettings.host_environment.service.platform_optedin_for_rootcerts=true) is set on the VMSS at creation time and inherits to all VM instances automatically.Why: VMSS-level tags propagate to VM instances, and wireserver reads the tag from the VM instance to determine opt-in status. In E2E tests, the positive tests set the tag via a
VMConfigMutatorat VMSS creation, while the negative test (Test_RCV1P_NotOptedIn) simply omits the tag to verify wireserver returnsIsOptedInForRootCerts=false.4.
Get-CACertificatesmoved outsideIsAKSCustomCloudguard (Windows)Decision:
Get-CACertificates -Location $Location -FailOnErrornow runs for all clouds, not just custom clouds.Why: RCV1P applies to all clouds. The function itself handles the location-based mode selection internally and gracefully skips cert installation when wireserver returns
IsOptedInForRootCerts=false(which is the case on public cloud without the feature flag).5. Wireserver failures are fatal after retries
Decision: If wireserver cert endpoints fail after exhausting retries, provisioning fails (
exit 1on Linux,throwon Windows with-FailOnError).Why: Cert installation is required for the selected mode. Silently continuing without certificates would leave the node in an inconsistent state. Retries with backoff handle transient wireserver issues (rate limiting, temporary unavailability).
6. Backward compatibility for Windows VHD/CSE version skew
Decision:
kuberneteswindowssetup.ps1guardsRegister-CACertificatesRefreshTaskwithGet-Commandchecks before calling it.Why: Windows VHD and CSE release independently. Newer CSE must not crash on older VHDs that don't have these functions. The guard falls back gracefully.
Testing Evidence
MSFT tenant (default E2E subscription)
Linux (Build 158446017):
IsOptedInForRootCertscheck works (skips on public cloud as expected)Windows (Build 158446024):
Get-CACertificates -Locationcorrectly selects rcv1p modeShould-InstallCACertificatesRefreshTaskreturns$falseon public cloud (correct)TME tenant (RCV1P_SUBSCRIPTION_ID set in pipeline, with PlatformSettingsOverride feature flag)
Linux — Validated end-to-end: wireserver returns
IsOptedInForRootCerts=true, certificates downloaded and installed into OS trust store, refresh schedule registered. Passed across Ubuntu 2204, Ubuntu 2404, AzureLinux V3, Flatcar, ACL.Windows (Build 161633049):
IsOptedInForRootCerts=true, certificates downloaded toC:\ca, scheduled taskaks-ca-certs-refresh-taskregisteredwindows-2022-containerdjob) was a pre-existingTest_Windows2022_VHDCachingissue unrelated to RCV1POperationRequestsinstead ofOperationsInfo) when parsing wireserver responses — this was the root cause of empty cert downloads. Fixed in commitb6cd4e4f68.Files Changed (31 files, +1979 / -1218)
init-aks-custom-cloud.sh,init-aks-custom-cloud-repos.sh(new), 3 removedkubernetesfunc.ps1,kuberneteswindowssetup.ps1kubernetesfunc.tests.ps1(new)scenario_rcv1p_test.go(new),scenario_rcv1p_win_test.go(new)vmss.go,types.go,validators.go,cluster.go,config/e2e-rcv1p.yaml(new)baker.go,const.go,variables.goPR File Breakdown: Functionality vs Tests
Functionality (1,859 lines — 51%)
parts/linux/cloud-init/artifacts/init-aks-custom-cloud.shparts/linux/cloud-init/artifacts/init-aks-custom-cloud-repos.shparts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.shparts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.shstaging/cse/windows/kubernetesfunc.ps1parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.shpkg/agent/variables.goparts/windows/kuberneteswindowssetup.ps1pkg/agent/const.goparts/linux/cloud-init/nodecustomdata.ymlaks-node-controller/parser/helper.goparts/linux/cloud-init/artifacts/cse_cmd.shaks-node-controller/parser/templates/cse_cmd.sh.gtplpkg/agent/baker.goTests / E2E Infra (1,795 lines — 49%)
e2e/scenario_rcv1p_test.gostaging/cse/windows/kubernetesfunc.tests.ps1e2e/scenario_rcv1p_win_test.goe2e/validators.goe2e/cluster.goe2e/vmss.goe2e/config/azure.goe2e/test_helpers.goe2e/types.gospec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.she2e/cache.goe2e/aks_model.goe2e/config/config.go.pipelines/e2e-rcv1p.yamle2e/kube.go.pipelines/scripts/e2e_run.sh.pipelines/templates/e2e-template.yamlSummary