[BUG]: Node 24 Hard Dependency in Agent Version 4.265.1 #5426
Description
What happened?
Our agent versions upgraded last night to 4.265.1 for our Managed DevOps Pool and now has a hard failure on various tasks due to demanding Node 24 be in the container as per error OCI runtime exec failed: exec failed: unable to start container process: exec /__a/externals/node24/bin/node: no such file or directory: unknown
It appears that the safety features of #5361 may not be working as expected? We've not explicitly set the env var AGENT_USE_NODE24 so it should be defaulting to false?
Versions
Azure DevOps Agent version 4.265.1
Environment type (Please select at least one enviroment where you face this issue)
- Self-Hosted
- Microsoft Hosted
- VMSS Pool
- Container
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
No response
Version controll system
No response
Relevant log output
Starting: Initialize job
Agent name: 'SWManagedPool 4'
Agent machine name: '7f5f34f9e000000'
Current agent version: '4.265.1'
Managed DevOps Pool
Agent running as: 'AzDevOps'
Prepare build directory.
Set build variables.
Download all required tasks.
Downloading task: PowerShell (2.266.0)
Downloading task: CmdLine (2.250.1)
Downloading task: AzureCLI (2.264.0)
Downloading task: Docker (2.265.1)
Checking job knob settings.
Knob: DockerActionRetries = true Source: $(VSTSAGENT_DOCKER_ACTION_RETRIES)
Knob: UseGitLongPaths = true Source: $(USE_GIT_LONG_PATHS)
Knob: EnableIssueSourceValidation = true Source: $(ENABLE_ISSUE_SOURCE_VALIDATION)
Knob: AgentEnablePipelineArtifactLargeChunkSize = true Source: $(AGENT_ENABLE_PIPELINEARTIFACT_LARGE_CHUNK_SIZE)
Knob: ContinueAfterCancelProcessTreeKillAttempt = true Source: $(VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT)
Knob: ProcessHandlerSecureArguments = false Source: $(AZP_75787_ENABLE_NEW_LOGIC)
Knob: ProcessHandlerSecureArguments = false Source: $(AZP_75787_ENABLE_NEW_LOGIC_LOG)
Knob: ProcessHandlerTelemetry = true Source: $(AZP_75787_ENABLE_COLLECT)
Knob: UseNewNodeHandlerTelemetry = True Source: $(DistributedTask.Agent.USENEWNODEHANDLERTELEMETRY)
Knob: ProcessHandlerEnableNewLogic = true Source: $(AZP_75787_ENABLE_NEW_PH_LOGIC)
Knob: EnableResourceMonitorDebugOutput = true Source: $(AZP_ENABLE_RESOURCE_MONITOR_DEBUG_OUTPUT)
Knob: EnableResourceUtilizationWarnings = true Source: $(AZP_ENABLE_RESOURCE_UTILIZATION_WARNINGS)
Knob: IgnoreVSTSTaskLib = true Source: $(AZP_AGENT_IGNORE_VSTSTASKLIB)
Knob: FailJobWhenAgentDies = true Source: $(FAIL_JOB_WHEN_AGENT_DIES)
Knob: CheckForTaskDeprecation = true Source: $(AZP_AGENT_CHECK_FOR_TASK_DEPRECATION)
Knob: CheckIfTaskNodeRunnerIsDeprecated246 = True Source: $(DistributedTask.Agent.CheckIfTaskNodeRunnerIsDeprecated246)
Knob: UseNode20ToStartContainer = True Source: $(DistributedTask.Agent.UseNode20ToStartContainer)
Knob: EnableTimeoutLogFlushing = True Source: $(DistributedTask.Agent.EnableTimeoutLogFlushing)
Knob: LogTaskNameInUserAgent = true Source: $(AZP_AGENT_LOG_TASKNAME_IN_USERAGENT)
Knob: UseFetchFilterInCheckoutTask = true Source: $(AGENT_USE_FETCH_FILTER_IN_CHECKOUT_TASK)
Knob: Rosetta2Warning = true Source: $(ROSETTA2_WARNING)
Knob: AddForceCredentialsToGitCheckout = True Source: $(DistributedTask.Agent.AddForceCredentialsToGitCheckout)
Knob: AddForceCredentialsToGitCheckoutEnhanced = True Source: $(DistributedTask.Agent.AddForceCredentialsToGitCheckoutEnhanced)
Knob: UseSparseCheckoutInCheckoutTask = true Source: $(AGENT_USE_SPARSE_CHECKOUT_IN_CHECKOUT_TASK)
Knob: AgentCDNConnectivityFailWarning = False Source: $(DistributedTask.Agent.AgentCDNConnectivityFailWarning)
Knob: UseEnhancedLogging = true Source: ${AZP_USE_ENHANCED_LOGGING}
Finished checking job knob settings.
Start tracking orphan processes.
Finishing: Initialize job
---
Final exit code for Docker pull: 0
/usr/bin/docker logout https://***
Removing login credentials for ***
/usr/bin/docker info -f "{{range .Plugins.Network}}{{println .}}{{end}}"
bridge
host
ipvlan
macvlan
null
overlay
/usr/bin/docker network create --label 5fd376 vsts_network_3b5bc712f704461592248840a4358bf4
80c176054e442945bd665833ed47f0da53f22ea6e0f67958e3e17d3167d9612f
/usr/bin/docker inspect --format="{{index .Config.Labels \"com.azure.dev.pipelines.agent.handler.node.path\"}}" ***/devops/deploy:alpine3.20
/usr/local/bin/node
/usr/bin/docker create --name buildDeploy_slimmingworldazurecriodevopsdeployalpine320_86a35f --label 5fd376 --network vsts_network_3b5bc712f704461592248840a4358bf4 -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/mnt/vss/_work/1":"/__w/1" -v "/mnt/vss/_work/_temp":"/__w/_temp" -v "/mnt/vss/_work/_tasks":"/__w/_tasks" -v "/mnt/vss/_work/_tool":"/__t" -v "/usr/local/vss-agent/4.265.1/externals":"/__a/externals":ro -v "/mnt/vss/_work/.taskkey":"/__w/.taskkey" ***/devops/deploy:alpine3.20 "/usr/local/bin/node" -e "setInterval(function(){}, 24 * 60 * 60 * 1000);"
3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559
DockerActionRetries variable value: True
/usr/bin/docker start 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559
3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559
Final exit code for Docker start: 0
/usr/bin/docker ps --all --filter id=3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 Up Less than a second
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 sh -c "command -v bash"
whoami
AzDevOps
id -u AzDevOps
1002
id -g AzDevOps
1002
id -gn AzDevOps
AzDevOps
Try to create a user with UID '1002' inside the container.
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 bash -c "getent passwd 1002 | cut -d: -f1 "
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 bash -c "if [[ -e '/etc/alpine-release' ]]; then echo 'Alpine-based image detected.'; fi"
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 apk list --installed | grep shadow
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 useradd -m -u 1002 AzDevOps_azpcontainer
Grant user 'AzDevOps_azpcontainer' SUDO privilege and allow it run any command without authentication.
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 groupadd azure_pipelines_sudo
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 usermod -a -G azure_pipelines_sudo AzDevOps_azpcontainer
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 su -c "echo '%azure_pipelines_sudo ALL=(ALL:ALL) NOPASSWD:ALL' >> /etc/sudoers"
Allow user 'AzDevOps_azpcontainer' run any docker command without SUDO.
stat -c %g /var/run/docker.sock
998
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 bash -c "cat /etc/group"
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 groupadd -g 998 azure_pipelines_docker
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 usermod -a -G azure_pipelines_docker AzDevOps_azpcontainer
/usr/bin/docker exec 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 bash -c "/__a/externals/node24/bin/node -v"
Finishing: Initialize containers
---
Starting: Get Dockerfile directory, sanitize branchName and format version 3.2 ./base/Dockerfile main
==============================================================================
Task : PowerShell
Description : Run a PowerShell script on Linux, macOS, or Windows
Version : 2.266.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
==============================================================================
Error relocating /__a/externals/node24/bin/node: fcntl64: symbol not found
##[error]Exit code 127 returned from process: file name '/usr/bin/docker', arguments 'exec -i -u 1002 -w /home/AzDevOps_azpcontainer 3f763e445560ca100be30fa8b1bfc86d2ac92b7a872a44cdcd4a7bed19870559 /__a/externals/node24/bin/node /__w/_temp/containerHandlerInvoker.js'.
Finishing: Get Dockerfile directory, sanitize branchName and format version 3.2 ./base/Dockerfile main