Blog
Blog Novacare

When Azure DevOps deploy suddenly dies

Published
by Henning Sjørbotten
azure devops

Having done nothing but deploying from Azure Devops start failing with crypto.randomUUID is not a function? The code is the same, the pipeline is the same, the servers are the same. And yet one afternoon a release that had deployed cleanly for years fails on the IIS Web App Deploy step with an error that, at first glance, looked like it had nothing to do with a .NET/IIS deployment at all:

Unhandled: crypto.randomUUID is not a function
TypeError: crypto.randomUUID is not a function
    at Vault.genKey (C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_...\vault.js:65:48)
    at new Vault (...\vault.js:21:14)
    at Object._loadData (...\internal.js:654:22)
    ...

If you run self-hosted deployment group agents on your own servers, there is a good chance this post will save you an afternoon. Here is what actually causes it, why "just retry" never works, and how to fix it safely.

The instinctive wrong move: retry

The first thing everyone does is hit Redeploy, because the failure feels transient. It isn't. It will fail identically every single time until the underlying cause is addressed. Retrying just burns time.

So what is actually failing? The error message points at a layer most of us never think about.

Two different "Node.js" live in the pipeline

There are two completely independent Node.js installations involved in a typical build-and-deploy setup, and they have nothing to do with each other:

  1. The Node your front-end build uses. If you have a NodeTool@0 task and an .nvmrc file, that Node runs during the build on the build agent (for npm ci, npm run build, and so on). This is the "Node version in the code" that most developers think of.
  2. The Node bundled inside the Azure DevOps agent itself. Every agent ships its own private Node runtime under externals\, and it uses that Node to execute the task handlers - including the IIS Web App Deploy task. This Node is invisible during normal work and has nothing to do with the application.

The crash is in number two. The IIS Web App Deploy task is implemented in JavaScript, and one of its helpers (Vault.genKey) calls crypto.randomUUID(). That function did not exist in Node until version 14.17. If the agent's bundled Node is older than that, the call throws is not a function.

The application never touches Node at runtime - it's .NET on IIS. Node here is purely the agent's own tool for moving files into place and cycling the app pool. So this is entirely a "getting files deployed" problem, not a "how the site runs" problem.

Why it broke without anyone changing anything

Here's the timeline that catches everyone out.

Microsoft periodically updates the built-in pipeline tasks cloud-side, automatically, without you doing anything. At some point the IIS Web App Deploy task (and the azure-pipelines-task-lib it depends on) was updated to a version that assumes a modern Node runtime and calls crypto.randomUUID.

Meanwhile, your self-hosted agent has been sitting on the same old version for years. Older agents bundle old Node. Mine were on agent version 2.194.0, whose externals\ folder contained only node (v6) and node10 (v10). Both well below the 14.17 threshold.

Self-hosted deployment group agents do not auto-update the same way Microsoft-hosted build agents do. They will happily stay outdated until you intervene.

Confirming the diagnosis

Two quick checks will nail it down.

Check the agent version in Azure DevOps. Go to your deployment group (Pipelines → Deployment groups → your group → the target). The Agent version field is shown right there. If it reads something like 2.194.0, that's your prime suspect.

Check the bundled Node on the server. Go to the server and look inside the agent's externals\ folder:

Get-ChildItem "C:\azagent\A1\externals" -Directory

On an old agent you'll see something like:

azcopy
git
node
node10
nuget
...

Only node and node10. No node16, no node20. Confirm the versions if you like:

& "C:\azagent\A1\externals\node10\bin\node.exe" --version

If the newest Node available to the agent is below v14.17, there is no way for the task to find crypto.randomUUID. No amount of pinning the task version helps, the IISWebAppDeploymentOnMachineGroup task only has one major version (0), and the breaking change happened within it.

The fix: upgrade the agent

The real fix is to upgrade the agent to a current version, which bundles a modern Node (the current agent ships Node 20 and Node 24). There is no config toggle for "use newer Node", you just have to replace the agent.

1. Get the registration script

In Azure DevOps, open your deployment group and click Register / New, choose Windows as the target type, and copy the generated PowerShell registration script. It downloads the latest agent and runs config.cmd with your organization URL, deployment group name, project, and (optionally) a PAT in it.

The agent name is set automatically from $env:COMPUTERNAME, so the same script works on every server - it adapts to the machine it runs on so both test and prod servers uses the same script.

2. Confirm which server you're on

This sounds trivial, but when you're jumping between a test and a prod box it's dangerously easy to run a destructive command on the wrong one:

hostname

Make sure it's the machine you intend before going further.

3. Remove the old agent

From an administrator PowerShell prompt:

cd C:\azagent\A1
.\config.cmd remove

This uninstalls the Windows service and unregisters the agent. It does not touch IIS or your site.

A gotcha worth flagging: on one of our servers this hung for 12+ minutes on "Removing agent from the server". If that happens, don't wait it out because that will probably never happen. The service is already removed at that point (Succeeded: Removing service), and the unregister step is just cleanup. Cancel it and kill the process:

Get-Process -Name "Agent.Listener" -ErrorAction SilentlyContinue | Select-Object Id, Name
Stop-Process -Id <PID> -Force

Then check the server can actually reach DevOps (a hung unregister is sometimes a network/proxy issue, and if so the registration will hang too):

Test-NetConnection dev.azure.com -Port 443

If TcpTestSucceeded is True, you're clear to proceed. An unclean remove is not blocking, the registration step cleans up the leftover entry via a prompt (see below).

On our other server, remove instead prompted for authentication type and a PAT - that prompt is what the first server was silently stuck on. Enter for PAT (from DevOps or just copy it from the script if it's added there), then paste the token.

4. Run the registration script

Paste the copied script into the same admin prompt. It downloads agent 5.277.0 into a new folder (e.g. C:\azagent\A2). Nothing old is being overwritten.

You'll be prompted along the way. The important answers:

  • Pool ... already contains an agent with name <SERVER>. Enter replace? (Y/N) → Y. This reuses the existing target entry instead of creating a duplicate, so the agent version updates in place. (This is also how the leftover from an unclean remove gets cleaned up.)
  • Enter deployment group tags for agent? (Y/N) → Y, then type the tag. This is the one thing that differs per server — ours was test on the test box and prod on production. Get this wrong and the release stage won't find its agent.
  • SERVICE_SID_TYPE_UNRESTRICTED (Y/N) → Enter (N).
  • User account to use for the service → Enter for NT AUTHORITY\SYSTEM, unless the old agent ran as a specific service account with special file/network rights. SYSTEM has ample local rights to write to IIS folders and cycle the app pool; its one limitation is network access (it acts as the machine account), which only matters if your deploy reaches a network share.
  • Prevent service starting immediately? (Y/N) → Enter (N), so the agent comes online right away.

5. Verify before deploying

Back in Azure DevOps, refresh the deployment group target and confirm:

  • Agent version: now 5.277.0 (not 2.194.0)
  • Status: Online
  • Tag: correct for that server (test/ prod)

And on the server, confirm the modern Node is present:

Get-ChildItem "C:\azagent\A2\externals" -Directory

You should now see node20 (and node24). That's the actual root cause resolved, and the task can now find crypto.randomUUID.

6. Redeploy

You don't need a new release. The original failed release still has its artifact attached, and the artifact was never the problem - the agent was. Open the failed release, and Redeploy. It reruns the same artifact against the upgraded agent and goes green.

There's a nice property here: redeploying the exact release that failed proves it was the agent and nothing else.

Takeaways

  • crypto.randomUUID is not a function on a deploy step is almost always an outdated self-hosted agent with a pre-14.17 bundled Node, not anything in your code.
  • It appears "spontaneously" because Microsoft updates the built-in tasks cloud-side while your agent's Node stays frozen.
  • Retrying and reverting code are both dead ends. The fix is to upgrade the agent.
  • The upgrade touches only the tooling that performs the deploy. The running IIS site is never affected.

Old infrastructure has a way of working perfectly right up until an external dependency shifts under it. This one had been quietly deploying for years; it just needed the agent brought into the current decade..