WSL icon indicating copy to clipboard operation
WSL copied to clipboard

WSL 2.5.1 breaks Docker Desktop

Open reynoldsbd opened this issue 10 months ago • 35 comments

Windows Version

Microsoft Windows [Version 10.0.26100.3476]

WSL Version

2.5.1.0

Are you using WSL 1 or WSL 2?

  • [x] WSL 2
  • [ ] WSL 1

Kernel Version

6.6.75.1-1

Distro Version

Azure Linux 3.0

Other Software

Docker Desktop for Windows, version 4.38

Repro Steps

Install Docker Desktop for Windows, then try to launch it.

Expected Behavior

Docker Desktop starts successfully, and I am able to build and run containers.

Actual Behavior

Docker Desktop attempts to start, then fails, and the UI shows "Docker Desktop is stopped"

At the time of failure, a Windows notification appears which reads "Errors occurred during WSL startup", with a button to open Event Viewer. I tried clicking the button, but event viewer never opened. I opened it manually, but I wasn't able to find the path for WSL logs. If somebody can tell me the path in Event Viewer I'd be happy to look up the logs.

After reverting WSL from 2.5.1 to 2.4.12, Docker Desktop is able to successfully start again.

Aside - I can see 2.5.1 is marked as pre-release, yet it somehow got deployed to my workstation. I'd love to understand how this happened and if there's a way to prevent my machine from receiving pre-release builds.

Diagnostic Logs

No response

reynoldsbd avatar Mar 18 '25 19:03 reynoldsbd

Logs are required for review from WSL team

If this a feature request, please reply with '/feature'. If this is a question, reply with '/question'. Otherwise please attach logs by following the instructions below, your issue will not be reviewed unless they are added. These logs will help us understand what is going on in your machine.

How to collect WSL logs

Download and execute collect-wsl-logs.ps1 in an administrative powershell prompt:

Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/WSL/master/diagnostics/collect-wsl-logs.ps1" -OutFile collect-wsl-logs.ps1
Set-ExecutionPolicy Bypass -Scope Process -Force
.\collect-wsl-logs.ps1

The script will output the path of the log file once done.

If this is a networking issue, please use collect-networking-logs.ps1, following the instructions here

Once completed please upload the output files to this Github issue.

Click here for more info on logging If you choose to email these logs instead of attaching to the bug, please send them to [email protected] with the number of the github issue in the subject, and in the message a link to your comment in the github issue and reply with '/emailed-logs'.

github-actions[bot] avatar Mar 18 '25 19:03 github-actions[bot]

+1 Updated to 2.5.1 manually as it contains an updated kernel, claiming to fix high CPU issue on Windows in Arm. Docker no longer starts with:

running engine: waiting for the Docker API: engine linux/wsl failed to run: starting WSL engine: error spotted in wslbootstrap log: "[2025-03-19T18:57:39.495145800Z][wsl-bootstrap][F] exit status 32"

I am not the author, but find my logs from WSL and Docker in attachment.

PS C:\Users\egrig> wsl --version
  WSL version: 2.5.1.0
  Kernel version: 6.6.75.1-1
  WSLg version: 1.0.66
  MSRDC version: 1.2.5716
  Direct3D version: 1.611.1-81528511
  DXCore version: 10.0.26100.1-240331-1435.ge-release
  Windows version: 10.0.26120.3576
PS C:\Users\egrig> wsl --status
  Default Distribution: Ubuntu
  Default Version: 2
PS C:\Users\egrig> wsl -d Ubuntu cat /etc/lsb-release
  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=24.04
  DISTRIB_CODENAME=noble
  DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"
PS C:\Users\egrig> wsl -d Ubuntu uname -a
  Linux RS-UNIT-0240 6.6.75.1-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC Sat Feb  8 02:03:26 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

The repro on my side:

  1. [Optional] Shutdown WSL with wsl --shutdown // log collection started here //
  2. Start Docker Desktop and wait
  3. In 20 seconds a message "Error occurred during WSL startup appears". The only button "Open EventViewer" is no-op, dismisses the notification.
  4. In the background Docker tries to start the engine, then changes to "Stopping the engine", and the "An unexpected error occurred" window appears. // log collection stopped here //

WslLogs-2023-10-04_11-50-46.zip

20250319190801.zip

evgri243 avatar Mar 19 '25 19:03 evgri243

There's definitely some interaction going on between the two. My WSL wouldn't start when I tried to open my default distro. It would just "hang" and then throw a timeout error. But if during this time I exit Docker Desktop, my default distro loads successfully.

Then I noticed the kernel was updated from 5 to 6 in the latest pre-release version, and there's a few more things seemingly broken/throwing error messages. I had the same issues last time kernel v6 was introduced in a pre-release version a few months ago, and that time it was reverted to v5 completely after a while.

My bad for installing a pre-release version and not being prepared for stuff breaking.. Even worse, this time, I completely missed the kernel upgrade in the changelog.. Gonna go back to v5 for now and just... put my head in the sand and wait for a while.

jtabox avatar Mar 19 '25 22:03 jtabox

Oh, downgraded and everything magically worked again. I pulled my hair for a few days there.

serbrech avatar Mar 20 '25 02:03 serbrech

Yeah I'm no longer able to repro after reverting back to 2.4.12.

Strangely, I even tried manually upgrading back to 2.5.1, in an attempt to repro the issue. But now Docker starts with no issue, even on 2.5.1.

reynoldsbd avatar Mar 20 '25 04:03 reynoldsbd

I am seeing the same issue after install the 2.5.1 pre-release. Strangely enough, the last release with the 6.6 kernel (https://github.com/microsoft/WSL/releases/tag/2.3.14) works fine with Docker Desktop. I wonder what changed?

ryanwinter avatar Mar 25 '25 22:03 ryanwinter

How do you downgrade?

sabbour avatar Mar 25 '25 22:03 sabbour

How do you downgrade?

Through prayer and sacrifice mainly.

I've heard you can also just download any 2.4.X version (e.g. the latest "Release" version) from https://github.com/microsoft/WSL/releases and run the installer.

But prayer and sacrifice feels simpler.

jtabox avatar Mar 26 '25 01:03 jtabox

I've heard you can also just download any 2.4.X version (e.g. the latest "Release" version) from https://github.com/microsoft/WSL/releases and run the installer.

Thanks for sharing! That works, without wiping out the existing distros.

rayluo avatar Mar 28 '25 20:03 rayluo

I assume I managed to figure out the issue. Interestingly, it should affect only AArch64 builds as far as I understand from configs. But it seems some run the x86_64 version of Docker at least. Anyone one with x86_64 device and the same error code, what error do you see in "C:\Users\<USER>\AppData\Local\Docker\log\host\com.docker.backend.exe.log"?

Until it is released or fixed by WSL itself, it requires the WSL kernel to be rebuilt. Interestingly, it is much easier than expected.

TL;DR; Rebuild WSL Kernel and modules with arch/arm64/configs/config-wsl-arm64 patched to support *.iso disks at the kernel level (CONFIG_ISO9660_FS=m).

Instructions

  1. Clone the repository into Linux. Consider using the depth parameter as the repository is huge:
git clone --depth=1 https://github.com/microsoft/WSL2-Linux-Kernel.git
cd WSL2-Linux-Kernel
  1. Open arch/arm64/configs/config-wsl-arm64 and enable CD-ROM/DVD configuration.
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
# end of CD-ROM/DVD Filesystems

Check my private branch in evgri243/WSL2-Linux-Kernel for an example.

  1. Following the instructions at the official README.md, rebuild the kernel and the modules. In short:
sudo apt install build-essential flex bison dwarves libssl-dev libelf-dev cpio qemu-utils

# rebuild the kernel
# use defaults (Enter) for unset configuration settings
make KCONFIG_CONFIG=Microsoft/config-wsl-arm64

# "Install" the compiled modules into a separate folder
make INSTALL_MOD_PATH="$PWD/modules" modules_install

# Build modules.vhdx
sudo ./Microsoft/scripts/gen_modules_vhdx.sh "$PWD/modules" $(make -s kernelrelease) modules.vhdx
  1. Copy the built arch/arm64/boot/Image and modules.vhdx onto Windows system drive. It may be worth doing it from Windows either via PowerShell or UI via \\wsl.localhost as file permissions issues may occur during WSL startup. If happens, make sure your current user has all permissions and ownership over these files in Windows. Nevertheless, adding bash commands for consistency:
mkdir /mnt/c/WSL
cp arch/arm64/boot/Image /mnt/c/WSL/
cp modules.vhdx /mnt/c/WSL/
  1. Configure WSL to use the custom Kernel and modules either via Develop Settings section of WSL Settings GUI application or add the lines below to yours C:/Users/<USER>/.wslconfig file:
[wsl2]
kernel=C:\\WSL\\Image
kernelModules=C:\\WSL\\modules.vhdx

There is no way to raise a PR or tag @kelsey-steele, the author of the commit flipping the flags 6 months ago. In case someone has permission to tag him:

@kelsey-steele, was there any particular reason to flip ISO9660-related modules or it was an overlook during merging to a later kernel version? Can it be returned in the upcoming release? It is required for Docker for Windows, and having the ability to mount .iso images may still be useful in WSL scenarios.

Image

Extra details

The full WSLBootstrapper error found in "C:\Users\<USER>\AppData\Local\Docker\log\host\com.docker.backend.exe.log.3" looked this way:

[2025-03-28T16:52:03.217865200Z][com.docker.backend.exe.socketforward] connected to WSL distro docker-desktop
[2025-03-28T16:52:03.233514600Z][com.docker.backend.exe.WSLBootstrap] [2025-03-28T16:52:03.233972700Z][wsl-bootstrap.isocache] docker-wsl-cli.iso is present in the cache
[2025-03-28T16:52:03.250933500Z][com.docker.backend.exe.WSLBootstrap] Error: mounting wslCLIDest: mount -o ro,exec /mnt/host/wsl/docker-desktop-data/isocache/entries/docker-wsl-cli.iso/a292b78221be200c9386b0d3afc391fc30668c8d70415bf8e1491f320757e91f /mnt/host/wsl/docker-desktop/cli-tools: mount: /mnt/host/wsl/docker-desktop/cli-tools: unknown filesystem type 'iso9660'.
[2025-03-28T16:52:03.250933500Z][com.docker.backend.exe.WSLBootstrap]        dmesg(1) may have more information after failed mount system call.
[2025-03-28T16:52:03.250933500Z][com.docker.backend.exe.WSLBootstrap] : exit status 32
[2025-03-28T16:52:03.251712000Z][com.docker.backend.exe.WSLBootstrap] [2025-03-28T16:52:03.251154100Z][wsl-bootstrap][F] exit status 32
[2025-03-28T16:52:03.251712000Z][com.docker.backend.exe.wslprovisioner][E] WSLBootstrap: error spotted in wslbootstrap log: "[2025-03-28T16:52:03.251154100Z][wsl-bootstrap][F] exit status 32"

Apparently, the same error was already discussed during the previous 6.6 kernel update #11742, and it concluded that

  1. Docker uses .iso disk images to mount CLI binaries;
  2. The latest Kernel version disables those kernel modules.

After some experimenting, Docker for Windows worked with CONFIG_ISO9660_FS=m enabled. Although, apparently only CONFIG_ISO9660_FS=m is required, the rest are enabled for consistency with x86_64 configuration in arch/x86/configs/config-wsl-arm64.

evgri243 avatar Mar 28 '25 21:03 evgri243

How do you downgrade?

@sabbour @rayluo @jtabox it should be safe from my experience to execute "wsl --uninstall" or remove it manually from "Add or remove programs", and then reinstall from Releases as mentioned by @jtabox . These operations shouldn't affect the distros and only uninstall and install the kernel with the WSL feature itself. "--unregister <Distro>" is, on the contrary, removes the repo unrecoverably.

evgri243 avatar Mar 28 '25 21:03 evgri243

How do you downgrade?

@sabbour @rayluo @jtabox it should be safe from my experience to execute "wsl --uninstall" or remove it manually from "Add or remove programs", and then reinstall from Releases as mentioned by @jtabox . These operations shouldn't affect the distros and only uninstall and install the kernel with the WSL feature itself. "--unregister " is, on the contrary, removes the repo unrecoverably.

Actually, I tried @jtabox 's method yesterday and it worked. So, you do NOT even need to manually uninstall the existing newer version.

rayluo avatar Mar 29 '25 18:03 rayluo

Issue crept up for me again today. Logs have been sent via email.

Two interesting observations:

  1. When Docker fails to start, I actually see several different error messages. So possibly there are multiple issues. Not clear which of them are on Docker side or WSL.
  2. In some cases (but not always?), having another WSL instance running in a terminal window before starting the Docker Desktop process seems to mitigate the issue.
    • Sometimes the Docker error message mentions an issue with the wsl$ share path. Possible explanation could be some race condition between Docker and WSL, e.g. Docker is trying to access the share path before WSL has fully initialized. Having some unrelated distro already running in the background before starting Docker somehow ensures that the right pieces (within WSL?) are already up and running, thereby sidestepping the race.

reynoldsbd avatar Mar 31 '25 16:03 reynoldsbd

There is a quite constant issue on my fix with docker failing to "something creating groups" on Ubuntu repo. Usually it just required to start the distro in the terminal and ask the docker to retry.

Generally, at least on arm64 the issue seems to come from the missing .iso driver, -- and it is clearly the cause as the arm64 kernel config has turned it off.

I wonder how to get the attention of the people behind the kernel to recompile it for arm64 and release a fix. It can't be intentional, there is no easy fix otherwise...

evgri243 avatar Apr 01 '25 20:04 evgri243

Should be fixed with https://github.com/microsoft/WSL/releases/tag/2.5.6, please let me know if you're still seeing issues.

benhillis avatar Apr 08 '25 20:04 benhillis

I'm on 2.5.7 on windows ARM64 and getting this error : running engine: waiting for the Docker API: engine linux/wsl failed to run: starting WSL engine: error spotted in wslbootstrap log: "[2025-04-25T13:13:10.679969741Z][wsl-bootstrap][F] exit status 32" The Docker support find this in the log : mount: ... unknown filesystem type 'iso9660' exit status 32

shaibarlev avatar Apr 25 '25 13:04 shaibarlev

@benhillis Running 2.5.7 and it doesn't appear to be fixed. WSL version: 2.5.7.0 Kernel version: 6.6.87.1-1 WSLg version: 1.0.66 MSRDC version: 1.2.6074 Direct3D version: 1.611.1-81528511 DXCore version: 10.0.26100.1-240331-1435.ge-release Windows version: 10.0.26100.3775

Same symptoms, same symptoms as discussed above: In 20 seconds a message "Error occurred during WSL startup appears". The only button "Open EventViewer" is no-op, dismisses the notification. Then the VM is unusable.

neilmfrench avatar May 05 '25 05:05 neilmfrench

Everyone, it seems that I have encountered a problem similar to yours, but the difference is that on my machine. This problem only causes the 'Errors occurred during WSL startup' pop-up window to keep popping up, and does not cause any substantial glitch

calloftimereiver avatar May 08 '25 15:05 calloftimereiver

I have the same problem on my machine. It also shows 'Errors occurred during WSL startup' even though nothing seems to be happening. However, it only happens once every time I reboot and start Docker for the first time. I tried quitting Docker then starting it again and it doesn't show that message anymore.

Image

I also just noticed that when I start Docker it suddenly pops up quite similar processes in Task Manager. Before: Image After: Image

NHKritDat avatar May 16 '25 03:05 NHKritDat

Everyone, it seems that I have encountered a problem similar to yours, but the difference is that on my machine. This problem only causes the 'Errors occurred during WSL startup' pop-up window to keep popping up, and does not cause any substantial glitch

Yes, same experience here, but it's very disconcerting for us users.

MisterTickle avatar May 28 '25 09:05 MisterTickle

Should be fixed with https://github.com/microsoft/WSL/releases/tag/2.5.6, please let me know if you're still seeing issues.

I am definitely still seeing it on 2.5.7, sorry.

MisterTickle avatar May 28 '25 09:05 MisterTickle

same issue with 2.5.7.

koepalex avatar May 28 '25 12:05 koepalex

Hey everyone, I'm running into the same issue as NHKritDat, and it's really wearing me out trying to figure it out. I'd really appreciate any help—please lend a hand!

brianSanders-ai avatar May 29 '25 12:05 brianSanders-ai

Hey everyone, I'm running into the same issue as NHKritDat, and it's really wearing me out trying to figure it out. I'd really appreciate any help—please lend a hand!

I recommend you use 2.4.x before Microsoft fix it

calloftimereiver avatar May 29 '25 13:05 calloftimereiver

Also reporting the same issue with 2.5.7. Uninstalled and installed https://github.com/microsoft/WSL/releases/download/2.4.13/wsl.2.4.13.0.x64.msi which is now working fine.

baumatron avatar May 29 '25 17:05 baumatron

I also had to downgrade to 2.4 in order to keep using WSL and Docker Desktop.

jbruni avatar May 29 '25 17:05 jbruni

Also reporting the same issue with 2.5.7. Uninstalled and installed https://github.com/microsoft/WSL/releases/download/2.4.13/wsl.2.4.13.0.x64.msi which is now working fine.

I updated everything today, both WSL and Docker and I have the same problem. How do you downgrade? Do you lose files in this downgrade?

devslimbr avatar May 29 '25 18:05 devslimbr

its happening for me too I think maybe yesterday my Windows updated when shutting down, what do I need to do? Its my main development machine and I have a lot of stuff on here, which will take a lot of work to restore. Can I just downgrade safely?

ro8inmorgan avatar May 30 '25 07:05 ro8inmorgan

Thank you for your support [baumatron, calloftimereiver ]. I followed the steps in the guide, but unfortunately, the issue still isn't resolved on my end. I don't know why. Please check below my PC's status. I installed Docker on Windows 11.

Image And below 2 files are logs. com.docker.backend.exe.log com.docker.backend.exe.log monitor.log monitor.log

Image

brianSanders-ai avatar May 30 '25 12:05 brianSanders-ai

Thank you for your support [baumatron, calloftimereiver ]. I followed the steps in the guide, but unfortunately, the issue still isn't resolved on my end. I don't know why. Please check below my PC's status. I installed Docker on Windows 11.

Image And below 2 files are logs. com.docker.backend.exe.log com.docker.backend.exe.log monitor.log monitor.log

Image

I can post my versions, I hope that can help you. Image

calloftimereiver avatar May 30 '25 20:05 calloftimereiver