Libraries could not be resolved by the "cdnjs" provider

[duykhiem]

Describe the bug

We are having an issue with the "cdnjs" provider when building our project in both Hanoi, Vietnam, and Grimstad, Norway. It is happening with both local build and Azure DevOps Pipeline build. Failed on some random libraries like [email protected], [email protected], [email protected]...

Error:

Restore operation started
##[error]libman.json(0,0): Error LIB002: The "[email protected]" library could not be resolved by the "cdnjs" provider
     1>libman.json : error LIB002: The "[email protected]" library could not be resolved by the "cdnjs" provider 
         One or more libraries failed to restore
         
Build FAILED.

       (LibraryManagerRestore target) -> 
         **libman.json : error LIB002: The "[email protected]" library could not be resolved by the "cdnjs" provider** 

To Reproduce

Steps to reproduce the behavior:

1. Using this version of Visual Studio 2022
2. Build the solution
3. See error:

Expected behavior

Build and restore client libraries successfully

Screenshots

Additional context

• We are using cdnjs as the default provider in LibMan
• It was working perfectly before but suddenly failed today without any changes related to Libman or package references

LibMan CLI version: 2.1.161+abc97ecc7d.RR Visual Studio 2022 Microsoft.Web.LibraryManager.Build" version="2.1.161" targetFramework="net472"

[duykhiem]

The workaround is to clear Libman cache https://github.com/cdnjs/cdnjs/issues/14141#issuecomment-1160220546

[stebueh]

possibly related to cdnjs API change, see https://github.com/cdnjs/cdnjs/issues/14140; it's being rolled out partially, which would explain the intermittent failure

[duykhiem]

The workaround is to clear Libman cache cdnjs/cdnjs#14141 (comment)

Ah. Cleaning cache is just a lucky shot

I think the main issue is related to https://github.com/cdnjs/cdnjs/issues/14140

Libman still uses the old endpoint: https://github.com/aspnet/LibraryManager/blob/8dd0ac5fdfe76c2d37ded284834c67633cc18d7f/src/LibraryManager/Providers/Cdnjs/CdnjsCatalog.cs#L23

Failed restore/install fall into their 10% production request https://github.com/cdnjs/cdnjs/issues/14140#issuecomment-1159470071

I think clean cache somehow and sometime helps our request to fall outside of above 10% :D

[duykhiem]

possibly related to cdnjs API change, see cdnjs/cdnjs#14140; it's being rolled out partially, which would explain the intermittent failure

yes I think so

[MattIPv4]

Hey folks, just wanted to note that after a quick read through the code here, it does seem like this is caused by the deprecation of the assets property returning every version on cdnjs.

The fix for libman here would be to use the versions property to determine the library versions, not assets. And then to use the api.cdnjs.com/libraries/:library/:version endpoint when it needs to actually get the assets for a specific version.

[jthomas81]

Does someone have an example of syntax to do this? This is what I have been doing: { "version": "1.0", "defaultProvider": "cdnjs", "libraries": [ { "library": "[email protected]", "destination": "wwwroot/lib/jquery-validate/" } ] }

[the-doctor1994]

Does someone have an example of syntax to do this? This is what I have been doing:

it's not a change for the end user, MattIPv4 is suggesting LibMan issue an update to their code, something that could take days, to conform to the altered API behavior.... instead of just reverting the cdnjs api change that is causing this issue

[MattIPv4]

Unfortunately, LibMan is not the only consumer of the cdnjs API, and the deprecation that caused this issue is part of larger ongoing work across the API to improve stability (the cdnjs API often runs 5xx errors for assorted consumers, not just LibMan users). We're not really in a position where we can revert this API change as doing so will only increase the failure rate for the API generally.

[the-doctor1994]

I understand the change is necessary and inevitable, but time could and should be given to allow for proper deprecation updates by all the users of the API who may be negatively affected (or outright broken) by this change. By your own proposal the deprecation was fast-tracked under the incorrect assumption nobody would be affected...

[MattIPv4]

I appreciate what you're saying, and this is why we're approaching this as a slow rollout with only 10% of traffic reaching endpoints that implement this deprecation, so that folks are made aware of if they're impacted by the deprecation (turns out it's pretty impossible to determine who consumes what properties of a public API just looking at unauthenticated GET requests), but can also still easily get around the deprecation by just re-requesting (falling into the 90% of traffic that is unaffected at present).

Until we hear more here in terms of what a fix would look like on LibMan's end, we don't intend to further rollout the change, so it will remain at 10% for now. That being said, ultimately we will have to continue with this deprecation as until we complete this rollout a large number of requests to the API from all consumers, not just LibMan users, will continue to fail unexpectedly.

[jimmylewis]

Just quickly jumping in here to say: I'm working on the code change for LibMan, but the rollout will require servicing each version of Visual Studio that includes it (2017, 2019, 2022). There will be some delays to prepare the LibMan builds and VS servicing changes in each product, and then ultimately it will require customers to install the updates as well.

I don't have timelines yet, I'm focusing first on making the code change for VS2022 and our CLI tool, then I will start investigating the servicing requirements for each VS release.

[duykhiem]

I think the safest and fastest way to workaround this issue is to include the successfully generated static files by LibMan to our project (when our restore client-side libraries command falls into the 90% of traffic that is unaffected at present) and then disable LibMan. We can re-enable it when the issue is fixed.

Another workaround is try to use other CDN providers like filesystem, jsdelivr, unpkg instead of cdnjs for LibMan

Cleaning LibMan cache is not really a workaround for this issue. Our build will still have a failure probability of 10% even though we tried to clear the LibMan cache

[gavin-theta]

I'm not sure if this will help but in our case we had a "dotnet restore" followed by a "dotnet publish" the publish was restoring the packages again. So I disable that by "dotnet publish --no-restore" and the build is now happy.

[ennerperez]

Also, Azure Pipelines are failing too.

[gavin-theta]

Also, Azure Pipelines are failing too.

Try my fix above. The azure restore seems to work as a different step. Then disable the restore in the publish. Worked for me.

[jimmylewis]

Alright, for folks using the CLI tool or MSBuild task could you please update to the latest version (2.1.174) and see if you have any issues with that? From a few projects I've tried, it's working fine. This will be the fastest way to workaround the issue (at least for unblocking CI or local builds) as the VS servicing cycle will take a longer to sort out.

Timeline on VS updates still TBD...

[calvaradocl]

On our azure pipelines we are getting this error after updating the package to the 2.1.174 version, this persists even if we have Newtonsoft.Json installed.

/root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: The "Microsoft.Web.LibraryManager.Build.RestoreTask" task failed unexpectedly. [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: System.TypeInitializationException: The type initializer for 'Microsoft.Web.LibraryManager.Cache.WebRequestHandler' threw an exception. ---> System.TypeInitializationException: The type initializer for 'Microsoft.Web.LibraryManager.Configuration.Settings' threw an exception. ---> System.IO.FileLoadException: Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621) [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: at Microsoft.Web.LibraryManager.Configuration.Settings..ctor() [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: at Microsoft.Web.LibraryManager.Configuration.Settings..cctor() [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: --- End of inner exception stack trace --- [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: at Microsoft.Web.LibraryManager.Configuration.ProxySettings.get_Default() [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: at Microsoft.Web.LibraryManager.Cache.WebRequestHandler..cctor() [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: --- End of inner exception stack trace --- [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: at Microsoft.Web.LibraryManager.Build.Contracts.Dependencies.Initialize() [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: at Microsoft.Web.LibraryManager.Build.RestoreTask.Execute() [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/src/Frontend/Frontend.csproj] /root/.nuget/packages/microsoft.web.librarymanager.build/2.1.174/build/Microsoft.Web.LibraryManager.Build.targets(35,9): error MSB4018: at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/src/Frontend/Frontend.csproj] 2 Warning(s) 1 Error(s)

[stebueh]

updating the MSBuild task fixed the issue for me building locally, thank you @jimmylewis for the timly fix; our azure devops pipeline is using the CLI tool and was running flawlessly all the time, so I guess it is/was still on the 90% of traffic unaffected by the API change.

[ardalis]

I appreciate what you're saying, and this is why we're approaching this as a slow rollout with only 10% of traffic reaching endpoints that implement this deprecation, so that folks are made aware of if they're impacted by the deprecation (turns out it's pretty impossible to determine who consumes what properties of a public API just looking at unauthenticated GET requests), but can also still easily get around the deprecation by just re-requesting (falling into the 90% of traffic that is unaffected at present).

Running into Hyrum's Law:

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

[farhadmammadli]

cdnjs issue is resolved in the new update of the LibraryManager package. the updated version is 2.1.174

[thiago-bessa]

I just tried the new version, and I'm still having problems:

C:\Users\thiago.bessa>dotnet tool install -g Microsoft.Web.LibraryManager.Cli
You can invoke the tool using the following command: libman
Tool 'microsoft.web.librarymanager.cli' (version '2.1.174') was successfully installed.

libman restore
[LIB002]: The "[email protected]" library could not be resolved by the "cdnjs" provider
[LIB002]: The "[email protected]" library could not be resolved by the "cdnjs" provider

[jimmylewis]

@thiago-bessa try again after running libman cache clean? We use the cache library metadata to disk for 1 day (IIRC of the top of my head) to avoid hitting the service repeatedly on queries, so if you have recently cached metadata from the older build, it might be conflicting with the changes.

I was able to restore fine with this libman.json file

{
  "version": "1.0",
  "defaultProvider": "cdnjs",
  "libraries": [
    {
      "library": "[email protected]",
      "destination": "lib\\jquery"
    },
    {
      "library": "[email protected]",
      "destination": "lib\\jquery-validate"
    }
  ]
}

[farhadmammadli]

I just tried the new version, and I'm still having problems:

C:\Users\thiago.bessa>dotnet tool install -g Microsoft.Web.LibraryManager.Cli
You can invoke the tool using the following command: libman
Tool 'microsoft.web.librarymanager.cli' (version '2.1.174') was successfully installed.

libman restore
[LIB002]: The "[email protected]" library could not be resolved by the "cdnjs" provider
[LIB002]: The "[email protected]" library could not be resolved by the "cdnjs" provider

have you tried to clean cache using this? libman cache clean

[thiago-bessa]

Sorry @farhadmammadli and @jimmylewis, I didn't copy/paste everything.

I did run libman cache clean before running libman restore, and it still failed.

Now I ran it an hour later (same cmd prompt, with the same project), and it could restore everything.

Then I ran it in other project of ours (after the first one restored correctly), and it again failed to resolve:

Then I just ran restore for 3 times in a row, it finally worked. It seems kind of intermittent.

[alex-jitbit]

I have the same issue, and clearing cache helps. Which probably means the issue is inside libman, not cdnjs?

[Cloudfare123]

Our Web solutions all fixed now locally and CI by updating to latest package. 👍

[easlavov]

On our azure pipelines we are getting this error after updating the package to the 2.1.174 version, this persists even if we have Newtonsoft.Json installed.

@calvaradocl, we're having the same issue after upgrading to 2.1.174. Did you manage to fix it?

[calvaradocl]

@easlavov, no, we still have the same issue even after multiple runs.

We had to copy our dependencies manually into a folder and use gulp, for now, to move them to the www-root folder.

[Smaiil]

Our Web solutions all fixed now locally and CI by updating to latest package. 👍

thank you!

[nimor109]

We have the same issues in production in our ci/cd pipelines. We also have to manually manage a cdnjs "feed" (shared folder) which is really bad. According to their thread this was done on purpose and won't be fixed. We'll see if our teams can use the NPM feed instead for their needs