WebFetch: 404 Error When Accessing Crates.io URLs

[littlejak20]

Bug Description Bug Report: WebFetch tool returns 404 for accessible crates.io URLs

Description: The WebFetch tool consistently returns "Request failed with status code 404" when trying to access valid crates.io URLs that are accessible in web browsers.

Steps to reproduce:

1. Use WebFetch tool with URL: https://crates.io/crates/docx-rust
2. Tool returns 404 error
3. Manual verification shows the URL is valid and accessible

Expected behavior: WebFetch should successfully retrieve the crates.io page content

Actual behavior: Returns "Request failed with status code 404" error

Additional context:

• URL is confirmed accessible in web browser
• WebSearch tool finds references to the same URLs but also encounters JavaScript requirement issues with crates.io
• This affects ability to verify crate versions and metadata
• Similar issue may affect other crates.io URLs

Impact: Prevents analysis of Rust crate information, version checking, and dependency verification workflows.

Environment Info

• Platform: macos
• Terminal: vscode
• Version: 1.0.6
• Feedback ID: 399bf6c5-efb1-478f-9cc7-f8af02fac4ea

Errors

[{"error":"AxiosError: Request failed with status code 404\n    at qH (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:534:1070)\n    at IncomingMessage.wA (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:535:9969)\n    at IncomingMessage.emit (node:events:530:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1698:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:90:21)\n    at Bc.request (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:537:2081)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Jw2 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1856:853)\n    at async Cw2 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1856:1729)\n    at async Object.call (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1856:4818)\n    at async dA5 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1827:7739)\n    at async IV1 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1827:5586)\n    at async OX1 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1465:1323)\n    at async mA5 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1827:4961)\n    at async gA5 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1827:4818)","timestamp":"2025-05-30T09:01:50.991Z"},{"error":"AxiosError: Request failed with status code 404\n    at qH (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:534:1070)\n    at IncomingMessage.wA (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:535:9969)\n    at IncomingMessage.emit (node:events:530:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1698:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:90:21)\n    at Bc.request (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:537:2081)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Jw2 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1856:853)\n    at async Cw2 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1856:1729)\n    at async Object.call (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1856:4818)\n    at async dA5 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1827:7739)\n    at async IV1 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1827:5586)\n    at async OX1 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1465:1323)\n    at async mA5 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1827:4961)\n    at async gA5 (file:///usr/local/lib/node_modules/@anthropic-ai/claude-code/cli.js:1827:4818)","timestamp":"2025-05-30T09:02:44.195Z"}]