Indexing of Docs only able to grab entrypoint page

[JarradJarrad]

1. If you can, please include a screenshot of your problem

When attempting to index DSPy docs via the chat window, I get this:

When attempting to index the same docs via cursor settings -> features -> docs section, I get this:

I have tried all of the below with the github repo the docs are built from, same thing happens.

2. Please include the name of your operating system

Version: 0.33.4 VSCode Version: 1.86.2 Commit: 2dd5b2855ac68f3a3d3babee87d22b399a9db7d0 Date: 2024-05-11T02:05:08.197Z Electron: 27.2.3 ElectronBuildId: undefined Chromium: 118.0.5993.159 Node.js: 18.17.1 V8: 11.8.172.18-electron.0 OS: Darwin arm64 23.1.0

3. If you can, steps to reproduce are super helpful

1. navigate to cursor settings -> features -> docs section

2. click 'add new doc'

3. enter the following (with or without /docs/ and with or without trailing slashes, it all results in the same outcome)

4. wait while Cursor indexes the docs

5. all it indexes is actually just the entrypoint page

[jamesryancooper]

I've had the same issue with numerous docs.

[jason-downs]

Yup, this is making the custom docs feature almost useless for me.

[davidalpha1]

Any update on this feature being able to navigate sitemaps better?

Tried to use https://www.xml-sitemaps.com/ , but cursor just indexes the entry page.

[fmontes]

I have the same problem with https://dev.dotcms.com/docs/table-of-contents, not even sending the sitemap directly like https://dev.dotcms.com/sitemap.xml

[kristojorg]

There are now many issues related to this, none of which have received information from the team :/.

• https://github.com/getcursor/cursor/issues/2750
• https://github.com/getcursor/cursor/issues/2749
• https://github.com/getcursor/cursor/issues/2488
• https://github.com/getcursor/cursor/issues/2034
• https://github.com/getcursor/cursor/issues/1994
• https://github.com/getcursor/cursor/issues/1744
• https://github.com/getcursor/cursor/issues/1726
• https://github.com/getcursor/cursor/issues/1479
• https://github.com/getcursor/cursor/issues/1163
• https://github.com/getcursor/cursor/issues/931
• https://forum.cursor.com/t/document-indexing-settings/682/3
• https://forum.cursor.com/t/documentation-indexing-problems/20860/15
• https://forum.cursor.com/t/doc-sets-only-seem-to-index-a-single-page/32605

[NatElkins]

This is affecting me as well, only the entrypoint page is indexed. Would love to see some progress on this!