Christian Meyer

**Problem Description** So Firecrawl respects the robots.txt but not meta tag - noindex. At least in my tests. `` **Proposed Feature** Check also meta tag with noindex besides robots.txt and...

[Feat] Scrape markdown response ideally is splitted in seperate chunks / parts

6

**Problem Description** Right now the scrape endpoint returns a huge string with the complete markdown of one URL. Storing the whole page in a vector store as one embedding is...