dify Firecrawl API updated to v1

Self Checks

[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
[X] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

The Firecrawl API has updated to v1. However, the tool in Dify is still using v0.

2. Additional context or comments

No response

3. Can you help us with this feature?

[ ] I am interested in contributing to this feature.

Sep 12 '24 13:09 tonycky

Hello @chankwongyin! I'm here to assist you with any bugs, questions, or becoming a contributor. I'm a bot named Dosu, and I'm here to help while you await a human maintainer.

I found a similar discussion regarding the Firecrawl API versioning issue. The tool in Dify is still using Firecrawl API v0 because all the endpoints in the provided code are pointing to version 0 of the Firecrawl API. For example:

def scrape_url(self, url: str, **kwargs):
    endpoint = f"{self.base_url}/v0/scrape"
    # ...

def search(self, query: str, **kwargs):
    endpoint = f"{self.base_url}/v0/search"
    # ...

def crawl_url(self, url: str, wait: bool = True, poll_interval: int = 5, idempotency_key: str | None = None, **kwargs):
    endpoint = f"{self.base_url}/v0/crawl"
    # ...

def check_crawl_status(self, job_id: str):
    endpoint = f"{self.base_url}/v0/crawl/status/{job_id}"
    # ...

def cancel_crawl_job(self, job_id: str):
    endpoint = f"{self.base_url}/v0/crawl/cancel/{job_id}"
    # ...

All these methods are using the /v0/ path in their endpoints, indicating that they are still using version 0 of the Firecrawl API ^[1].

Additionally, there are existing tests for the Firecrawl API integration in the Dify codebase that are using version 1 of the Firecrawl API, as indicated by the base URL "https://api.firecrawl.dev" and the parameters used in the test ^[2].

To update the tool to use Firecrawl API v1, you would need to change the API version in these endpoints from "v0" to "v1".

To continue talking to Dosu, mention @dosu.

Sep 12 '24 13:09 dosubot[bot]

Man, I wish I had searched issues instead of trying to fix my workflow...

Glad I'm not the only one running into issues with this.

Sep 13 '24 04:09 ther3zz

Man, I wish I had searched issues instead of trying to fix my workflow...

Glad I'm not the only one running into issues with this.

@ther3zz what problems you have? Right now the Firecrawl plugin is compatible with v0 API. If you are using it in Dify, you probably should use v0 API reference instead of v1 API reference.

Sep 13 '24 04:09 tonycky

Hello @chankwongyin! I'm here to assist you with any bugs, questions, or becoming a contributor. I'm a bot named Dosu, and I'm here to help while you await a human maintainer.

I found a similar discussion regarding the Firecrawl API versioning issue. The tool in Dify is still using Firecrawl API v0 because all the endpoints in the provided code are pointing to version 0 of the Firecrawl API. For example:
def scrape_url(self, url: str, **kwargs):
    endpoint = f"{self.base_url}/v0/scrape"
    # ...
def search(self, query: str, **kwargs):
    endpoint = f"{self.base_url}/v0/search"
    # ...
def crawl_url(self, url: str, wait: bool = True, poll_interval: int = 5, idempotency_key: str | None = None, **kwargs):
    endpoint = f"{self.base_url}/v0/crawl"
    # ...
def check_crawl_status(self, job_id: str):
    endpoint = f"{self.base_url}/v0/crawl/status/{job_id}"
    # ...
def cancel_crawl_job(self, job_id: str):
    endpoint = f"{self.base_url}/v0/crawl/cancel/{job_id}"
    # ...
All these methods are using the /v0/ path in their endpoints, indicating that they are still using version 0 of the Firecrawl API ^[1].

Additionally, there are existing tests for the Firecrawl API integration in the Dify codebase that are using version 1 of the Firecrawl API, as indicated by the base URL "https://api.firecrawl.dev" and the parameters used in the test ^[2].

To update the tool to use Firecrawl API v1, you would need to change the API version in these endpoints from "v0" to "v1".

To continue talking to Dosu, mention @dosu.

@dosu There are changes on parameters in which some are removed and some are moved to the root. So it's not working if simply making update from v0 to v1 at the url.

Sep 13 '24 04:09 tonycky

Man, I wish I had searched issues instead of trying to fix my workflow... Glad I'm not the only one running into issues with this.

@ther3zz what problems you have? Right now the Firecrawl plugin is compatible with v0 API. If you are using it in Dify, you probably should use v0 API reference instead of v1 API reference.

For me, when using firecrawl to insert knowledge in the knowledge module, it works fine. If I attempt to use the scrape option in chatflow, it only returns values in the json array and not the text field... I cant use the json array as it breaks the chatflow node...

Sep 13 '24 11:09 ther3zz

Man, I wish I had searched issues instead of trying to fix my workflow... Glad I'm not the only one running into issues with this.

@ther3zz what problems you have? Right now the Firecrawl plugin is compatible with v0 API. If you are using it in Dify, you probably should use v0 API reference instead of v1 API reference.

For me, when using firecrawl to insert knowledge in the knowledge module, it works fine. If I attempt to use the scrape option in chatflow, it only returns values in the json array and not the text field... I cant use the json array as it breaks the chatflow node...

currently, the scrape tool response both text and json array:

Sep 14 '24 07:09 hjlarry

Man, I wish I had searched issues instead of trying to fix my workflow... Glad I'm not the only one running into issues with this.

@ther3zz what problems you have? Right now the Firecrawl plugin is compatible with v0 API. If you are using it in Dify, you probably should use v0 API reference instead of v1 API reference.

For me, when using firecrawl to insert knowledge in the knowledge module, it works fine. If I attempt to use the scrape option in chatflow, it only returns values in the json array and not the text field... I cant use the json array as it breaks the chatflow node...

currently, the scrape tool response both text and json array:

Once I updated to the latest version, the issue was resolved

Sep 14 '24 11:09 ther3zz