crawl4ai icon indicating copy to clipboard operation
crawl4ai copied to clipboard

Issue running with Colab

Open saikrn112 opened this issue 1 year ago • 0 comments

when I run the colab I get the following error

[INIT].... → Crawl4AI 0.4.247
[ERROR]... × https://crawl4ai.com/... | Error: 
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ × Unexpected error in _crawl_web at line 833 in execute_hook (../usr/local/lib/python3.10/dist-                       │
│ packages/crawl4ai/async_crawler_strategy.py):                                                                         │
│   Error: custom_hook_workflow.<locals>.<lambda>() got an unexpected keyword argument 'url'                            │
│                                                                                                                       │
│   Code context:                                                                                                       │
│   828           hook = self.hooks.get(hook_type)                                                                      │
│   829           if hook:                                                                                              │
│   830               if asyncio.iscoroutinefunction(hook):                                                             │
│   831                   return await hook(*args, **kwargs)                                                            │
│   832               else:                                                                                             │
│   833 →                 return hook(*args, **kwargs)                                                                  │
│   834           return args[0] if args else None                                                                      │
│   835                                                                                                                 │
│   836       def update_user_agent(self, user_agent: str):                                                             │
│   837           """                                                                                                   │
│   838           Update the user agent for the browser.                                                                │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

when this piece of code is executed. let me know if I missed some instruction or setting

async def custom_hook_workflow(verbose=True):
    async with AsyncWebCrawler(config=BrowserConfig( verbose=verbose)) as crawler:
        # Set a 'before_goto' hook to run custom code just before navigation
        crawler.crawler_strategy.set_hook("before_goto", lambda page, context: print("[Hook] Preparing to navigate..."))

        # Perform the crawl operation
        result = await crawler.arun(
            url="https://crawl4ai.com",
            config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
        )
        print(result.markdown_v2.raw_markdown[:500].replace("\n", " -- "))

saikrn112 avatar Jan 06 '25 10:01 saikrn112