browser-use icon indicating copy to clipboard operation
browser-use copied to clipboard

Add HarRecordingWatchdog for HTTPS HAR capture

Open kalil0321 opened this issue 2 months ago • 5 comments

#3063


Summary by cubic

Add HarRecordingWatchdog to capture HTTPS network traffic via CDP and write a HAR 1.2 file on browser shutdown. Auto-attaches when browser_profile.record_har_path is set and supports configurable content storage and capture modes.

  • New Features

    • Records HTTPS requests/responses (bodies and request postData) and saves to browser_profile.record_har_path on stop.
    • Supports record_har_content: embed (text or base64), attach (sidecar dir: <har_stem>_har_parts), or omit.
    • Supports record_har_mode: full or minimal (main doc + same-origin subresources).
    • Adds browser/version metadata, page refs, and DOMContentLoaded/load timings.
  • Migration

    • Set browser_profile.record_har_path to enable.
    • Optional: configure record_har_content (embed|attach|omit) and record_har_mode (full|minimal).

Written for commit 6dbdaab6b163824cc7365d567abd3ead097f03fe. Summary will update automatically on new commits.


[!NOTE] Introduce HarRecordingWatchdog to capture HTTPS traffic via CDP and write a HAR 1.2 file on shutdown, configurable for content storage and capture scope, and auto-attached to sessions.

  • Watchdogs:
    • HarRecordingWatchdog (new):
      • Captures HTTPS requests/responses via CDP (Network, Page) and writes HAR 1.2 on BrowserStopEvent.
      • Configurable: record_har_content (embed/attach/omit) with sidecar dir for attach; record_har_mode (full/minimal).
      • Records response bodies, headers, sizes, compression, timings; includes pages with titles and DOMContentLoaded/load metrics; embeds browser metadata.
  • Session Integration:
    • Import, initialize, and attach HarRecordingWatchdog in BrowserSession.attach_all_watchdogs() (auto-enabled when record_har_path is set).

Written by Cursor Bugbot for commit b53a593d334e048eabe758e9727106763089e4ec. Configure here.

kalil0321 avatar Nov 06 '25 12:11 kalil0321

TODO: missing req & resp body size + resp content; pages: title, strateddt, timings;

kalil0321 avatar Nov 06 '25 13:11 kalil0321

Hi @kalil0321 - what's the purpose of this?

mertunsall avatar Nov 26 '25 18:11 mertunsall

also can we disable it by default please?

mertunsall avatar Nov 26 '25 18:11 mertunsall

Hi @mertunsall, thanks a lot for merging the PRs :)! This PR records the HAR to capture network activity. I will link the related issue ASAP and disable it.

kalil0321 avatar Nov 27 '25 17:11 kalil0321

Hi @kalil0321 , of course sorry for being late.

I like this - let's just disable this by default.

mertunsall avatar Nov 27 '25 18:11 mertunsall