token usage tracking (Issue #268)

[andrewgcodes]

why

being able to track input/output/total tokens when using stagehand can be helpful for controlling costs, picking which model to use, etc.

See Issue #268 (https://github.com/browserbase/stagehand/issues/268)

what changed

• Added LLMUsageEntry interface to track tokens per function call
• Implemented usage recording in OpenAI and Anthropic clients
• Added recordUsage() and getUsage() methods to Stagehand class
• Added getTotalTokensUsed() for aggregating total token usage

Works for act(), observe(), and extract() 12 files edited, added tests and and example (hackernews.ts) demonstrating how to log tokens used. Please see diff

test plan

i wrote a few test cases and it seems to work but someone on the browserbase team should definitely test as well before merging!

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: 931020d16f79840b0c4e84760d0ca6a3e4f49a56

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[kamath]

whoa!! thanks @andrewgcodes! i'll take a more detailed look at this in a bit, but thank you so much for taking the time to do this!

[andrewgcodes]

Hey Anirudh, thanks for the quick review and kind words! I'll ping you on the Slack

[filip-michalsky]

this is amazing! looking for the discussion in Slack. Also I see some Devin in here @andrewgcodes :D

[cmaycumber]

Is there any reason this stalled out? Can I help at all?

[seanmcguire12]

token usage can be accessed via stagehand.metrics

[cyberpwnn]

token usage can be accessed via stagehand.metrics

No it cant, it doesnt work, its not documented, and it always shows 0. Please provide a way to access metrics.

[seanmcguire12]

@cyberpwnn either set useAPI: false or env: "LOCAL" in your stagehand constructor and you should see them if you run something like:

const metrics = stagehand.metrics;
console.log(JSON.stringify(metrics, null, 2));