[Feature]: add xpath selector like stagehand

[carlvine500]

What problem does this feature solve?

refer : https://docs.stagehand.dev/examples/best_practices

code eg:

const [topAction] = await page.observe("Click the quickstart link");

/** The action will map 1:1 with a Playwright action:
{
	description: "The quickstart link",
	action: "click",
	selector: "/html/body/div[1]/div[1]/a",
	arguments: [],
}
**/

// NO LLM INFERENCE on observe results
await page.act(topAction)

What does the proposed API look like?

function aiInput(text: string, locate: string, options?: Object,selector: string)

[carlvine500]

我们自研的前端代码, 已经有约定的选择器选择元素(有几个关键的最短路径定位元素组件), 如果能暴露这个口子给我们, 会带来极大的便利并加快运行速度 .

[quanru]

Can you directly use playwright or puppeter page to operate a selector? @carlvine500

[carlvine500]

@quanru 感谢您的建议, 期望业务人员与技术人员能在不同的地方配合 .

//  业务人员写的操作步骤
const [topAction] = await page.observe("在电子邮箱字段输入 %email% "); 

// 因邮箱字段图标设计缺陷, 定位不准, 技术人员通过代码判断只要进入这个环节自动 添加selector让stagehand识别
/** The observe result will be an object with the following shape: 
{
	description: "The email input field",
	action: "type",
	selector: "/html/body/div[1]/div[1]/input",
	arguments: ["%email%"],
}
*/

await page.act({
	...topAction,
	// This prevents LLMs from seeing sensitive information
	// No LLM inference is taken on observe results
	arguments: [sensitiveEmail],
})

参考stagehand文档: https://docs.stagehand.dev/examples/best_practices

[quanru]

@carlvine500 你的目的是为了缓存吗？建议你先手动修改 cache yaml 文件里的 xpath

[carlvine500]

@quanru 是为了准确快速的定位元素, 比如同一页面出现多个"全部"按钮, 有纵向导航的, 有横向导航的, 有标签查询的等等, 如果用语言描述太麻烦了, 想直接写好一个选择器免得大量文字描述不清 .

[quanru]

@carlvine500 直接用现成的 Playwright 或者 Puppeter 里的 page 进行选择可以吗？如果定位到元素了，直接用传统的方式进行操作呢？

[quanru]

close by #844