Improve selector inference on complex SPAs with flexible data models

[twschiller]

Context

• Our selector inference works well for in-house tools
• It works very poorly for popular SaaS apps that were designed to have customizable data models (Salesforce, Zendesk)

Discussion

Situations where our automatic selector inference performs poorly:

• SPAs with multiple in-app tabs where the app doesn't unmount tabs. (So there are non visible elements with the same selector)
• Apps with custom data entry grids/property table for custom objects. Including mixed input types, e.g., drop downs, etc. These generally don't put field names/ids on the fields
• The same information repeated in multiple places on a screen. Our inference needs to be smart enough to find a robust anchor for that fieldset/section of the screen

Ideas

• Provide a way for us (the PixieBrix team) to provide selector hints to inference for certain applications. These would be provided by the server so we can update the list without updating the extension
  • Easier: Pre-construct good anchors. Similar to how our inference automatically currently looks for good anchors like ids to anchor sub-selectors too, we could provide custom anchors for certain applications. To start, this could just be a list constant anchors to try. For example, trying .consoleRelatedRecord as an anchor on Salesforce. (See Example Human-Written Selectors below). The gotcha here is ensuring that when the anchor is used, it's still targeting the element the user had selected, and not a different one on the page
  • Harder: Suggest inference patterns/stencils potentially with placeholders for our inference algorithm to fill in, e.g., .test-id__field-label-container:contains("{{TEXT}}") + div. These should probably be encoded as JSON tree structure so we don't have to parse the stencil

Notable Apps

• Salesforce
• Zendesk

Example Human-Written Selectors

Salesforce Service Console

 .oneWorkspace[aria-expanded="true"] .consoleRelatedRecord .test-id__field-label-container:contains("Case Number") + div

Explanation

• .oneWorkspace[aria-expanded="true"]: select active in-app tab
• .consoleRelatedRecord: the field set selector, because the field can appear multiple times in different fieldsets
• .test-id__field-label-container:contains("Case Number"): find the field
• + div: the field value. For certain field types, additional selectors may be required to avoid picking up junk, e.g., the helper text from a dropdown on the field

[twschiller]

@jcforest I can walk you through this today or Monday during a sync