bigtest icon indicating copy to clipboard operation
bigtest copied to clipboard
verified publisher icon thefrontside

Epic: Release Interactors Beta, Platform Alpha

Open cowboyd opened this issue 5 years ago • 9 comments

We want to release what we have at the beginning of December with a bang, and what that means is that we want everything that we do have to be "swipe to unlock". As it currently stands, we feel that the interactors as a standalone library fits that bill. You can take it and install it in Jest, Cypress, Mocha, and yes, in our own runner, and experience immediate, palpable benefit.

On the other hand, the runner, test syntax, and dev experience, etc... still feels a bit rough around the edges and some time is needed for it to settle. Part of this is that a library is simpler to drop in than something that has a CLI and lots of moving parts. The other is that we still have a lot that feels experimental in our runner.

As a result, we are going to push the interactors as the main feature of the release, and then bill everything else as Platform that not only contextualizes interactors with the important motivating factors about why we are doing all this work, but also provides an on ramp for early adopters that want to get started with writing the tests in a new way.

We initially thought about calling it "runner", but that doesn't really do the system justice. From here on out, when we talk about BigTest Platform, we're talking about the CLI, Orchestrator, driver, agent, etc...

Along side the code, we want to release an entire site dedicated to bigtest that will hang directly off of the main frontside.com website: https://frontside.com/bigtest that will contain everything that users will need to know about bigtest. This will including marketing material such as rationale, and value proposition as well as detailed technical guides for the various pieces which today include interactors and platform. 90% of our effort will be focused on the interactor story since we feel that this is what is most actionable and ready as of today. But we'll spend 10% of our effort on the large picture and a very minimal guide for starting the

@jorgelainfiesta, @cherewaty and I created draft sitemap for the release site that will contain all of the relevant pages/sections. So once this sitemap is in place, each page can be picked off and executed, or executed in pieces. The person spearheading each piece is marked next to that piece. That doesn't necessarily mean that that person will be doing all of the work on that section, but it does mean that they'll be the main point of contact for it.

  • [x] create https://frontside.com/bigtest placeholders for the following sitemap @jorgelainfiesta
  • [x] /bigtest/ @jorgelainfiesta
    • [x] value proposition: interactors
    • [x] call to action: interactors (link to interactors guide)
    • [x] value proposition: platform (not as emphasized as interactors, mostly recycle from current marketing site)
    • [x] call to action: platform (link to platform guide)
  • [x] /bigtest/about @cowboyd @jnicklas #710
  • [x] /bigtest/interactor/: @jenweber
  • [x] /bigtest/interactor/guide/: @jenweber
  • [x] /bigtest/interactor/api/: @jnicklas #659
  • [x] /bigtest/platform/ @jorgelainfiesta - Focus on the cross platform story, “ideas refridas” from bigtestjsi.io - CTA: getting started with bigtest platform - Conversion metrics: npm installs, users in discord
  • [x] /bigtest/platform/guide @minkimcello (Goal here is to get early adopters enthusiastic and on discord)
    • [x] Getting started shows the critical path
      • Installation
      • Writing a test with steps and assertions
    • [x] Architecture guides to get folks enthusiastic about the platform @cowboyd
      • Diagrams
      • What we believe this enables
  • [ ] version package libraries with 1.0 beta and 1.0 alpha tags. We need to figure out how to do this with our release system @minkimcello
  • [ ] update individual package README to point to the public website and if possible, where in the website that thing officially fits in @minkimcello
  • [x] Integrations
    • [x] separate integrations workspace for @bigtest/cypress #759
    • [x] ensure that bigtest jest integration works (show the problem, how to fix without special package) @minkimcello
  • [ ] API
    • [x] finish fluid interactor definition syntax #737 #735 #799 @minkimcello @jenweber
    • [x] ~~composable filters @jnicklas @cowboyd #773~~ WONTFIX
    • [x] ~~access filter values inside filters and actions #773~~ WONTFIX
    • [x] deprecate perform() API https://github.com/thefrontside/bigtest/discussions/770
    • [ ] custom matcher API #743 #824
    • [ ] Improved error handling #820 #682

cowboyd avatar Nov 19 '20 19:11 cowboyd

Updated layout:

interactors/

platform/ ⚠️wip

  • / ⚠️wip
    • installation
    • writing-your-first-test
    • running-tests

minkimcello avatar Nov 23 '20 18:11 minkimcello

Small Updates

  • We originally had a page set aside for explaining asynchrony but we decided to merge that together into the introduction page (or the landing page of interactors).
    • I changed overview to introduction for now as overview implies it's just a summary of the pages to come but we might be diving into the features/benefits of interactors in the overview page.
  • Since we're soft soft launching the platform as alpha, we deleted the interactors/getting-started/why-bigtest page and splitting it into the landing pages of bigtest and interactor.

Questions

Small

  • [ ] Should mentions of Interactors and BigTest always be capitalized? I've generally been lower casing them when they're an object of a sentence and capitalizing when it's the subject. 🤷‍♂️
  • [ ] Node12 is listed as one of the prerequisites for Interactors. Is that the only version of Node that's compatible?
  • [ ] Should we include the Page interactor in the page that lists all the pre-made ones offered by BigTest?
  • [ ] Should we rename the interactors section in the interactors guide to either Getting Started or Core Concepts? Having a Interactors section in the Interactors docs feels unnatural.

Medium

  • [ ] In our example we use the Button interactor. We would like to swap it out for TextField but I was debating whether to wait for the jsdom issue to be resolved or if it's safe to assume we'll have a solution before the launch.
  • [ ] Are .exists(), .has(), .is(), .absent(), and .find() technically actions? We introduce actions in the context of writing your own interactors but I'm wondering if we should introduce the five mentioned as the "default actions that are available on all interactors".
    • In the quick start examples we assert against exists() and absent() but don't really connect the dot between these methods and interactors. If we say they're actions, we could include a short sentence to mention that these are technically actions and are returning a boolean.

Large

  • [ ] Here, we have a common questions section and Jen wants to know if there are any other common questions we should include.
    • [ ] And regarding the second question, what should the user do if the problem is not accessibility?
  • [ ] We're thinking of providing a very simple sample app for the interactors doc. Thoughts?
    • It could potentially replace @bigtest/todomvc.
    • Using npx, users would just need to run npx @bigtest/sample.
    • The sample app would come with Jest, Cypress, and BigTest alpha already set up.
      • With custom commands in package.json, they could run yarn jest, yarn cypress. yarn bigtest:alpha to see the interactors in action.
    • Jen had a neat idea of how we could decorate the app with BigTest logos.

Preview of our current progress: Netlify.

minkimcello avatar Nov 24 '20 19:11 minkimcello

Node12 is listed as one of the prerequisites for Interactors. Is that the only version of Node that's compatible?

That weird since interactors work in the DOM. Maybe node 12 is required to build the interactor library?

Should we include the Page interactor in the page that lists all the pre-made ones offered by BigTest?

Yes. It's hard to imagine writing a test case without using the Page interactor to visit the url, and since that's what you'd also do to assert that you're on the proper URL, or what the page title is, I'd say it's critical to cover

Should we rename the interactors section in the interactors guide to either Getting Started or Core Concepts? Having a Interactors section in the Interactors docs feels unnatural.

Is this on the index page?

In our example we use the Button interactor. We would like to swap it out for TextField but I was debating whether to wait for the jsdom issue to be resolved or if it's safe to assume we'll have a solution before the launch.

I'd say it's safe to say that we'll get it working, or if there is a workaround that needs to happen, then we can include that in our documentation.

Are .exists(), .has(), .is(), .absent(), and .find() technically actions? No. For exists, has and is, they, like actions, return an promise, but they are more to read and assert on a particular state, whereas actions designed to alter the state of the UI.

action: click(), focus(), fillIn() assertion: has()/is()/exists()

find() is in its own category in that it returns a new interactor scoped within the current interactor, and is generally used for composing actions from interactor primitives:

createInteractor('DatePicker')({
  actions: {
    open: (picker) => picker.find(Button).click()
  }
});

cowboyd avatar Nov 24 '20 20:11 cowboyd

we have a common questions section and Jen wants to know if there are any other common questions we should include. And regarding the second question, what should the user do if the problem is not accessibility

I'd also add that if you are going to write your own interactors, never fear! You can still use the built in interactors in the manner in which I alluded to before, you would just delegate to them. Does that mean we should have a section on using interactors from interactors?

We're thinking of providing a very simple sample app for the interactors doc. Thoughts? Triple +++ ❤️ ❤️ that idea

cowboyd avatar Nov 24 '20 20:11 cowboyd

@minkimcello @jenweber Had a look over the initial preview, and as I mentioned before, it's coming along very nicely. I've separated my thoughts into structural, stylistic, and content

Structural

I don't know if this is something intrinsic to docosaurus, but it feels like the content is broken up into tiny pages-. This is my own personal preference, but when I'm reading docs, I prefer to have the navigation go to internal anchors within a single page rather than have 10 different pages. Maybe this will change as the individual sections get larger? /cc @jorgelainfiesta

Stylistic

I'm a fan of using long options in documentation, and short options in usage, so for example, I'd say npm install --save-dev bigtest rather than npm install -D bigtest

Content

statements like this on the builtins page:

You can construct your own customer interactors which we will cover in the Write Your Own Interactors section. But first let's have a closer look at what locators, filters, and actions are.

or this on the locators, filters and actions page:

Locators, filters, and actions are the key ingredients for creating new Interactors.

seem to imply that locators, filters and actions are related to creating your own interactors, when in fact, they are key to the builtin interactors as well. Maybe we can tweak the language to have a more universal tone

Locators

  • we always want to reinforce the message that interactors help your tests approach your app from the same perspective as your user, and locators are an example of this message in action. You user is thinking in terms of the "Submit" button, therefore, your interactor is Button("Submit"). The intro to the button locator goes straight to the explanation of how the locator is implement with the textContent attribute of the <button> element without stopping to explain why.

Also, it might be worth dropping in an actual <button>Submit</button> into the docs to reinforce visually the idea.

Filters

  • Can we work into the narrative that filters are the main way of making custom assertions? I feel like this is a key insight that makes working with filters easier if you're coming from a classic testing library. Whereas in jest or mocha we might say expect(value).toEqual(expected) in bigtest we would use a filter.
  • worth mentioning in the section that says a locator is optional that a locator is kinda like a default filter?

Writing your own Locators

On the four things to decide on how to build your custom interactor, I think the example of using the class name is not a good pattern. Is the css class name something that a user would identify? Not really. Usually, a checkbox would be identified by its corresponding label. That may not be available with an interactor.

Also, delegating to another interactor is definitely part of the story in creating custom interactors. Maybe our custom interactor should be a checkbox with a label that delegates to the internal checkbox for it's toggle action?

cowboyd avatar Nov 25 '20 15:11 cowboyd

@minkimcello @jenweber

Here is a dump of my notes after a read of the interactor guide. I was in a bit of a hurry, so apologies if it's a bit rough, but I figured getting it jotted down was the most important thing:

Builtin Interactors

assertions should generally use a filter + is()/has(), so while

Page({ url: 'localhost:3000' }).exists();

would work, the friendlier form would be something more like

Page({ url: 'localhost:3000' }).exists();

Also, I'd assert on the title since that seems more likely use-case

Page.has({ title: 'BigTest Example Application: ' })

Every interactor has some things in common, whether it is built in or you wrote it yourself. In the next section, we will have a closer look at what locators, filters, and actions are.

Can we get a preview here? "some things in common" might be better as more specific. What are those things? Actions, which advance the state of your app, and filters which match on it and help you do things like make assertions

locators, filters, and actions page

All interactors have some things in common, whether they are built-in or written by you. They each can have locators, filters, and actions. In this section, you will learn what these are and some more details about how to use them.

Again, I think that stating the purpose before we introduce the term is helpfuul. How about something like "All interactors have some things in common, whether they are built-in or written by you. They have to be able to find elements in the page, manipulate them like a user would, and ultimately make assertions based on how they appear. To do these things, interactors use Locators, actions, and filters"

One benefit of Interactors is that they help you align your tests with how a user actually interacts with the app, starting with finding what to click on.

How about "finding what to act upon?" since not every interactor has a click actions

A typical user would try to find a button with the word "Submit" on it

I'd rephrase this a bit because finding buttons isn't what users do so much as identify them and divine their purpose and perhaps use an example. Maybe "A typical user identifies a button by the words printed across it, so for example, they would think of a button with the word 'Submit' on it as the "Submit" button. Interactors use

This code snippet has a syntax error:

Button({
id: 'submit-button-1',
title: 'Sign Up Form',
visible
}).exists();

it should be `visible: true``

One limitation is that mutable APIs such as NodeList cannot be used in a filter.

We don't fully understand this issue, but we should unpack it if we're going to reference it in a guide.

Asserting with filters

In this section, I think it would be good to really play up the importance of filters in assertions with explanatory text. Something like "filters can be very convenient for finding matching UI elements, but where they really shine is in making assertions about what you expect your application to be showing.

It also might help to include that this is the equivalent of 'expect' in Jest, and should in Cypress, and that whenever you use those constructs, you would instead use an interactor's filter.

Finally, the rule for when to use is/has needs to be fleshed out, we could even present it as a light-bulb tip or something, but the general rule is that if the filter is an adjective, then you should use is(), if it is a noun, then it would be has(). Thus, Layout().has({ height: 200 }), but Layout().has({ direction: 'right-to-left'})

Writing interactors

The example, which uses an id for the locator goes against the advice we gave for interactor in the previous section. The id is for computers. I know we want to make it different, but maybe we can use aria-label for the locator instead? That's something a user with assistive technology would use to identify the UI element.

In this example we've configured the selector as 'button, input[type=button]' which will target both

What does "target" mean? It could mean a couple of things, so maybe it's worth expanding on this to say that the selector chooses a flat list top level elements that will be considered. Filters and locators are used to narrow this list.

Now let's import the new interactor and add it to a test:

The code snippet only has a rendition in Platform, not Cypress or Jest

Below is a more complex demonstration of what you can do with interactors:

Rather than put the implementation first here with the table cell interactor, talking about how you can use filters to make very complex, yet very readable assertions is one of the great strengths of interactors. Leading with an example of how awesome it is to use the power filter is going to sell more than the somewhat large implementation, which without seeing the benefit first, is hard to evaluate in context.

Something like this:

TableCell({ columnTitle: 'politics', row: 3 }).has({ value: '$600' });

General

It looks like there is some flash / jank on the navigation? where they resize momentarily?

cowboyd avatar Dec 10 '20 23:12 cowboyd

General feedback:

  • The indentation style of the examples is inconsistent, sometimes the ending parenthesis is on the same line, sometimes it is on the next line, some examples which are similar have different indentation styles, which makes them look more different than they are.
  • Some examples use placeholder interactors (e.g. MyInteractor('Some locator text', { id: 'my-id' }).exists();), while most of them use the builtin interactors. We should stick to the built-in interactors IMO.

Specific feedback:

Locators are the simplest way to find a specific element in a user interface.

Whenever you use an Interactor in a test, you can pass it a string, This string argument is used by the Locator.

I find this somewhat confusing. While "This string argument is used by the Locator" is technically correct, the follow up is then: what is a locator, and that is not really explained. Even though this is somewhat confusing, I would almost advocate to refer to the actual value as the locator, and so the function which is passed to the specification is a locator function and it evaluates to the locator. I think this would be easier to understand.

One limitation is that mutable APIs such as NodeList cannot be used in a filter.

Is this really true?

You can add as many filters as you need to your own Interactors, as covered in the next article.

This is a bit strange, maybe something like "Later you will learn how to add your own filters to interactors", or something like that?

Button('Submit').has({ id: 'submit-button-1' });

This example doesn't seem very realistic, can we use a better example, like asserting on the value of a text field, for example?

Button({ id: 'submit-button-1' }).is({ visible });

This should probably be:

Button({ id: 'submit-button-1' }).is({ visible: true });

find

The section on find is in the wrong place, it should belong to "Writing interactors". In its place it would be good to have a section on using find to scope interactors, this is challenging since we don't currently have any built-in interactors which lend themselves to scoping, but there is one interactor we could easily add which would work well for scoping: Fieldset!

Nearly every app has at least one user interaction that is strange

"strange" seems like a very charged word, can we choose something more neutral like "unusual" or "non-standard" or something?

Writing your first interactor

Can we start with something even simpler? Something like Sidebar or something would be nice?

Writing your second interactor

There is a very significant jump in complexity in this example. This seems like the type of the interactor we should bundle, but it doesn't seem great as a teaching aid.

jnicklas avatar Dec 11 '20 15:12 jnicklas

It looks like there is some flash / jank on the navigation? where they resize momentarily?

@jorgelainfiesta have you noticed this problem?

minkimcello avatar Dec 14 '20 16:12 minkimcello

@jnicklas @cowboyd at the end of the quick start, we're thinking of adding some real life examples like this:

"see how simple testing can be with interactors:"

import { Button, Page, test } from 'bigtest';
import { Modal } from './MyModalInteractor';

export default test('BigTest')
  .step(
    Page.visit('/'),
    Button('Sign In').click())
  .assertion(
    Modal({ id: 'sign-in-modal' }).exists();
    Button('Log In').exists());

👆 it's just a rough example but the purpose here is to show an example of UI interaction that is usually a pain in the butt to test. due to my lack of experience in writing tests, i'm having a hard time trying to think of an example that people would want to see. is there a particular interaction that we should showcase here?

minkimcello avatar Dec 15 '20 17:12 minkimcello