Data-navigator: Enabling navigable chart elements with alt text

[frankelavsky]

This PR is demonstrating a prototype of in-progress functionality and is subject to change.

Description

This work is part of an active consultation on the accessibility of React Spectrum Charts. Under the hood we want to enable accessible interaction and navigation of every chart in RSC, regardless of whether canvas or svg are used. In addition, we want to provide an API that enables core RSC developers to have opinionated design control over the navigation and interaction experience of each visualization type.

We are targeting smart defaults that make it easy for downstream developers to get accessible, rich, interactive data visualizations out of the box. But we also want to enable overrides for complex or advanced (but valid) use cases.

In general, this is a PR that is intended to demonstrate a working prototype (at minimum) and at best actually contribute working code into RSC across the ecosystem.

Related Issue

#482

Motivation and Context

If a chart can be hovered, clicked: a user should be able to do this with just a keyboard. If a chart can't easily be summarized in just a couple of sentences: a screen reader user should be able to navigate the data and hear alt text for each element. If a chart is being used for important decisions or is in an exploratory context: a user should be able to navigate and explore the data in a meaningful, structured way without requiring the use of a mouse or eyesight.

Presently, these experiences are not possible.

How Has This Been Tested?

Screenshots (if appropriate):

Element-level navigation

Alt text at the element-level

Group-level navigation

The goal will be to use Vega's signals to actually show the focus indication, while my new Navigator component will handle actually making accessible navigation possible (even if the visualization is rendered using canvas).

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• [x] I have signed the Adobe Open Source CLA.
• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [ ] I have added tests to cover my changes.
• [ ] All new and existing tests passed.

Current status

• [x] = first pass, mostly working

Navigation (tested with screen reader and keyboard-only)

• [x] Bar
• [x] Dodge/stack
• [ ] Line
• [ ] Area
• [ ] Scatter
• [ ] Pie
• [ ] Combo

Semantics/labels

• [x] Bar
• [x] Dodge/stack
• [ ] Line
• [ ] Area
• [ ] Scatter
• [ ] Pie
• [ ] Combo

Interactivity (clicking/tooltips)

• [x] Bar (partially, technically the events are there and emitting)
• [x] Dodge/stack (partially, technically the events are there and emitting)
• [ ] Line
• [ ] Area
• [ ] Scatter
• [ ] Pie
• [ ] Combo

Focus Indication w/ signals

• [x] Bar
• [x] Dodge/stack
• [ ] Line
• [ ] Area
• [ ] Scatter
• [ ] Pie
• [ ] Combo

Bonus

• [ ] Exit element
• [ ] Help menu
• [ ] Key re-mapping?
• [ ] Dev-friendly API options?

[frankelavsky]

While this PR isn't ready to be merged, what I hope for is some feedback on the general approach before populating the prototype's approach across all other visualization types.

Right now the navigation capabilities are pretty cool, even with a never-before accomplished navigation type:

• Bar can be navigated left/right according to the left-right encoding (the "dimension" prop). In data navigator these encodings are all called "dimensions" via our dimensions structure generation api. This part isn't novel (pretty standard).
• However, the encoding for "metric" can also be turned into a navigable dimension. This means that for a visualization that might be more complex or have more going on, you can actually navigate from smallest -> largest value (or vice-versa) as well. This is the up-down navigation. Pretty much no other library does this right now (you typically have to resort and redraw visualizations to change nav order), save for tools like Olli but that only applies to scatterplots.
• For stacked/dodged you can also navigate across color using [ and ] keys. It may make more sense for these visualizations to use up/down and left/right for the categorical encodings and then [/] for metric-navigation.

The key bindings are set in the constants file, but obviously we can change these across charts and even allow end-users to set their own keybinds too (outside of the scope of this PR but worth thinking about!).

[marshallpete]

@frankelavsky sounds good. @c-lamoureux and I will try and provide feedback in the next couple of days. I'll try it out today and look through the approach.

[frankelavsky]

@marshallpete sounds good and thanks! Note that I recommend entering the chart via keyboard. It'll show a big group indicator and comma-periodgoes between high level groups. These are visually all the same at the high level (i use console logs to know where I'm at but a screen reader works too). But one is dimensions, one is metric, and (optionally) one is color. Enter goes down a level. If divisions exist (nested, for example) that is level 2. Otherwise, straight to the elements. Going back up is different depending on which "dimension" you want to traverse up.

I've got some design/ui ideas to help with wayfinding and keyboard commands we can think about too.

And any of this, from nav structure to keybinds to code quality can all be points of feedback. Nothing is sacred, it's a prototype after all.

[majornista]

• For stacked/dodged you can also navigate across color using [ and ] keys. It may make more sense for these visualizations to use up/down and left/right for the categorical encodings and then [/] for metric-navigation.

I think we agree here. I lean towards using left / right / up / down for navigation, Space/Enter for drilling in, and Escape for drilling out. The fewer unfamiliar interaction patterns, like [ and ] for metric-navigation, the better. I used to work at an educational publisher, building interactive learning materials and games for K-6, and a wise mentor would provide feedback like, "That's great, now make it make sense to a 2nd grader." We need to keep in mind that its rare for charts to be keyboard or screen reader accessible, so it will be more helpful to use familiar interaction patterns, rather than introducing something exciting and new that adds to the cognitive load of data navigating.

[marshallpete]

I like what you have here. I think the approach makes sense.

As this is intended to be a POC/Prototype, I'm not too worried about code structure and cleanliness.

The behavior lines up with what we had discussed and what I understood from our discussions.

The styling of things will probably be tweaked for a production version. We will have our designers provide feedback but once again, since this is a POC/prototype, I think it serves it's purpose.

[majornista]

@frankelavsky We have to account for the mobile/touch screen reader experience when navigating the chart. Javascript cannot handle swipe gestures to navigate from a mobile screen reader in the same way that it can handle keyboard or pointer events when the screen reader is not running, it's a privacy issue. The data navigator will need to include both the focused element and any item before or after it in the navigation order, so that as the user swipes to navigate, there is an element within the chart to navigate to.

Also, since javascript cannot interpret complicated gestures with a screen reader running, we should make sure that any ways to change the navigation mode are available to mobile screen reader users using simple double tap to activate gestures.

[frankelavsky]

The data navigator will need to include both the focused element and any item before or after it in the navigation order, so that as the user swipes to navigate, there is an element within the chart to navigate to.

@majornista - heck yes, absolutely agree. We used the 1-before, 1-after approach at Visa specifically for this reason (rendering these gives fallback support). I reckon we should default to the previous/next element that exists in the same dimension that we are assigning to the left/right directions.

I lean towards using left / right / up / down for navigation, Space/Enter for drilling in, and Escape for drilling out.

Also totally agree.

This means that we will limit dimensional navigation to 2 dimensions. So for cases like a color-encoded scatterplot we might run into some trouble! The left/right would move along x values, and up/down along y values.... but what do we do if there is also a legend that shows color categories?

In the Zong et al research project ("rich screen reader experiences", figure below), they had separate navigation spaces that were completely separated from each other: one was up/down/left/right and drill in/out (for x and y) and the other was via categories with left/right and drill in/out. Maybe we could let users choose which path at the start? Or maybe have a key command (like pressing X or something) that swaps between "modes?"

Also (in additional to the above stuff) I think that we need at least 2 things to make this prototype really usable:

1. a place for interaction instructions to live (like in a menu with each chart or via a link that can be discovered that points to a single adobe-federated website). I like the idea of a link because then you can have a single place for the instructions to live that won't be part of the charting library/maintenance/package size/etc and it will also reduce redundancy for users (rather than having a menu for every chart, like we did at Visa)
2. once we have 1, we should also consider on-demand interaction/navigation help (like pressing F1 or equivalent while on an element) to hear/see where/how to navigate (nobody really does this in other libraries right now but many users have requested this in past research studies I've ran). Maybe a modal, popup, or aria-live="polite" caption region under the chart?

[frankelavsky]

Just a reflection of the coding to-dos before this will be hand-off ready:

• [x] Limit navigation to 2 dimensions (up/down, left/right) plus drill in/out (@frankelavsky)
• [ ] Add 3 element strategy (as a fallback for mobile nav) (@frankelavsky)
• [x] Pre-gen or pre-determine ids for: mark, group (series is a stretch goal) and pass downstream (@frankelavsky)
• [x] Emit focus/blur from data navigator to RSC (@frankelavsky)
• [x] RSC activate signal via vega for focused element (or blurred)
• [x] Add new signal to spec, unique to focusing

We have a few to-dos that I didn't assign to myself, because someone else could definitely build them out in parallel. If anyone wants to do new work, I think that adding the signal for focusing to the spec would be the most helpful.

And feel free to mention if there is anything I've forgotten here!

[marshallpete]

Ok I think I added all the needed focus rings as well as the signals and calls to control them. It aint pretty but we are in POC mode.

[frankelavsky]

Ok I think I added all the needed focus rings as well as the signals and calls to control them. It aint pretty but we are in POC mode.

Heck yeah! Awesome. The focus indication looks great. It looks like I've got a little bug to fix now on where the invisible element is actually being rendered though. I'll tackle that tomorrow (unless you know what caused this, you can do it). I think one of the last steps will be to visually turn off focus indication for the navigator element too, so only yours shows.

[frankelavsky]

Okay, so my final major to-do has been to explore the 3-element strategy as a fallback for mobile. Unfortunately, I don't think we will be able to get the prototype working in time. We just need more testing. This is a pretty tricky thing to design for and I'm not satisfied with how the UX is right now.

I'll be able to work some more on this this week, but we might have to move this to a stretch goal, since my time on this project wraps up this Friday.

If anyone wants to give the project a look in the current status, feel free to do so (@majornista, @marshallpete, @c-lamoureux, etc). I'll have some time set aside Friday as my final day, so I'll be prioritizing any items that we think I can tackle that day (feedback, tweaks, more explanations added for things, etc).

[frankelavsky]

Heads up, I'll have another commit coming in - I've been wrangling the UX (and ARIA) for our mobile fallback and I think I may have settled on something that works. Fingers crossed that gets in today, but if not it might bubble over into Monday.

[frankelavsky]

While doing the last bit of work on this, I noticed that directional nav wasn't correct on the childmost elements of stack. Originally, I thought "oh, the left/right and up/down will change based on chart orientation/etc so we just need to set those props." But then I realized that I actually needed to add a new feature to data navigator itself because the division navigation was correct but it just didn't match the children. I was baffled at first, but then I realized that it was just because of the way data navigator groups dimensions by default: you always navigate within divisions but in a stack you want to nav across them.

Anyway, you can navigate the difference between two stacked bars (represented as their graphs) in DN's little demo test page. Note: enter button followed by enter and this is the division level. Going left and right here is across date. But going down into the dates (enter again), left and right now stay within that date, rather than continuing to change the date values. This isn't what we intended.

The diagrams in our v2.2.0 release notes communicate this new feature in simpler terms:

Default/previous behavior, or using new "within" dimension.behavior option:

                           dimension
                               |
            division <- - [left/right] - -> division
                |
    a <- - [left/right] - -> b

Using new "across" dimension.behavior option:

                           dimension
                               |
            division <- - [left/right] - -> division
               |                                |
               a <- - - - [left/right] - - - -> a

You'd want the first style in a regular bar chart, but the second style for a stacked bar chart (it just makes more sense intuitively). I'll have a commit up with these changes too. Look forward to this stuff on Monday and thanks for being patient!

[frankelavsky]

Okay folks, here is my last official set of commits, I reckon. Thanks for being patient!

Notes:

• We are now using the spec, and not props, to generate data navigator's structure (this works much better!). I think this will make it much easier to spread our pattern to future charts.
• Part of using the spec means that x/y is detected in the code as we generate, so we can more easily map that to up/down and left/right (check out bar navigation as vertical versus horizontal)
• We are using the new version of data navigator (mentioned in my previous comment above)
• I've hidden VegaChart using aria, since it is a bunch of meaningless elements right now
• The "enter navigation area" button now only appears visually when it is focused (so you have to nav to it)
• The focus signal now correctly disappears when you leave the chart
• And lastly (perhaps controversially) I have chosen to remove the mobile fallback elements

Again (on that last point), I just can't settle on a good UX for this. And I can get a halfway decent experience working in plain html and vanilla js, but this doesn't translate over to our react environment for some reason. I have a feeling that focus handling and other things in react ruins this for us. I'd need time to really dig into this. I'd probably sleep at night if I could somehow replicate my plain environment in ours, but I wouldn't be happy with a mobile user experiencing what we had currently. It was just buggy and strange.

I'm sure mobile use is nearly non-existent, but still: I'll be thinking about how to solve this. It's possible that a small "control" panel (of buttons) that had aria live announcements would solve this (like my messy sketch below).

[marshallpete]

Working storybook link: https://opensource.adobe.com/react-spectrum-charts/PR-483/?path=/story/rsc-bar-dodged-bar--color