Proposal for slimmed down data model

[joneit]

Background

This document essentially describes the minimum standard interface for a Hypergrid data source. The motivation here is to be able to easily switch out the standard data source for a custom data source.

Along the way, we also discuss a formal interface object, handed to the data source on instantiation, that describes the expected interface.

The want: A "black-box" data source

While our data source is complex, with a base class that supports a pipeline of data transformers to reindex the rows, sometimes building synthetic rows, etc., Hypergrid doesn't want to know about the complexity. We want to be able to tell application developers to "bring your own" data source. To make this work, the data source interface needs to be "black box" with a simple interface. Defining that interface is the purpose of this document. Hypergrid's only interest in any advanced data source capability should be limited to signaling the data source with the relevant UI events.

The problem: Handling unimplemented methods

The current data source base class is hard-coded with an implementation for all known methods — so that if the datasource doesn't handle the method itself, the base class from which it descends will catch it (and fail silently).

One problem was that whenever we added a new method to a datasource, we also updated the base class, which had grown to include a large list of methods that various data source modules implemented.

This quickly proved unmanageable for several reasons. A better approach would be to eliminate entirely this requirement that the base class support every possible method that any descendant class would ever need.

Hypergrid 2.0.2 used data controllers to communicate with the data source without making non-standard method calls. This works ok but was an arcane and non-standard implementation of publish/subscribe.

Standard publish/subscribe is another strategy that allows unrecognized calls on a data source to fail silently without throwing an error (and without having to put such calls inside a try ... catch block); unhandled pub/sub messages are simply lost. However, the base class handlers described above are a bit more complicated, bubbling the call through the data source pipeline; so the pub/sub messages would likewise need to bubble. Compared to the custom publish method that would be required, I realized the original design was actually simpler, possibly more performant, and certainly more elegant as its just a normal method call.

The solution: A formally defined interface

The solution is to remove the handlers from the base class entirely. Rather than constantly updating the base class source code, we should instead install handlers as needed. In fact, I realized this could be done in a formal programmatic way.

The new approach is to define an interface which Hypergrid hands to the datasource on instantiation. The base class looks for options.interface and calls a function that sets up all the handlers.

The standard data source interface (as proposed)

The term interface as used here refers only to:

1. The names of methods the data source is expected to implement.
2. Optional fallback functions when the methods are unimplemented by the data source.

The interface handles requirements through fallback implementations:

• Required standard methods There are only three data source methods routinely called by Hypergrid that all data sources are required to implement. Fallbacks for these throw an error.

• Optional standard methods Also routinely called by Hypergrid, but Hypergrid provides generic fallbacks so implementation is optional.

• Discretionary methods Called by Hypergrid only if the application is using certain optional features or making certain explicit API calls (which Hypergrid on its own never calls). An application's data source only needs to implement these methods if the features they support are enabled. Fallbacks for these (may or may not) issue a console warning that the method implementation is missing from the data source.

• Custom methods May be called by the application layer and by plug-ins; they are not called by the Hypergrid core engine. Applications and plug-ins should provide a list of these. Fallback functions that issue console warnings are automatically generated from the list.

NOTE: All of the following interface methods (except for the custom methods) are standard and are defined by Hypergrid and passed to the data source constructor. Application developers do not need to define these; they only need to list custom methods. (That said, any fallback can be overridden with a call to dataSource.add().)

Required standard methods (fallbacks fail with error)

Implementations are required for these methods which support core features:

• getSchema()
• getValue(x, y)
• getRowCount()

If any of the above methods are unimplemented in the data source, an error will be thrown when the first one is called:

HypergridError: Expected data source to implement method `<method-name>()`.

Note that the something in setData(something) is implementation-dependent.

• For a local data source this is an array of data row objects, as used by the JSON behavior module expects.
• For a remote data source this might be a database query, as used by a hypothetical "remote" behavior module.

Optional standard methods (fallbacks provided by Hypergrid)

Implementations are optional for the following methods. Although these methods support core features, Hypergrid supplies defaults in the form of fallback functions.

• getRow(y) Calling getRow is generally discouraged. The supplied fallback uses a data row-like object with getters for each column value that defer to getValue(). A native implementation is preferred if it would be more efficient. There are only a handful of calls to getRow() remaining in Hypergrid. This fallback works well for these cases — which remain under review.
• getData() Calling getData() is generally discouraged. It only survives because it is super-practical for saving data with JSON.stringify(dataSource.getData()).
• getColumnCount() While Hypergrid does make calls to getColumnCount(), when the data source does not implement it, Hypergrid supplies an efficient fallback (getSchema().length).
• getDataIndex(y) Used by Hypergrid to tag selected rows before transforming data source so they can be reselected afterwards. Because rows in the transformed data source will have different indexes, this method gives the underlying index. Hypergrid supplies a generic fallback that simply returns it's input (y), which assumes the data source does not support transformations (see apply below).
• setRowMetadata(y, metadata) Only called by Hypergrid if your app tries to set row and cell properties. If unimplemented, a warning is issued to the console on the first attempt to set such a property.
• getRowMetadata(y, metadata) If unimplemented, calls to this method made during render fail silently.

Discretionary methods

Implementations for these methods support optional features and are only needed if such features are enabled. In any case, the interface should be specified.

There are two types of discretionary methods:

• With warnings: Methods called only when feature is enabled
• Without warnings: Methods called always

Discretionary methods with warnings

If unimplemented the following method fallbacks fail silently after a console warning message is issued that the method is missing (on its first invocation only):

• setData(something)
• setValue(x, y, value) Only called by Hypergrid if your app allows cell editing.
• setSchema(columnSchema[]) Only called by Hypergrid if your app passes an explicit schema on instantiation (or to subsequent calls to grid.setData()). (If a schema is not set explicity, getSchema() will return an inferred schema.)

Following are added by the row-edit plug-in:

• setRow(y, dataRow)
• addRow(dataRow, y) Caveat: Calling this method alters the row indexes of all rows that come after.
• delRow(y, rowCount) Caveat: Calling this method alters the row indexes of all rows that come after.

Discretionary methods without warnings

The following methods have no fallbacks which means they simply fail silently without issuing a warning message:

• apply() Used by Hypergrid to transform data, possibly changing the grid dimensions.
• isDrillDown Called by row drill-down UI events, and editing rows (when the relevant plugins are installed).
• isDrillDownCol

NOTE: Currently the above methods are routinely called by Hypergrid which is why they are not issuing warnings. However, this is under review. They should not be called routinely; they should only be called in response to sorting, filtering, and row drill-down UI events when the relevant plugins are installed (and the features are enabled, if they have an enable property).

Custom methods (fallbacks issue a one-time console warning)

Applications developers can specify additional methods with the interfaceExtensions option or by calling dataModel.add(interfaceExtensions). Plug-in developers can call add after instantiation.

Typical extensions existing plug-ins (in the old-addons repo) may need to make include:

• click
• getGrandTotals
• revealRow
• isLeafNode
• viewMakesSense

Summary

Hypergrid's data source controllers are being retired. This is a breaking change. There is no deprecation warning; we are not keeping the code around. This is why this is a major version upgrade.

Hypergrid no longer has any sense of the internal workings of the data source. In particular, it no longer deals with any kind of a data source pipeline; that is totally up to the internal implementation of the data source. As such all methods that deal with the pipeline are similarly being retired without warning.

Also being retired are the data source and data model methods such as getFields & setFields, getHeaders & setHeaders, and getCalculators. These are all handled in the schema (dataModel.schema).

Hypergrid now gives data sources a list of methods that intends to call in an instantiation option. Data sources should respect this list and call the provided callbacks to handle unimplemented methods gracefully.