Feature Request: Cursor-based Pagination

[danielkurniadi]

I wonder if we can support cursor-based pagination instead of offset based. I was hoping to get something like: Slack API Pagination

We are hoping to get a RESTful feature for this cursor-based because our micro services have been using cursor for fast lookup. It saves time not to scan though n records in database that we don't retrieve where n is the number of offset. I think this benefit greatly for speed. The database that use some sort of sorted table, merge tree, or B+tree can benefit greatly because cursor translate to a where clause index >= <YOUR VALUE> which can skip data partitions where condition is not met, pruning a lot of scanning. Where as N offset means the database have to count from 0 - offset before retrieving items, wasting time on scanning N items.

I propose that:

1. resource.ItemList has 1 more field called Cursor string.
2. Cursor is optional. If suppose the request includes ?cursor="<base64 or whatever string encoded>" then the storer can respect this condition and fetch the database for a given cursor.

NOTE:

• Cursor string can translate to multi fields in database, let say Base64Encoded(CreatedAt, UpdatedAt). This means filter out item that are CreatedAt < cursorCreatedAt and UpdatedAt < cursorUpdatedAt

[danielkurniadi]

@rs can help me on this? Edge beta release will be greatly appreciated. I might be naive but it should be a simple feature in my opinion. Otherwise, can you suggest me how to work around with cursor base pagination with rs/rest-layer?

[danielkurniadi]

For now I was thinking the cursor based pagination can be achieved with filter greater than or lesser than.

However this means the client is forced to be aware on this pagination logic. It can be okay-ish if the design from beginning is to allow client / consumer to moderate pagination by specifying the filtering field + limit. But we have been following the strategy for pagination in Slack API Pagination such that if the fields used for this cursor changed in the Backend logic, Frontend or client doesn't need to care with the API changes. The result, yes might be different but the interface is still same.

[smyrman]

Just to give a solution you can use today; you can use:

• filter={id:{$lt:<last ID in response>}}. // or $gt, based on ordering.
• sort=-id // or id
• limit=<limit>

This can be implemented in client logic without any rest-layer changes. You can of course also combine the id filter with your own filter, e.g. using an outer $and to allow other filters on id.

This require the id field to be set as Sortable and Filterable. IF you use XIDs, they would be chronologically ordered based on create time.

PS! Iterating/sorting on other fields this way is possible, but require that the field is sortable, filterable AND unique (for the given filter).