JSON object unnecessarily large for paged, server-side datatables; sends seq_len(DT)

[nograpes]

This is a feature request to prevent dt_rows_all being set to a vector that has one integer for each row in the entire table; instead it should be trimmed to match dt_rows_current.

An unnecessarily large JSON object is sent over the network for very large, paged, server-side datatables. DT_rows_all is set to seq_len(n) (or some reordering), which is a very large vector, even though data is only sent for the rows in DT_rows_current. Indeed, for my application, which uses a 70K row table, the majority of the data sent over the network is DT_rows_all.

I would write a reproducible example, but it is clear that the authors are aware of this issue, it is marked as TODO in the code itself:

https://github.com/rstudio/DT/blob/116377810ec0acbbd8c693632ee5f03ac5a79371/R/shiny.R#L493-L502

I think, ideally, DT_rows_all would be set to iCurrent, which would make it redundant with dt_rows_current, but it would have less chance of breaking existing user code. It would also have to change here:

https://github.com/rstudio/DT/blob/a314d79c012e53adda2bf9119b34c4dc4da2379f/R/shiny.R#L399

Alternatively, the filterFun function in sessionDataURL function could be modified to trim DT_rows_current before encoding it as JSON.

Without changing the package at all, I could write a custom filter for large data tables, which would trim dt_rows_all. I'm not quite sure how to write a custom filter, and I couldn't find any examples. I saw the lack of examples referenced in issue https://github.com/rstudio/DT/issues/194 as well, perhaps this would be a good example to provide? Any quick tips would be very helpful as well.

I would be happy to provide a PR, but I wanted to get a sense of what strategy you think would work best for this (making DT_rows_all the same as DT_rows_current, or trimming it in filterFun).

[nuno-agostinho]

Hello, could you please look into this issue? I am getting really large JSON objects simply because the tables have many rows. Thanks!

[nuno-agostinho]

As a workaround, I am now using the following code to remove the information from DT_rows_all (as I don't need this functionality for my app):

# Avoid large JSON response from DT
dt_mod <- getFromNamespace("dataTablesFilter", "DT")
dt_rows_all_line <- grep("DT_rows_all = iAll", body(dt_mod))

if (length(dt_rows_all_line) == 1) {
    mod <- gsub("DT_rows_all = iAll", "DT_rows_all = iCurrent", fixed=TRUE,
                body(dt_mod)[dt_rows_all_line])
    body(dt_mod)[[dt_rows_all_line]] <- parse(text=mod)[[1]]
    assignInNamespace("dataTablesFilter", dt_mod, "DT")
}

In context: https://github.com/nuno-agostinho/cTRAP/blob/master/R/shinyInterface_session.R#L456-L465