Add support for rolling functions

[zsz00]

Hope support Roll a function over data, run a statistic along a [weighted] data window

ref: https://github.com/JeffreySarnoff/RollingFunctions.jl

[JeffreySarnoff]

as with https://github.com/JuliaData/DataFrames.jl/issues/3163 What do you need that is missing from RollingFunctions?

[juliohm]

Thank you for opening the issue @zsz00. It would be nice to see what we can do to make these rolling functionality more integrated with Tables.jl tables. I am assuming that you have time series columns in mind?

@JeffreySarnoff feel free to explore our codebase here to see if it is worth adding a couple of transforms over columns. We have quite sophisticated pipelines already. I will try to create a tutorial for this current release.

[zsz00]

yes, I have time series columns need roll process.

1. I hope more integration with TableTransforms.jl or dataframes.jl. or more using doc.
2. need rolling(function, data, windowsize, stride)
3. RollingFunctions.jl newest realse in 2019.3, I mistakenly thought it was not actively developed.

[zsz00]

I think the combination of features is better to let this package do it than to let the user do it himself.

[JeffreySarnoff]

RollingFunctions.jl has been stable for that long. There are some innovations that I am considering -- none would be necessary for most uses, though some may be welcome. The upgrade, if any, is not next month.

On Mon, Sep 19, 2022 at 9:30 AM zy @.***> wrote:

I think the combination of features combination is better to let this package do it than to let the user do it himself.

— Reply to this email directly, view it on GitHub https://github.com/JuliaML/TableTransforms.jl/issues/121#issuecomment-1251024664, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM2VRUIBVLIK7YQGJWT7KDV7BTHDANCNFSM6AAAAAAQQB7OCE . You are receiving this because you were mentioned.Message ID: @.***>

[juliohm]

@JeffreySarnoff in case you want to integrate with the TableTransforms.jl API, all you need to do is implement the function applyfeat, and if the transform isrevertible, you can also implement revertfeat. So you can do the following in your RollingFunctions.jl package:

import TableTransforms: applyfeat, revertfeat

# optionally define some pre-processing step
import TableTransforms: preprocessing

function preprocess(transform::MyRollingTransform, table)
  # do some preprocessing on the table
  "preprocessed object"
end

function applyfeat(transform::MyRollingTransform, table, prep)
  # do something with cols of table
  cols = Tables.columns(table)

  # materialize the result into the original table type
  result = map(roll, cols)

  # cache in case we need to revert transform
  cache = nothing

  # return result and cache
  result, cache
end

If your rolling functions operate on each column individually, then you can do something even simpler. Take a look at our Colwise transforms, for example: https://github.com/JuliaML/TableTransforms.jl/blob/master/src/transforms/center.jl

This is all you need to get these column-based transforms running in parallel with Transducers.jl

[JeffreySarnoff]

thank you

On Mon, Sep 19, 2022 at 11:03 AM Júlio Hoffimann @.***> wrote:

@JeffreySarnoff https://github.com/JeffreySarnoff in you case you want to integrate with the TableTransforms.jl API, all you need to do is implement the function applyfeat, and if the transform isrevertible, you can also implement revertfeat. So you can do the following in your RollingFunctions.jl package:

import TableTransforms: applyfeat, revertfeat

optionally define some pre-processing stepimport TableTransforms: preprocessing

function preprocess(transform::MyRollingTransform, table)

do some preprocessing on the table

"preprocessed object"end function applyfeat(transform::MyRollingTransform, table, prep)

do something with cols of table

cols = Tables.columns(table)

materialize the result into the original table type

result = map(roll, cols)

cache in case we need to revert transform

cache = nothing

return result and cache

result, cacheend

If you rolling functions operate on each column individually, then you can do something even simpler. Take a look at our Colwise transforms, for example: https://github.com/JuliaML/TableTransforms.jl/blob/master/src/transforms/center.jl

This is all you need to get these column-based transforms run in parallel with Transducers.jl

— Reply to this email directly, view it on GitHub https://github.com/JuliaML/TableTransforms.jl/issues/121#issuecomment-1251148046, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM2VRVAD276ONNHFGYJXLDV7B6CHANCNFSM6AAAAAAQQB7OCE . You are receiving this because you were mentioned.Message ID: @.***>

[juliohm]

I will close this issue as no one is working actively on it.

[JeffreySarnoff]

@juliohm reopen it -- I have been rearchitecting RollingFunctions.jl, in part to allow interoperability simply and both as client and as servicer. I am glad you posted, and this is now part of the direction -- not there yet, however each such request/suggestion better shapes the way of it.

[ I am not altering current code, as it needs to stay good and is somewhat set in its ways. To get the new made, it is faster and more reliable for me to implement through redesign. for instance https://github.com/JeffreySarnoff/RollingFunctions.jl/issues/33 ]

[juliohm]

@JeffreySarnoff I am reopening it assuming that something will be implemented either here or in RollingFunctions.jl to allow support for rolling function transforms.

[juliohm]

I am closing this issue again, we can re-open whenever we initiate developments related to TableTransforms.jl