Kun-Lung Wu

Can I use spark.createDataFrame() with a list of ObjectRef from various remote Ray workers?

18

I am trying to see if I can create a Spark DataFrame with a list of ObjectRef from several remote Ray tasks that have put large objects into their local...

Support better integration between Ray and Spark in passing ObjectRef without actually moving data

3

## Overview As a Codeflare user, I want to use Ray and Spark alternately to execute my end-to-end ML jobs. Some steps might be executed more efficiently using Ray, while...

### Search before asking - [X] I searched the [issues](https://github.com/IBM/data-prep-lab/issues) and found no similar issues. ### Component Library/core ### Feature When running a transform, the logging INFO on my terminal...

### Search before asking - [x] I searched the [issues](https://github.com/data-prep-kit/data-prep-kit/issues) and found no similar issues. ### Component Python Runtime ### Feature DPK `lakehouse` config only allows input/output `tables` to be...

### Search before asking - [x] I searched the [issues](https://github.com/data-prep-kit/data-prep-kit/issues) and found no similar issues. ### Component transforms/text_encoder ### What happened + What you expected to happen Here is my...