Hima Patel

Results 46 comments of Hima Patel

add license check transform

@shivdeep-singh-ibm This module (folder) should be called license filtering

[Bug] text_encoder tests are not verifying the resulting embeddings to avoid floating point rounding errors

@dolfim-ibm Is this issue solved?

Update KFP super pipeline for code

It can go after malware. Can the super pipeline be automatically generated based on the order of transforms given as input? For eg, I may want to call the filter...

Update KFP super pipeline for code

Both of these are separate transforms and both need to be added. Yes there is a dependency here. @Param-S @shivdeep-singh-ibm

[Feature] Filter toxic words/sentences from a given text

No, it is not. There is some overlap in the labels of hate, toxicity but they have been used as separate labels as well in the literature.

[Bug] pdf2parquet must calculate hash and size on the file

@sujee Can you test and see if this can be closed?

Adding a resources page

I dont have a strong opinion here. My point is that it should be easy for the user to find the information and usually I have seen repos have this...

[Bug] pdf2parquet ray version erroring out when downloading models for the very first time

Folks, is this issue fixed? cc @touma-I

[Bug] Rootdir not found in util package for Document Quality Module

@touma-I for you to triage pls..

[Feature] a transform to perform file level de-dupe (exact)

@touma-I @daw3rd pls add in your comments and suggestions.