feast-dev
Snowflake support does not include list features
Expected Behavior
Snowflake offline store works with list features
Current Behavior
Right now Snowflake produces errors when using list features. This isn't caught by tests because of this skip with the aws provider: https://github.com/feast-dev/feast/blob/master/sdk/python/tests/integration/registration/test_universal_types.py#L39
@adchia @woop here is the docs related to semi-structured data in Snowflake: https://docs.snowflake.com/en/sql-reference/data-types-semistructured.html#array
Arrays for list feature types are possible in Snowflake, but there is no enforcement of data type within the array, its stored as type VARIANT, where VARIANT, can store values of any other type, including OBJECT and ARRAY, up to a maximum size of 16 MB. Snowflake does not currently support fixed-size arrays or arrays of elements of a specific non-VARIANT type.
Im curious to your initial thoughts on this. We can certainly test an array if it complies with a certain data type that is supported or throw an error. If it does then execellent, if not we throw an error to remove the feature
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
bump
On Tue, Apr 25, 2023 at 8:30 PM stale[bot] @.***> wrote:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
— Reply to this email directly, view it on GitHub https://github.com/feast-dev/feast/issues/2280#issuecomment-1522633970, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATSRCU5DJDXD5YTTP23ET2LXDB3FFANCNFSM5NPKRUQQ . You are receiving this because you were assigned.Message ID: @.***>
Looks like we'll need this for our implementation so I'll be taking a look at implementing this in our sprint starting tomorrow.
@JohnLemmonMedely I am afraid snowflake does not support structured types on the server side. All semistructured types are currently returns as strings. We are working to support this in the next couple of quarters.
@sfc-gh-madkins I was planning on using the method you suggested earlier and attempt to cast them to what they need to be and throwing errors when we can't. It'll be better to guarantee data types via Snowflake but until that's possible I can enforce those guarantees myself via the data pipeline.
We can certainly test an array if it complies with a certain data type that is supported or throw an error. If it does then execellent, if not we throw an error to remove the feature
will look forward to the PR :)
On Tue, Sep 5, 2023 at 4:51 PM John Lemmon @.***> wrote:
@sfc-gh-madkins https://github.com/sfc-gh-madkins I was planning on using the method you suggested earlier and attempt to cast them to what they need to be and throwing errors when we can't. It'll be better to guarantee data types via Snowflake but until that's possible I can enforce those guarantees myself via the data pipeline.
We can certainly test an array if it complies with a certain data type that is supported or throw an error. If it does then execellent, if not we throw an error to remove the feature
— Reply to this email directly, view it on GitHub https://github.com/feast-dev/feast/issues/2280#issuecomment-1707355768, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATSRCU3WGHHQXHAVBAE467LXY6NH7ANCNFSM5NPKRUQQ . You are receiving this because you were mentioned.Message ID: @.***>
@sfc-gh-madkins I had to pause my work on the PR but we were able to use my fork in production. I'm trying to see if it makes sense at this point to get the PR ready to merge or if I should just wait on Snowflake adding structured types instead. Is that still on the Roadmap to add early next year?
let me ping to see where this is at
On Fri, Dec 8, 2023 at 11:41 AM John Lemmon @.***> wrote:
@sfc-gh-madkins https://github.com/sfc-gh-madkins I had to pause my work on the PR but we were able to use my fork in production. I'm trying to see if it makes sense at this point to get the PR ready to merge or if I should just wait on Snowflake adding structured types instead. Is that still on the Roadmap to add early next year?
— Reply to this email directly, view it on GitHub https://github.com/feast-dev/feast/issues/2280#issuecomment-1847578811, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATSRCU4FFUY62RAO4MRN3XTYINGNJAVCNFSM5NPKRUQ2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBUG42TOOBYGEYQ . You are receiving this because you were mentioned.Message ID: @.***>
@sfc-gh-madkins just following up on if Snowflake still plans on supporting structured types soon. I just need to know if work will start on it over the next quarter or not.
negative. this is continuing to be pushed out due to lack of customer demand. however, we do now have vector support capabilities for both fixed and floating point 32 bit numbers, however, the client arent ready yet
On Fri, Dec 15, 2023 at 10:09 AM John Lemmon @.***> wrote:
@sfc-gh-madkins https://github.com/sfc-gh-madkins just following up on if Snowflake still plans on supporting structured types soon. I just need to know if work will start on it over the next quarter or not.
— Reply to this email directly, view it on GitHub https://github.com/feast-dev/feast/issues/2280#issuecomment-1858122582, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATSRCUZRMMJRFCX2LJLWSOLYJRY5JAVCNFSM5NPKRUQ2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBVHAYTEMRVHAZA . You are receiving this because you were mentioned.Message ID: @.***>