spline-spark-agent Provide Spark 3.4 Support for Spline w/ Backwards Compatibility

Addressing #705 ,

Makes changes to failing tests to perform different behavior based on Spark version
No new functionality actually required to meet compatibility

TODO:

[x] Create Agent Bundle POM
[ ] Validate approach is sensible
[x] Rule out additional regression from changed Spark behavior
[ ] Request CI addition for Spark 3.4

Mar 08 '24 22:03 rycowhi

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

Mar 11 '24 12:03 sonarqubecloud[bot]

@rycowhi Thanks a million for your effort. Guys let me take a look at those event pairs more closely and I'll get back to you later. I'm a bit out of capacity at the moment, but will try to find some time during the next day or two.

Mar 12 '24 14:03 wajda

Just checking on status of this PR. We are waiting on this feature. @wajda any updates.

Mar 18 '24 03:03 devsaik

Just checking on status of this PR. We are waiting on this feature. @wajda any updates.

Not yet, but it's high on my todo list for this week.

Mar 18 '24 13:03 wajda

@wajda thanks for taking a look at @rycowhi's PR. Let me know if you'll be able to get to it this week - we're also waiting on this feature.

Mar 21 '24 15:03 luke3481

@wajda Looks like this feature is in high demand. Our team is also watching the PR merge, highly appreciate if you could review this as early as possible. Thank you in advance.

Mar 21 '24 18:03 kaimalsreeja

Sorry folks, I've still been buried in work on another high-priority project, and have yet to obtain time allocation approval for this one. The Spline project was essentially put on hold by the company last year, that's the reason why it received almost no support for the past X months. I'm not giving up, but I can't really spend my free time on this any more, so trying to get some official allocation from the employer.

Apr 04 '24 14:04 wajda

Sorry folks, I've still been buried in work on another high-priority project, and have yet to obtain time allocation approval for this one. The Spline project was essentially put on hold by the company last year, that's the reason why it received almost no support for the past X months. I'm not giving up, but I can't really spend my free time on this any more, so trying to get some official allocation from the employer.

Hello! Is there any news regarding this Spark 3.4 support? I've been using Spline since a long time and the fact that there is no support for Scala 2.13 is going to affect a lot of projects and maybe the need to find a new listener.

If there is any way I could contribute to this feature, or to speed things up, please let me know!

Thanks!

Edit: I've just seen Upgrade org.scala-lang:scala-library from 2.12.17 to 2.13.14 #806

May 24 '24 10:05 ramonje5

@ramonje5 thank you for the message. Unfortunately our company has yet to sort out its plans and book of work for Spline. Currently the team has zero capacity to work on it.

If there is any way I could contribute to this feature, or to speed things up, please let me know!

Thanks!

Edit: I've just seen https://github.com/AbsaOSS/spline-spark-agent/pull/806

The PR #806 is automatically created by Snyk and won't work. Please ignore it. Upgrading to Scala 2.13 cannot be a simple change like that, it would require adding another vertical into the build matrix, so to speak. Basically the agent is built for every supported Sclala+Spark version combination.

If you want to help that would be awesome if you take this current PR, test it, address @rycowhi's points in the related issue #705. When testing succeeds and those points are addressed we can merge and release this PR.

May 24 '24 11:05 wajda

@ramonje5 thank you for the message. Unfortunately our company has yet to sort out its plans and book of work for Spline. Currently the team has zero capacity to work on it.

If there is any way I could contribute to this feature, or to speed things up, please let me know! Thanks! Edit: I've just seen #806

The PR #806 is automatically created by Snyk and won't work. Please ignore it. Upgrading to Scala 2.13 cannot be a simple change like that, it would require adding another vertical into the build matrix, so to speak. Basically the agent is built for every supported Sclala+Spark version combination.

If you want to help that would be awesome if you take this current PR, test it, address @rycowhi's points in the related issue #705. When testing succeeds and those points are addressed we can merge and release this PR.

At least for the versions maintained off of the existing cross-builds is there anything else to look into? I believe the only outstanding piece was seeing that there are now two (different commands) emitted off of a CTAS in 3.4 but have confirmed this is a new Spark behavior (by seeing the same, multiple events happen on a barebones query listener). Is the outstanding piece to see how that would look in the Spline UI, or?

May 25 '24 16:05 rycowhi

Fixed a small issue (missing toLowerCase on the righthand side of the assert statement) in the test above.
Added a cross-build config for Spark 3.4 + Scala 2.12 on the TeamCity.

When all builds pass I'll test it manually on my end and if no issue found I'll merge the PR.

Jul 09 '24 19:07 wajda