Nicoda-27

chore(deps): upgrade to pydantic-2

7

This PR aims at - upgrade pydantic dependency to version 2 - update and fix code and test after upgrade

**What is the new container you'd like to have?** Spark connect introduces a decoupled client-server architecture to allow remote connectivity to spark server, official documentation is [here](https://spark.apache.org/docs/latest/spark-connect-overview.html). It's used by...