Terry Kim

[FEATURE REQUEST]: Support StructArray as a return type in ArrowFunctions.VectorUdf

3

We need to support `StructArray` as a return type in `ArrowFunctions.VectorUdf`. To repro: ```csharp Func udf = ArrowFunctions.VectorUdf((Func) someUDFWrapper); ``` You will get: ``` Unhandled Exception: System.ArgumentException: Apache.Arrow.StructArray is not...

Fix E2E tests for Spark 3.0.

2

#348 enabled building/testing against Spark 3.0. The following tests have been disabled to track separately: #### Microsoft.Spark.E2ETest.IpcTests * [x] `BroadcastTests.*` * [x] `DataFrameTests.TestDataFrameGroupedMapUdf` / `DataFrameTests.TestGroupedMapUdf` * [x] `DataFrameTests.TestVectorUdf` / `DataFrameTests.TestDataFrameVectorUdf`...

Extract common classes from src/scala/microsoft-spark-<version>.

5

We create multiple jars during our builds to accommodate multiple versions of Apache Spark. In the current approach, the implementation is copied from one version to another and then necessary...

Automate C# formatting using dotnet-format

8

Explore adopting https://github.com/dotnet/format to automate the process of formatting source code similar to scalafmt (#153).

[Bug] Hyperspace.explain does not display "Indexes used" if indexes are incrementally refreshed

1

### Describe the issue If indexes are incrementally refreshed, `Hyperspace.explain` will not display `Indexes Used` even if indexes are applied. The culprit is https://github.com/microsoft/hyperspace/blob/ad4b3af899f402544f04840ae7f4204e0627018a/src/main/scala/com/microsoft/hyperspace/index/plananalysis/PlanAnalyzer.scala#L216 and https://github.com/microsoft/hyperspace/blob/ad4b3af899f402544f04840ae7f4204e0627018a/src/main/scala/com/microsoft/hyperspace/index/plananalysis/PlanAnalyzer.scala#L149 , since `indexLocation` doesn't...

[FEATURE REQUEST]: Ability to specify path to create index outside system path.

7

**Feature requested** Currently, Hyperspace creates indexes under the system path specified by `spark.hyperspace.system.path`. The user should be able to specify different path to create/search the indexes. Note that #242 removes...

Update azure-pipelines.yml to build C#/Scala example apps

1

### Describe the issue #90 and #94 introduced example apps, but the build pipeline was not updated. The [azure-pipelines.yml](https://github.com/microsoft/hyperspace/blob/master/azure-pipelines.yml) needs to updated to make sure they compile.

[FEATURE REQUEST]: Publish SNAPSHOT JARs as a CI process

2

**Feature requested** It will be nice to publish SNAPSHOT JARs daily (when there are new commits to master). **Acceptance criteria** SNAPSHOT JARs are available here: https://oss.sonatype.org/content/repositories/snapshots/com/microsoft/hyperspace/

Update Scala Coding Style guideline for scalafmt

1

Update https://github.com/microsoft/hyperspace/blob/master/docs/coding-guidelines/scala-coding-style.md on how to set up `scalafmt` in IDE so that contributors get consistent result.

Revisit logical plan serialization

16

### Describe the issue Hyperspace serializes the logical plan of a dataframe used for creating an index so that the plan can be reused for refreshing the index, etc. The...