hyperspace
hyperspace copied to clipboard
microsoft
An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.
**Feature requested** As a **developer at Airbnb**, I want **to be able to index over s3 based scan nodes**, in order to **create hyperspace indexes**. I am not entirely sure...
### Describe the issue Is Project HyperSpace Deprecated? or is it moved to different repo? ### To Reproduce ### Expected behavior ### Environment
**Feature requested** As a data engineer, I wish to use hyperspace to optimize hudi data. **Acceptance criteria** Describe what you consider (e.g., functionality requirement) has to be true to mark...
### Describe the issue I tried to use this `MinMaxAnalysisUtil` to analyze distribution of column. It worked well on small data set, however threw exception on my TPC-H dataset which...
### Describe the issue If indexes are incrementally refreshed, `Hyperspace.explain` will not display `Indexes Used` even if indexes are applied. The culprit is https://github.com/microsoft/hyperspace/blob/ad4b3af899f402544f04840ae7f4204e0627018a/src/main/scala/com/microsoft/hyperspace/index/plananalysis/PlanAnalyzer.scala#L216 and https://github.com/microsoft/hyperspace/blob/ad4b3af899f402544f04840ae7f4204e0627018a/src/main/scala/com/microsoft/hyperspace/index/plananalysis/PlanAnalyzer.scala#L149 , since `indexLocation` doesn't...
**Feature requested** Currently, Hyperspace creates indexes under the system path specified by `spark.hyperspace.system.path`. The user should be able to specify different path to create/search the indexes. Note that #242 removes...
**Feature requested** As a developer, I want this feature in order to leverage Hyperspace index in Presto/Trino. Not sure if this is similar request as https://github.com/microsoft/hyperspace/issues/514. I'm wondering if anyone...
### Describe the issue I'm wondering if what-if API mentioned in Hyperspace available for index recommendation? I checked docs, blogs and PRs but couldn't find anything related. Did I miss...
### Describe the issue Currently hs.explain looks like this: ``` ============================================================= Indexes used: ============================================================= deptIndex1:/indexes/deptIndex1/v__=0 empIndex1:/indexes/empIndex1/v__=0 ``` remove the extra `v__=0` from this ### To Reproduce ### Expected behavior ###...
### Describe the issue Trying to use hyperspace API produce a runtime exception. `java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(Lscala/PartialFunction;)Lorg/apache/spark/sql/catalyst/plans/logical/LogicalPlan; at com.microsoft.hyperspace.index.covering.CoveringIndexRuleUtils$.transformPlanToUseIndexOnlyScan(CoveringIndexRuleUtils.scala:110) at com.microsoft.hyperspace.index.covering.CoveringIndexRuleUtils$.transformPlanToUseIndex(CoveringIndexRuleUtils.scala:79) at com.microsoft.hyperspace.index.covering.FilterIndexRule$.applyIndex(FilterIndexRule.scala:148) at com.microsoft.hyperspace.index.rules.HyperspaceRule.apply(HyperspaceRule.scala:75) at com.microsoft.hyperspace.index.rules.HyperspaceRule.apply$(HyperspaceRule.scala:62) at com.microsoft.hyperspace.index.covering.FilterIndexRule$.apply(FilterIndexRule.scala:129) at com.microsoft.hyperspace.index.rules.ScoreBasedIndexPlanOptimizer.$anonfun$recApply$3(ScoreBasedIndexPlanOptimizer.scala:56) at...
