parquet-java icon indicating copy to clipboard operation
parquet-java copied to clipboard

PARQUET-1711: support recursive proto schemas by limiting recursion depth

Open jinyius opened this issue 3 years ago • 5 comments

  • This is an alternative approach to supporting recursion to apache#445 and apache#988.
  • This approach could address the other recursion related issues (PARQUET-129, PARQUET-554).
  • TODO: ReadSupport

Jira

  • [x] My PR addresses the following Parquet Jira issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR"
    • https://issues.apache.org/jira/browse/PARQUET-1711
    • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

  • [x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
    • ProtoSchemaConverterTest#test*Recursion
    • ProtoWriteSupportTest#test*Recursion

Commits

  • [x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • [x] In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain Javadoc that explain what it does

jinyius avatar Sep 08 '22 07:09 jinyius

ping

jinyius avatar Sep 14 '22 22:09 jinyius

fixed missing dep issue. can someone approve the ci flow?

jinyius avatar Sep 28 '22 18:09 jinyius

thanks for the review. updated to handle the logging perf concern as well as fixing the javadoc errors.

jinyius avatar Sep 30 '22 18:09 jinyius

ping

jinyius avatar Oct 07 '22 16:10 jinyius

Mostly looks reasonable, I'm not too familiar with parquet-mr @shangxinli can you recommend someone who might be able to give a better review?

emkornfield avatar Oct 09 '22 03:10 emkornfield

Mostly looks reasonable, I'm not too familiar with parquet-mr @shangxinli can you recommend someone who might be able to give a better review?

pinging @shangxinli :)

jinyius avatar Oct 17 '22 17:10 jinyius

@ggershinsky Can you have a look?

shangxinli avatar Oct 17 '22 17:10 shangxinli

I would also like to recommend adding @matthieun as a co-author to this PR, per the discussion in the parallel PR.

ggershinsky avatar Oct 19 '22 07:10 ggershinsky

I would also like to recommend adding @matthieun as a co-author to this PR, per the discussion in the parallel PR.

~how do you do this?~ i think i did it right...

jinyius avatar Oct 19 '22 21:10 jinyius

can someone retry the github actions? there seemed to have been a transient issue that caused one of the test/build targets to fail. i'd like to get this change in this week.

jinyius avatar Oct 24 '22 17:10 jinyius

@ggershinsky what is the process to merge this? Does parquet-mr just use the github UI?

emkornfield avatar Oct 31 '22 04:10 emkornfield

yep, just the squash/merge button.

ggershinsky avatar Oct 31 '22 14:10 ggershinsky

@ggershinsky

i'd love to just hit the button. i don't see it. the workflow for travis ci had a failure due to a transient connection issue, and so it wasn't giving me the option to merge. the ui messaging also states that "Only those with write access to this repository can merge pull requests."

jinyius avatar Oct 31 '22 15:10 jinyius

@shangxinli are you ok with this PR in its current form?

ggershinsky avatar Nov 01 '22 07:11 ggershinsky

yeah, i still don't see a button to merge. it now shows everything approved, checks passed, and no conflicts.

i think a committer needs to merge.

jinyius avatar Nov 01 '22 15:11 jinyius

@jinyius only committers can see the button. I was asking because different repos have different commit procedures. Should be able to merge this soon as long as @shangxinli doesn't express concerns.

emkornfield avatar Nov 01 '22 18:11 emkornfield

LGTM

shangxinli avatar Nov 02 '22 14:11 shangxinli