apache
HDFS-16749. RBF: Gets the wrong directory information from Trash
Description of PR
As described in HDFS-16749.
Purpose ：[RBF] Fix ls user trash path returns mountpoint path that real nameservice's trash path is not exists
There are two nameservices ns0 ns1, and ns0 is the default nameservice.
(1) Add moutTable
/home/data -> (ns0, /home/data)
/data1/test1 -> (ns1, /data1/test1 )
/data2/test2 -> (ns1, /data2/test2 )
(2) add file
/home/data/file1
/data1/test1/file1
(3) mv files to trash
ns0: /user/test-user/.Trash/Current/home/data/file1
ns1: /user/test-user/.Trash/Current/data1/test1/file1
(4) client via DFSRouter: hdfs dfs -ls /user/test-user/.Trash/Current ,this will return
/user/test-user/.Trash/Current/home
/user/test-user/.Trash/Current/data1
/user/test-user/.Trash/Current/data2 (But this in fact not exist in ns1)
So, this PR is for this purpose, if a namservice that mount point dir in trash is not really exists, it should be not display in user trash view via DFSRouter
@LiuGuH Thank you for your contribution! Could you please provide more details about your proposed modification?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 42m 52s | trunk passed | |
| +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 30s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 42s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 43s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 21s | trunk passed | |
| +1 :green_heart: | shadedclient | 32m 54s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed | |
| +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 31s | the patch passed | |
| +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 30s | the patch passed | |
| -1 :x: | blanks | 0m 0s | /blanks-eol.txt | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 17s | /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 3 new + 3 unchanged - 0 fixed = 6 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| -1 :x: | spotbugs | 1m 23s | /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html | hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 32m 31s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 22m 2s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | |
| 144m 57s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
| org.apache.hadoop.hdfs.server.federation.resolver.FileSubclusterResolver.getMountPointsWithSrc(String, SortedMap) makes inefficient use of keySet iterator instead of entrySet iterator At FileSubclusterResolver.java:of keySet iterator instead of entrySet iterator At FileSubclusterResolver.java:[line 184] | |
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpc |
| hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6317 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 575e6ad58f8a 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c0b98fc3d73cbf7902ea917593d3fa86aa70c9e5 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/1/testReport/ |
| Max. process+thread count | 3448 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@zhangxiping1 I see that you created HDFS-16749, pr #5039, and @LiuGuH also submitted a PR. Can we compare and explain the difference between the two prs?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 12m 11s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 43m 18s | trunk passed | |
| +1 :green_heart: | compile | 0m 39s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 35s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 29s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 41s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 41s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 31s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 19s | trunk passed | |
| +1 :green_heart: | shadedclient | 32m 19s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed | |
| +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 30s | the patch passed | |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 28s | the patch passed | |
| -1 :x: | blanks | 0m 0s | /blanks-eol.txt | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 18s | /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 18s | the patch passed | |
| +1 :green_heart: | shadedclient | 32m 21s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 21m 39s | hadoop-hdfs-rbf in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | |
| 155m 31s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6317 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux deaf1fa46cc3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e73bf6d30fee4231d6c465225f5f3e25270e6213 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/2/testReport/ |
| Max. process+thread count | 2840 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 43m 21s | trunk passed | |
| +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | checkstyle | 0m 30s | trunk passed | |
| +1 :green_heart: | mvnsite | 0m 40s | trunk passed | |
| +1 :green_heart: | javadoc | 0m 42s | trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 32s | trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 19s | trunk passed | |
| +1 :green_heart: | shadedclient | 32m 14s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed | |
| +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 0m 31s | the patch passed | |
| +1 :green_heart: | compile | 0m 29s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 0m 29s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 18s | /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | spotbugs | 1m 19s | the patch passed | |
| +1 :green_heart: | shadedclient | 32m 19s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 21m 44s | hadoop-hdfs-rbf in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | |
| 143m 49s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6317 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 289fa402ff9b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 41aeea702f4d5e4683c708fc1b3fef606459190d |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/3/testReport/ |
| Max. process+thread count | 2841 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Thanks @LiuGuH for your works.
(4) client via DFSRouter: hdfs dfs -ls /user/test-user/.Trash/Current ,this will return
I am confused why it returns wrong subdirs now? Thanks.
@zhangxiping1 I see that you created HDFS-16749, pr #5039, and @LiuGuH also submitted a PR. Can we compare and explain the difference between the two prs? Is the same problem, returning the extra entry is the Trash prefix with mount point information(top subdirectory)
(4) client via DFSRouter: hdfs dfs -ls /user/test-user/.Trash/Current ,this will return
I am confused why it returns wrong subdirs now? Thanks.
getListing can be separated into two parts (1) getListingInt
subtractTrashCurrentPath("/user/test-user/.Trash/Current") will get "" as moint point in default nameservices ns0, then construct monit point /user/test-user/.Trash/Current -> (ns0,/user/test-user/.Trash/Current) And then execute getListingInt.
(2) subclusterResolver.getMountPoints(src) getMountPoints("/user/test-user/.Trash/Current") will get all children of "" , this will return all mount points
home
data1
data2
and then it be display
/user/test-user/.Trash/Current/home
/user/test-user/.Trash/Current/data1
/user/test-user/.Trash/Current/data2
getMountPoints is necessary because we will not see all subtrash dirs without it. The problem is that these mountpoint dir maybe not exists if user does not delelt any dir or file into trash. And now it is simple display all mount point.
My way of dealing with it is that for every child path，found its mount point ,and getFileInfo with it . If it is really exits, display it. @Hexiaoqiao @slfan1989
I think this PR is based on HDFS-17263, but we found that it introduced another bug, which was filed as HDFS-17842. In our environment, we reverted HDFS-17263 and have backported #5039 instead for now.
