Hadoop/Spark slower on Windows VS Linux

Open khaledmahmoudintel opened this issue 2 years ago • 2 comments

Hi,

I have two machines that have identical Hardware. CPU, RAM, and BIOS configurations are exactly the same. I am running Spark 3.3.1 with Hadoop 3.3.1. The benchmark is also exactly the same. I am not using any HDFS at all.

Problem: Spark on Windows runs slower than Linux

Any idea why Windows implementation is slower? What is exactly inside hadoop.dll and winutils.exe.

Feb 24 '24 01:02 khaledmahmoudintel

Thank you for your email.We’ll disposal and reply as soon as possible~Best Wishes~

Feb 24 '24 01:02 luanchao19990

I share the same experience. In my linxu server the hdfs dfs -ls / command return the result immediately. In the windows server, it takes about 1 mins. The windows server has better hardware the the linux server.

May 16 '24 13:05 pengfei99