Connection aborted white submitting spark job

[k-pratik]

I am submitting spark job using following command: aztk spark cluster submit --id spark --name kp1 /Users/kumar.pratik/Documents/codebase//targettest-1.0-SNAPSHOT-jar-with-dependencies.jar --class com.pratik.test.MainTest

I always get following error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/aztk", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/aztk_cli/entrypoint.py", line 35, in main
    run_software(args)
  File "/usr/local/lib/python3.6/site-packages/aztk_cli/entrypoint.py", line 63, in run_software
    func(args)
  File "/usr/local/lib/python3.6/site-packages/aztk_cli/spark/endpoints/spark.py", line 31, in execute
    func(args)
  File "/usr/local/lib/python3.6/site-packages/aztk_cli/spark/endpoints/cluster/cluster.py", line 81, in execute
    func(args)
  File "/usr/local/lib/python3.6/site-packages/aztk_cli/spark/endpoints/cluster/cluster_submit.py", line 139, in execute
    wait=False
  File "/usr/local/lib/python3.6/site-packages/aztk/spark/client.py", line 92, in submit
    cluster_submit_helper.submit_application(self, cluster_id, application, wait)
  File "/usr/local/lib/python3.6/site-packages/aztk/spark/helpers/submit.py", line 115, in submit_application
    task = generate_task(spark_client, cluster_id, application)
  File "/usr/local/lib/python3.6/site-packages/aztk/spark/helpers/submit.py", line 26, in generate_task
    use_full_path=False)
  File "/usr/local/lib/python3.6/site-packages/aztk/utils/helpers.py", line 121, in upload_file_to_container
    blob_client.create_blob_from_path(container_name, blob_path, file_path)
  File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/blockblobservice.py", line 393, in create_blob_from_path
    timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/blockblobservice.py", line 513, in create_blob_from_stream
    initialization_vector=iv
  File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_upload_chunking.py", line 73, in _upload_blob_chunks
    range_ids = list(executor.map(uploader.process_chunk, uploader.get_chunk_streams()))
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_upload_chunking.py", line 140, in process_chunk
    return self._upload_chunk_with_progress(chunk_offset, chunk_bytes)
  File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_upload_chunking.py", line 154, in _upload_chunk_with_progress
    range_id = self._upload_chunk(chunk_offset, chunk_data) 
  File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_upload_chunking.py", line 169, in _upload_chunk
    timeout=self.timeout,
  File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/blockblobservice.py", line 848, in _put_block
    self._perform_request(request)
  File "/usr/local/lib/python3.6/site-packages/azure/storage/storageclient.py", line 266, in _perform_request
    raise ex
  File "/usr/local/lib/python3.6/site-packages/azure/storage/storageclient.py", line 242, in _perform_request
    raise AzureException(ex.args[0])
azure.common.AzureException: ('Connection aborted.', timeout('The write operation timed out',))

[jafreck]

This error indicates that there was a client-side connection issue with Azure storage. Do you have network rules on the client running this command that might prevent connections to Azure storage?

If not, and this is just a client-side connection issue, we can not much except increase the timeout duration for the request.

What version of AZTK are you using?

[k-pratik]

@jafreck I don't there is network rules as couple of days back I was able to run the same command and my job was failing for some other reason (Had raised another ticket for that related to wasb path in hadoop file path) Size of my jar file is 129 MB I am using aztk version 0.7.0b12

[k-pratik]

@jafreck Could you please help me in closing this https://github.com/Azure/aztk/issues/500