dolphinscheduler-sdk-python dolphinscheduler 3.2.0 support

In the dolphinscheduler 3.2.0, TaskDefinition has two non-null field: isCache and taskExecuteType.
If the former is null, workflow cannot be submitted, and if the latter is null, task will be retried with infinite fault-tolerance.

Nov 23 '23 07:11 liangjihua

hi @liangjihua , the latest code in main branch in project https://github.com/apache/dolphinscheduler-sdk-python/ already support version 3.2.0, but we still do not release it

you can install it via GitHub by yourself via command

pip install git+https://github.com/apache/dolphinscheduler-sdk-python.git#egg=apache-dolphinscheduler

it will install the latest code in branch main

Nov 24 '23 06:11 zhongjiajie

and btw, we will release version 4.1.0 to match dolphinscheduler 3.2.0, but we still have some unsolved issue which you can see in https://github.com/apache/dolphinscheduler-sdk-python/milestone/2

Nov 24 '23 06:11 zhongjiajie

hi @liangjihua , the latest code in main branch in project https://github.com/apache/dolphinscheduler-sdk-python/ already support version 3.2.0, but we still do not release it

you can install it via GitHub by yourself via command
pip install git+https://github.com/apache/dolphinscheduler-sdk-python.git#egg=apache-dolphinscheduler
it will install the latest code in branch main

please ping me if you have further question via this command

Nov 24 '23 06:11 zhongjiajie

[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# pip3 install git+https://github.com/apache/dolphinscheduler-sdk-python.git#egg=apache-dolphinscheduler
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Collecting apache-dolphinscheduler
  Cloning https://github.com/apache/dolphinscheduler-sdk-python.git to /tmp/pip-install-uxghsecs/apache-dolphinscheduler_6d58731b65e34eb89457ff249e7bbd51
  Running command git clone -q https://github.com/apache/dolphinscheduler-sdk-python.git /tmp/pip-install-uxghsecs/apache-dolphinscheduler_6d58731b65e34eb89457ff249e7bbd51
  fatal: unable to access 'https://github.com/apache/dolphinscheduler-sdk-python.git/': Failed connect to github.com:443; Connection refused
WARNING: Discarding git+https://github.com/apache/dolphinscheduler-sdk-python.git#egg=apache-dolphinscheduler. Command errored out with exit status 128: git clone -q https://github.com/apache/dolphinscheduler-sdk-python.git /tmp/pip-install-uxghsecs/apache-dolphinscheduler_6d58731b65e34eb89457ff249e7bbd51 Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement apache-dolphinscheduler (unavailable) (from versions: 0.1.0, 0.1.1, 2.0.5, 2.0.5.1, 2.0.7, 3.0.0a0, 3.0.0b1, 3.0.0b2, 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.1.1, 4.0.0, 4.0.1, 4.0.2a1, 4.0.2a2, 4.0.2a3, 4.0.2a4, 4.0.2a5, 4.0.2a6, 4.0.2a7, 4.0.2a8, 4.0.2a9, 4.0.2a10, 4.0.2a11, 4.0.2a12, 4.0.2, 4.0.2.1, 4.0.3, 4.0.4)
ERROR: No matching distribution found for apache-dolphinscheduler (unavailable)

Jan 12 '24 08:01 czxin788

@czxin788 fatal: unable to access 'https://github.com/apache/dolphinscheduler-sdk-python.git/': Failed connect to github.com:443; Connection refused You may need to check if your network can access github.com, you may not have configured git to use a proxy.

Jan 12 '24 09:01 liangjihua

@czxin788 fatal: unable to access 'https://github.com/apache/dolphinscheduler-sdk-python.git/': Failed connect to github.com:443; Connection refused You may need to check if your network can access github.com, you may not have configured git to use a proxy.

thank you ,now there have a new problem:

[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# curl www.baidu.com
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>使用百度前必读</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a>&nbsp;京ICP证030173号&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>
[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# 
[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# 
[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# pip3 install git+https://github.com/apache/dolphinscheduler-sdk-python.git#egg=apache-dolphinscheduler
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Collecting apache-dolphinscheduler
  Cloning https://github.com/apache/dolphinscheduler-sdk-python.git to /tmp/pip-install-26zwv6xs/apache-dolphinscheduler_62acdc9cedd24c89b4d291e51eccc7f4
  Running command git clone -q https://github.com/apache/dolphinscheduler-sdk-python.git /tmp/pip-install-26zwv6xs/apache-dolphinscheduler_62acdc9cedd24c89b4d291e51eccc7f4
  Resolved https://github.com/apache/dolphinscheduler-sdk-python.git to commit e04b836e54fb7bfd96527007cfeaa0fef4444b07
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/python3/bin/python3.6 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-26zwv6xs/apache-dolphinscheduler_62acdc9cedd24c89b4d291e51eccc7f4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-26zwv6xs/apache-dolphinscheduler_62acdc9cedd24c89b4d291e51eccc7f4/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-5pr0c_w0
       cwd: /tmp/pip-install-26zwv6xs/apache-dolphinscheduler_62acdc9cedd24c89b4d291e51eccc7f4/
  Complete output (6 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-26zwv6xs/apache-dolphinscheduler_62acdc9cedd24c89b4d291e51eccc7f4/setup.py", line 19
      from __future__ import annotations
      ^
  SyntaxError: future feature annotations is not defined
  ----------------------------------------
WARNING: Discarding git+https://github.com/apache/dolphinscheduler-sdk-python.git#egg=apache-dolphinscheduler. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement apache-dolphinscheduler (unavailable) (from versions: 0.1.0, 0.1.1, 2.0.5, 2.0.5.1, 2.0.7, 3.0.0a0, 3.0.0b1, 3.0.0b2, 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.1.1, 4.0.0, 4.0.1, 4.0.2a1, 4.0.2a2, 4.0.2a3, 4.0.2a4, 4.0.2a5, 4.0.2a6, 4.0.2a7, 4.0.2a8, 4.0.2a9, 4.0.2a10, 4.0.2a11, 4.0.2a12, 4.0.2, 4.0.2.1, 4.0.3, 4.0.4)
ERROR: No matching distribution found for apache-dolphinscheduler (unavailable)

Jan 12 '24 09:01 czxin788

[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# pwd
/root/czx/dolphinscheduler-sdk-python-main
[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# 
[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# 
[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# 
[root@vmware-k8s-master dolphinscheduler-sdk-python-main]# python3 setup.py  install
  File "setup.py", line 19
    from __future__ import annotations
    ^
SyntaxError: future feature annotations is not defined
[root@vmware-k8s-master dolphinscheduler-sdk-python-main]#

Jan 12 '24 09:01 czxin788

Try python3.7+

Jan 12 '24 09:01 liangjihua

pip3 install git+https://github.com/apache/dolphinscheduler-sdk-python.git#egg=apache-dolphinscheduler

not good, try python 3.8?

[root@vmware-k8s-master ~]# pip3.7  install git+https://github.com/apache/dolphinscheduler-sdk-python.git#egg=apache-dolphinscheduler
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Collecting apache-dolphinscheduler
  Cloning https://github.com/apache/dolphinscheduler-sdk-python.git to /tmp/pip-install-jdha1_k1/apache-dolphinscheduler_5fcc61d47604417fa23e7d90fdb751d3
  Running command git clone --quiet https://github.com/apache/dolphinscheduler-sdk-python.git /tmp/pip-install-jdha1_k1/apache-dolphinscheduler_5fcc61d47604417fa23e7d90fdb751d3
  Resolved https://github.com/apache/dolphinscheduler-sdk-python.git to commit e04b836e54fb7bfd96527007cfeaa0fef4444b07
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [63 lines of output]
      running dist_info
      creating /tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler.egg-info
      writing /tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler.egg-info/dependency_links.txt
      writing entry points to /tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler.egg-info/entry_points.txt
      writing requirements to /tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler.egg-info/requires.txt
      writing top-level names to /tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler.egg-info/top_level.txt
      writing manifest file '/tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      writing manifest file '/tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler.egg-info/SOURCES.txt'
      creating '/tmp/pip-modern-metadata-0euot2z6/apache_dolphinscheduler-4.1.0.dev0.dist-info'
      Traceback (most recent call last):
        File "/usr/local/python3.7.17/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/python3.7.17/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/python3.7.17/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 380, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 488, in run_setup
          self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 94, in <module>
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/command/dist_info.py", line 104, in run
          bdist_wheel = self.get_finalized_command('bdist_wheel')
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 304, in get_finalized_command
          cmd_obj = self.distribution.get_command_obj(command, create)
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 860, in get_command_obj
          klass = self.get_command_class(command)
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 979, in get_command_class
          self.cmdclass[command] = cmdclass = ep.load()
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 208, in load
          module = import_module(match.group('module'))
        File "/usr/local/python3.7.17/lib/python3.7/importlib/__init__.py", line 127, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
        File "<frozen importlib._bootstrap>", line 983, in _find_and_load
        File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 728, in exec_module
        File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 27, in <module>
          from .macosx_libfile import calculate_macosx_platform_tag
        File "/tmp/pip-build-env-yxmf_xk2/overlay/lib/python3.7/site-packages/wheel/macosx_libfile.py", line 43, in <module>
          import ctypes
        File "/usr/local/python3.7.17/lib/python3.7/ctypes/__init__.py", line 7, in <module>
          from _ctypes import Union, Structure, Array
      ModuleNotFoundError: No module named '_ctypes'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Jan 12 '24 09:01 czxin788

`ModuleNotFoundError: No module named '_ctypes' Try reinstalling python?

Jan 12 '24 09:01 liangjihua

I have try python 3.7 and 3.11, is not ok. but python 3.8 is ok. thank you for support

Jan 13 '24 03:01 czxin788

Good to hear you fix the problem @czxin788 and thanks for you helping @liangjihua 👍

Jan 14 '24 01:01 zhongjiajie