Build pyOCD into Single Executable File - file missing in pyyaml

[kristofmulier]

I followed the guide https://github.com/mbedmicro/pyOCD/blob/master/docs/how_to_build.md to build an executable from pyOCD. Before diving into the problem, let's first list my system settings.  

1. My system

• I work on 64-bit Windows 10
• I've got Python 3.8
• I did not set up a virtual environment when starting the procedure, but work directly from my Python 3.8 installation.  

2. The procedure

I issue the following commands in a Windows console:

# We need to use upstream version of pyinstaller due to
# http://comments.gmane.org/gmane.comp.python.pyinstaller/6457
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

# Create single-file executables
pyinstaller --onefile pyocd/tools/gdb_server.py
pyinstaller --onefile pyocd/tools/flash_tool.py
pyinstaller --onefile pyocd/tools/pyocd.py

Everything goes well until the last command:

FileNotFoundError: [Errno 2] No such file or directory:
'c:\\python38\\lib\\site-packages\\pyyaml-5.2b1-py3.8-win-amd64.egg\\EGG-INFO\\top_level.txt'

I check if I've got pyyaml properly installed. I run the command pip install pyyaml and everything seems okay:

C:\Seafile\EMBEETLE IDE\pyOCD>pip install pyyaml
    Requirement already satisfied:
    pyyaml in c:\python38\lib\site-packages\pyyaml-5.2b1-py3.8-win-amd64.egg (5.2b1)

 

3. Solution

I believe I found a solution. I created the file top_level.txt myself in the right place:

After googling the keywords yaml and top_level.txt, I discovered that this file should have only two lines:

_yaml
yaml

I issue again the failed command:

$ pyinstaller --onefile pyocd/tools/pyocd.py

Now it works! At least - the compilation works.  

4. Problem with the executable

Unfortunately, the executable itself doesn't work. I get import errors:

C:\Seafile\EMBEETLE IDE\pyOCD\dist>pyocd.exe --version
    Traceback (most recent call last):
      File "pyocd\tools\pyocd.py", line 35, in <module>
    ImportError: attempted relative import with no known parent package
    [7508] Failed to execute script pyocd

[flit]

Sorry, that how_to_build.md document is waaay out of date. It should be removed.

Take a look at my feature/binary_release branch. This has a .spec file that will correctly build pyocd using PyInstaller. The branch is a little out of date, but should still work if you rebase it.

I haven't merged the changes from this branch because I intend to integrate it into CI.

[flit]

Sure, that would have the same effect as rebasing.

[kristofmulier]

Hi @flit , Maybe it was a naive attempt, but this is what I just tried:

1. I cloned the master branch of PyOCD to a folder on my computer.

2. I copy-pasted your pyocd.spec into that folder.

3. I install a couple of Python packages, because of your suggestions in dev-requirements.txt:

pip install pytest
pip install pytest-cov
pip install coverage
pip install elapsedtimer
pip install pylint
pip install tox

The pyinstaller package should already be okay on my computer, because I had already issued the following command before:

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

4. I navigate to the PyOCD folder and issue the command pyinstaller pyocd.spec:

C:\Seafile\EMBEETLE IDE\pyOCD>pyinstaller pyocd.spec

59 INFO: PyInstaller: 4.0.dev0+9dd34bdfba
59 INFO: Python: 3.8.0
60 INFO: Platform: Windows-10-10.0.18362-SP0
61 INFO: UPX is not available.
Traceback (most recent call last):
  File "<string>", line 2, in <module>
ModuleNotFoundError: No module named 'capstone'
Traceback (most recent call last):
  File "c:\python38\lib\site-packages\PyInstaller\utils\hooks\__init__.py", line 319, in get_module_file_attribute
    attr = loader.get_filename(package)
AttributeError: 'NoneType' object has no attribute 'get_filename'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\python38\lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\python38\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python38\Scripts\pyinstaller.exe\__main__.py", line 7, in <module>
  File "c:\python38\lib\site-packages\PyInstaller\__main__.py", line 112, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "c:\python38\lib\site-packages\PyInstaller\__main__.py", line 63, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "c:\python38\lib\site-packages\PyInstaller\building\build_main.py", line 732, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "c:\python38\lib\site-packages\PyInstaller\building\build_main.py", line 679, in build
    exec(code, spec_namespace)
  File "pyocd.spec", line 52, in <module>
    capstone_libs = collect_dynamic_libs("capstone")
  File "c:\python38\lib\site-packages\PyInstaller\utils\hooks\__init__.py", line 673, in collect_dynamic_libs
    pkg_base, pkg_dir = get_package_paths(package)
  File "c:\python38\lib\site-packages\PyInstaller\utils\hooks\__init__.py", line 535, in get_package_paths
    file_attr = get_module_file_attribute(package)
  File "c:\python38\lib\site-packages\PyInstaller\utils\hooks\__init__.py", line 337, in get_module_file_attribute
    raise ImportError
ImportError

[kristofmulier]

Hi @flit , I just noticed my question is actually not complete. I've added a fourth paragraph explaining what happens when I run the resulting executable.

My apologies.

[flit]

Why wouldn't you just install directly from dev-requirements.txt? Similar question for pyinstaller, why not just pip install pyinstaller? For the error, just install capstone.

[cederom]

Hey @flit, @kristofmulier is kinda fresh in Python, he is just moving away from OpenOCD towards pyOCD.. for some reason still wants to build binaries, where VirtualEnv and interpreter would do the job, but that seems like a nice training anyway.. after some time he will get into that point :-)

Kris you can install all python dependencies bundle with pip install -r dev-requirements.txt this is why that file was created :-) If you want to install capstone just run pip install captone.

Read the docs in the first place :-)

• https://pip.pypa.io/en/stable/
• https://pyinstaller.readthedocs.io/en/stable/

[kristofmulier]

Hi @flit and @cederom , Thanks a lot to both of you guys. It worked!

I installed capstone as suggested:

pip install capstone

Then I navigated to the cloned PyOCD folder and issued the command:

pyinstaller pyocd.spec

with 'pyocd.spec' being the spec-file from @flit . I get a nice pyocd.exe in the dist folder. I run it to see its version, and that works:

C:\Seafile\EMBEETLE IDE\pyOCD\dist>pyocd.exe --version
    0.23.1.dev10

---

Note: I just noticed the following in the error message from yesterday:

ModuleNotFoundError: No module named 'capstone'

I should have seen that before asking you guys for help. My sincere apologies.

[flit]

No worries. I'm really glad it worked! Sorry if I sounded rude earlier, was just in a hurry.

Can this issue be closed?

[kristofmulier]

Hi @flit , no worries :-) Thanks a lot for your great help. I think it would be good to push your pyocd.spec and dev-requirements.txt files from https://github.com/flit/pyOCD/tree/feature/binary_release to the master branch. This way, anybody can compile PyOCD.

Again many thanks for your help :-)

[flit]

The intent was to integrate CI support first, before merging the feature/binary_release branch. Though I guess it would make sense to go ahead and merge and deal with CI later.

[flit]

I also need to look into PyOxidizer.

[kristofmulier]

Hi @flit , PyOxidizer looks promising. So that would replace PyInstaller?

[flit]

Yep.

[carlosperate]

In the meantime, if you are still planning to use PyInstaller I'd recommend to have a look at perhaps implementing the logic from the spec file as a hook in pyOCD, since there are plans to provide a way to expose them to PyInstaller: https://github.com/pyinstaller/pyinstaller/issues/4232

Btw, also wanted to thank you @flit for that spec file in your https://github.com/flit/pyOCD/tree/feature/binary_release branch, it has certainly helped me as well.

[jroweFlint]

It looks like the binary_release branch is no longer in the repo. Is there any way to see what was in that .spec file?

[carlosperate]

In case its useful, this is the spec file I'm currently using (you can ignore the stuff that excludes and replaces svd files, as I am doing that only to reduce the executable size): https://github.com/carlosperate/ubittool/blob/master/package/pyinstaller-cli.spec

[jroweFlint]

Thanks so much for taking the time @carlosperate, That solved the issue we were having.