Migrate bsddb3 -> berkeleydb

[spencerwilson]

Hi, Thanks for your work on this. Wanted to report that

• bsddb3 is deprecated; Python 3.9 is the final minor release line of Python that it'll support
• bsddb3's final release, 6.2.9, was only tested to work with Berkeley DB 6.2, and empirically fails (see below) with Berkeley DB 18.1.40.
• Only bsddb3's successor, berkeleydb (also at the page linked above), supports Berkeley DB versions 18.1.x (the current release line)
• => walletool is difficult to use on systems that "need" to have installed Berkeley DB 18.1.x (e.g., in Homebrew world, vim depends on perl depends on berkeley-db)

Workaround for macOS homebrew: brew install berkeley-db@4, a different formula that can be installed alongside berkeley-db. Then do something like

% BERKELEYDB_DIR=/usr/local/Cellar/berkeley-db@4/4.8.30 pip install bsddb3===6.2.9
Collecting bsddb3===6.2.9
  Using cached bsddb3-6.2.9.tar.gz (230 kB)
Using legacy 'setup.py install' for bsddb3, since package 'wheel' is not installed.
Installing collected packages: bsddb3
    Running setup.py install for bsddb3 ... done
Successfully installed bsddb3-6.2.9

Example below, from my attempts on running on macOS:

% pip install bsddb3===6.2.9
Collecting bsddb3===6.2.9
  Using cached bsddb3-6.2.9.tar.gz (230 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/ssw/.pyenv/versions/3.9.4/bin/python3.9 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-install-eanky500/bsddb3_16d389d6a4d44d0fa5267086b17faeec/setup.py'"'"'; __file__='"'"'/private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-install-eanky500/bsddb3_16d389d6a4d44d0fa5267086b17faeec/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-pip-egg-info-g_pi460g
         cwd: /private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-install-eanky500/bsddb3_16d389d6a4d44d0fa5267086b17faeec/
    Complete output (2 lines):
    Can't find a local Berkeley DB installation.
    (suggestion: try the --berkeley-db=/path/to/bsddb option)
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/f0/24/92034482656945fc6ceb10551222b43a0ff8d0c87e15839120487820067e/bsddb3-6.2.9.tar.gz#sha256=70d05ec8dc568f42e70fc919a442e0daadc2a905a1cfb7ca77f549d49d6e7801 (from https://pypi.org/simple/bsddb3/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement bsddb3===6.2.9
ERROR: No matching distribution found for bsddb3===6.2.9

Then I googled a bit, and found how to effectively pass that option via environment variable:

% BERKELEYDB_DIR=/usr/local/Cellar/berkeley-db/18.1.40 pip install bsddb3===6.2.9
Collecting bsddb3===6.2.9
  Using cached bsddb3-6.2.9.tar.gz (230 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/ssw/.pyenv/versions/3.9.4/bin/python3.9 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-install-8e8zt5db/bsddb3_a8250a56b2774fd697c0d453d1bd8b89/setup.py'"'"'; __file__='"'"'/private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-install-8e8zt5db/bsddb3_a8250a56b2774fd697c0d453d1bd8b89/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-pip-egg-info-e5951msm
         cwd: /private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-install-8e8zt5db/bsddb3_a8250a56b2774fd697c0d453d1bd8b89/
    Complete output (8 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-install-8e8zt5db/bsddb3_a8250a56b2774fd697c0d453d1bd8b89/setup.py", line 42, in <module>
        import setup3
      File "/private/var/folders/bb/zz6yzy3154b3py7_qq2xv5kc0000gn/T/pip-install-8e8zt5db/bsddb3_a8250a56b2774fd697c0d453d1bd8b89/setup3.py", line 354, in <module>
        db_ver = (int(fullverstr[0]), int(fullverstr[2]))
    ValueError: invalid literal for int() with base 10: '.'
    Trying to use the Berkeley DB you specified...
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/f0/24/92034482656945fc6ceb10551222b43a0ff8d0c87e15839120487820067e/bsddb3-6.2.9.tar.gz#sha256=70d05ec8dc568f42e70fc919a442e0daadc2a905a1cfb7ca77f549d49d6e7801 (from https://pypi.org/simple/bsddb3/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement bsddb3===6.2.9
ERROR: No matching distribution found for bsddb3===6.2.9

Final note: I'm unsure if my workaround is actually working, because when running

python wt_extract_keys.py -d ~/Desktop/wallet.dat -v 30

nothing is being written to stdout or stderr, and exit status is 0. TBD what the problem is.

[ZuesSSSSS]

Just saw this issue. Didn't notice and didn't add it to my PR. Let me know what you think