Backblaze
sync doesn't work
b2 sync runs, but doesn't upload anything
b2 version: 4.3.0
python: 3.12.3
OS: Ubuntu
this command has been running for hours and hasn't done anything. if i use the verbose mode, it just outputs some logs. the communication to the bucket seems to work fine
@ppolewicz
real 0m0.004s
user 0m0.001s
sys 0m0.004s
I will say that this is the root drive of my ZFS so it contains a LOT of files, but I have in the past used rsync, and it is able to calculate all the files in the directory
It shows zero progress after such a long time?
You said "some" logs, which suggests the process is not stuck, but doing something. What do the logs show?
By a lot do you mean a million? billion?
b2 sync should be able to handle a billion files but according to my estimate just listing through all that would take about a week. Are there any remote drives mounted under /kapital or something like that, which could cause listing files to be slow?
@ppolewicz I recorded the first 40s of the --verbose output. I also limited it to a subfolder of my backup drive. this volume is in fact mounted on a network, but it's not really seeing any major traffic at the moment. Maybe you can help tell where it's stuck. I'll try another smaller subfolder
https://github.com/user-attachments/assets/f2055208-4236-46ee-91fa-6e7fdde6119c
I resolved my issue! I set --destination-server-side-encryption=SSE-B2 and it worked. I was so confused
I would highly suggest some sort of encryption check. I was assuming that b2 sync could just get the encryption type from the bucket but I understand it is written here in documentation 😞
I left this command running for 8 hourss
there must be an upper file limit to this script, but i don't think i should be hitting that.
Also at this point the script cannot be terminated. It's unresponsive
It might be a bit unpleasant in terms of verbosity, but please try to run it again this time prefixing the command with strace . This will track and display system calls and I bet one of them is going to get stuck when the program will try to list a directory or something like that. We need the last few lines of the output.
Sorry for a delayed response, but I was able to determine that my memory was bad and producing errors. This could have caused problems with the b2 command, and explains my storage array issues as well. I replaced the memory with some ECC RAM, and now I'm trying the b2 sync again with strace. If it works flawlessly, I'll go ahead and close this Issue, otherwise, I'll see if I can post any more information to help debug
I was able to upload files successfully this time. It was an issue of memory. sorry for the misunderstanding!
@ppolewicz so I did actually run into this again, but since my memory issues, the command ran a LOT longer before erroring. I'm able to reproduce with strace
munmap(0x7a7aa9d00000, 1048576) = 0
munmap(0x7a7aa9e00000, 1048576) = 0
munmap(0x7a7aaa000000, 1048576) = 0
munmap(0x7a7aaa100000, 1048576) = 0
munmap(0x7a7aaa200000, 1048576) = 0
munmap(0x7a7aaa300000, 1048576) = 0
munmap(0x7a7aaa400000, 1048576) = 0
munmap(0x7a7aaa500000, 1048576) = 0
munmap(0x7a7aaa600000, 1048576) = 0
munmap(0x7a7aaa700000, 1048576) = 0
munmap(0x7a7aaaa00000, 1048576) = 0
munmap(0x7a7aaab00000, 1048576) = 0
munmap(0x7a7aaad00000, 1048576) = 0
munmap(0x7a7aaae00000, 1048576) = 0
munmap(0x7a7aab000000, 1048576) = 0
munmap(0x7a7aab100000, 1048576) = 0
munmap(0x7a7aab200000, 1048576) = 0
munmap(0x7a7aab400000, 1048576) = 0
munmap(0x7a7aab600000, 1048576) = 0
munmap(0x7a7aab700000, 1048576) = 0
munmap(0x7a7aab800000, 1048576) = 0
munmap(0x7a7aab900000, 1048576) = 0
munmap(0x7a7aaba00000, 1048576) = 0
munmap(0x7a7aabb00000, 1048576) = 0
munmap(0x7a7a94000000, 1048576) = 0
write(1, " "..., 71) = 71
munmap(0x7a7a93000000, 1048576) = 0
munmap(0x7a7a93e00000, 1048576) = 0
futex(0x31076970, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb47b80, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb47b88, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7a79f4000b70, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0
futex(0xb47b84, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=611156, tv_nsec=125136668}, FUTEX_BITSET_MATCH_ANY) = 0
futex(0xb47b88, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0xb47b80, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb47b88, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7a7a30000b70, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0
write(1, " "..., 71) = 71
getpid() = 1462799
newfstatat(AT_FDCWD, "/home/herman/.venv/lib/python3.12/site-packages/b2/_internal/console_tool.py", {st_mode=S_IFREG|0664, st_size=196936, ...}, 0) = 0
openat(AT_FDCWD, "/home/herman/.venv/lib/python3.12/site-packages/b2/_internal/console_tool.py", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=196936, ...}) = 0
ioctl(3, TCGETS, 0x7ffe43e230a0) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
read(3, "#!/usr/bin/env python3\n# PYTHON_"..., 4096) = 4096
read(3, "rt validated_loads\nfrom b2._inte"..., 8192) = 8192
read(3, "if mode == EncryptionMode.NONE:\n"..., 8192) = 8192
read(3, " parser, '--source-server-sid"..., 8192) = 8192
read(3, "_argument(\n parser, '"..., 8192) = 8192
read(3, "s_registry.values():\n "..., 8192) = 8192
read(3, "or details on how the location o"..., 8192) = 8192
read(3, " '--metadata-dire"..., 8192) = 8192
read(3, "e_prefix=args.name_prefix,\n "..., 8192) = 8192
read(3, "sMixin}\n {SourceSseMixin}\n "..., 8192) = 8192
read(3, "r = file_name_completer\n "..., 8192) = 8192
read(3, ":\n yield from self.ap"..., 8192) = 8192
read(3, "_event = threading.Event()\n "..., 8192) = 8192
read(3, "actions that would be taken.\n\n "..., 8192) = 8192
read(3, "ne,\n metavar='TIMESTA"..., 8192) = 8192
read(3, "\"\"\n\n @classmethod\n def _se"..., 8192) = 8192
read(3, "le_id, int),\n buf"..., 8192) = 8192
read(3, "gs.retain_until\n "..., 8192) = 8192
read(3, "lities:\n\n - **readReplication"..., 8192) = 8192
read(3, "userpath/master/LICENSE.txt',\n "..., 8192) = 8192
read(3, "Described):\n \"\"\"\n .. warni"..., 8192) = 8192
read(3, "=rule)\n return 0\n\n\n@Bucke"..., 8192) = 8192
read(3, "thorize'\n\n\[email protected]_"..., 8192) = 8192
read(3, " replaced_by_cmd = (File, FileHi"..., 8192) = 8192
read(3, "d not issubclass(\n "..., 8192) = 4424
read(3, "", 8192) = 0
close(3) = 0
newfstatat(AT_FDCWD, "/home/herman/.venv/lib/python3.12/site-packages/b2sdk/_internal/sync/sync.py", {st_mode=S_IFREG|0664, st_size=14597, ...}, 0) = 0
openat(AT_FDCWD, "/home/herman/.venv/lib/python3.12/site-packages/b2sdk/_internal/sync/sync.py", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=14597, ...}) = 0
ioctl(3, TCGETS, 0x7ffe43e230a0) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
read(3, "################################"..., 4096) = 4096
read(3, "ting which determines handling f"..., 8192) = 8192
read(3, "tal(1)\n reporter."..., 8192) = 2309
read(3, "", 8192) = 0
close(3) = 0
write(2, "ERROR:b2._internal.console_tool:"..., 803ERROR:b2._internal.console_tool:ConsoleTool command error
Traceback (most recent call last):
File "/home/herman/.venv/lib/python3.12/site-packages/b2/_internal/console_tool.py", line 5456, in run_command
return command.run(args)
^^^^^^^^^^^^^^^^^
File "/home/herman/.venv/lib/python3.12/site-packages/b2/_internal/console_tool.py", line 1082, in run
return self._run(args)
^^^^^^^^^^^^^^^
File "/home/herman/.venv/lib/python3.12/site-packages/b2/_internal/console_tool.py", line 3204, in _run
synchronizer.sync_folders(
File "/home/herman/.venv/lib/python3.12/site-packages/b2sdk/_internal/sync/sync.py", line 233, in sync_folders
raise IncompleteSync('sync is incomplete')
b2sdk._internal.sync.exception.IncompleteSync: Incomplete sync: sync is incomplete
) = 803
write(2, "ERROR: Incomplete sync: sync is "..., 43ERROR: Incomplete sync: sync is incomplete
) = 43
getpid() = 1462799
write(2, "INFO:b2._internal.console_tool:\\"..., 128INFO:b2._internal.console_tool:\\ ======================================== exit=1 ======================================== //
) = 128
exit_group(1) = ?
+++ exited with 1 +++