slackdump icon indicating copy to clipboard operation
slackdump copied to clipboard
verified publisher icon rusq

Resume-from-failed functionality

Open rusq opened this issue 3 years ago • 5 comments 

  After 17GB of export, my slackdump process died with a 500 response from Slack.  I'd love to be able to start again where it left off.  It was a 500 error, so hopefully it was resolved on their end (and not due to bad data etc). 

2022/12/06 21:32:28 application error: export error: channels: error: failed to dump "dataeng-alerts-high" (CM0BW29LZ): callback error: failed to dump channel:thread CM0BW29LZ:1575031520.121400: slack server error: 500 Internal Server Error

Being able to resume where it died would be huge for someone trying to get a complete workspace export. And it's probably as easy as enumerating everything (into a file) on process start, and removing lines from the file as tasks get completed. On start up, it can see if this file already exists and start with the next line as the first task.

Thoughts?

edit: Thinking about it a bit more - an "if this file already exists, skip it" check before exporting a channel/thread/file might help speed things up dramatically and accomplish almost the same thing. Hmm

Originally posted by @jjcovert in https://github.com/rusq/slackdump/discussions/172

rusq avatar Dec 07 '22 22:12 rusq

Another usecase for stored state: https://github.com/rusq/slackdump/discussions/193

rusq avatar Feb 25 '23 00:02 rusq

I'd use the https://github.com/rusq/slackdump/issues/205 case as well.

G-Harmon avatar Apr 13 '23 09:04 G-Harmon

Hey @G-Harmon, yeah, I'm planning to combine those two in v3.

I haven't decided on the CLI flags yet, but the general idea is the following:

  1. once the "resume" or "continue" flag is given along with the existing ZIP archive or directory; do
  2. If the directory is empty - STOP
  3. Detect the type
  4. Read the existing files and detect the latest message time for each of the channels
  5. Start processing each channel from the respective timestamp.

Meanwhile, you could check the shell script that levygroker has contributed recently: https://github.com/rusq/slackdump/tree/master/contrib/incremental_backup (You can read the whole conversation in #193 )

rusq avatar Apr 13 '23 10:04 rusq

It's been 3 days that I was trying to export the workspace. Single disconnect. And you start it over :pain:

2naive avatar Dec 17 '23 15:12 2naive

Hey @2naive, sorry to hear that. When you say "disconnect" do you mean unstable connection, or slack server error?

rusq avatar Jan 06 '24 22:01 rusq