Shipit-deploy: copy previous release

[jonaaix]

When deploying with shipit staging deploy, i can see in terminal logs:

Copy previous release to "/home/xy/projectX/releases/20181127210354"
Running "cp -a /home/xy/projectX/releases/20181125015850/. /home/xy/projectX/releases/20181127210354" on host "dummy.com".
Copy project to remote servers.
"copy" method is deprecated, please use "copyToRemote", "copyFromRemote", "scpCopyToRemote" or "scpCopyFromRemote". It will break in v5.0.0.
Copy "/Users/xy/projectX/dist/" to "[email protected]:/home/xy/projectX/releases/20181127210354" via rsync

Copy previous release

Can you explain what it is used for? This task takes a lot of time, although it is not needed becuase new files were pulled/copied from local, and i can't see any of the copied files in the new directory. So I guess they were overwritten afterwards by my new files from rsyncFrom.

[gregberge]

Copying files locally (on the remote) is faster than transferring all files from the local to the remote.

So we assume that between two releases there is not a lot of changes. By coping previous release, the rsync will have less files to transfer. It results in a faster deploy process.

[jonaaix]

@neoziro Okay, and the process automatically detects which files have changed and just upload them?

[gregberge]

Yes it is

[jonaaix]

Great to know that. Does this check happen before copying from the previous version? I suspect that the node_modules folder is copied and removed afterwards, because it will never be synced from the client. Checking the files before copying would increase the speed a lot.

[gregberge]

@btxtiger oh yes probably, maybe we could optimize it!

[fTrestour]

Hi! This behaviour actually seems to create some issues when rsync is not installed on the machine (as explained in this issue: https://github.com/shipitjs/shipit-deploy/issues/137)

I just ran into this problem while renaming a file to use typescript (changed a level.js to a level.ts) When the CI container ran the deploy command (which didn't have rsync), the javascript file was not deleted and was then used instead of the typescript file.

Is anything planned to fix this unexpected behaviour?

[gregberge]

@fTrestour nothing planned, however I am open for PR.

[jonaaix]

But there seems to be a fallback to scp implemented in the code shipit/packages/ssh-pool/src/Connection.js:375

@neoziro I'm currently searching for a way to copy only the required files. I noticed there is a parameter ignores that is used for remoteCopy(). Can't we use this array as filter for copyPreviousRelease() ? I think, the final result should remain the same, doesn't it? However this would mean, that we need to use rsync instead cp on remote for copyPreviousRelease() to include the ignore list

[jonaaix]

This works for me:

async function copyPreviousRelease() {
  // const copyParameter = shipit.config.copy || '-a'
  if (!shipit.previousRelease || shipit.config.copy === false) return
  shipit.log('Copy previous release to "%s"', shipit.releasePath)

  const formatRsyncCommand = (options) => {
    const excludes = options.excludes.reduce(
      (args, current) => [...args, '--exclude', `"${current}"`],
      [],
    ).join(' ');

    return `rsync --recursive ${options.src} ${options.dest} ${excludes}`
  };
  await shipit.remote(
    formatRsyncCommand(
      {
        src: path.join(shipit.releasesPath, shipit.previousRelease),
        dest: shipit.releasePath,
        excludes: shipit.config.ignores || []
      }
    )

    // util.format(
    //   'cp %s %s/. %s',
    //   copyParameter,
    //   path.join(shipit.releasesPath, shipit.previousRelease),
    //   shipit.releasePath,
    // ),
  )
}

However the progress should be disabled, because the stdout crashes if too much files were copied (I included node_modules for testing). This way copyPreviousRelease() is extremely fast on my remote server, less than a second. I'm curious about testing it on a raspberry server tomorrow, where this step took 30-60 seconds previously.

[Edo78]

Instead of modifying the code you should set shipit.config.copy to false this way you achieve the same goal without messing with the code.

[std4453]

Instead of modifying the code you should set shipit.config.copy to false this way you achieve the same goal without messing with the code.

This should be in the docs. Great help.

[Edo78]

This should be in the docs. Great help.

Glad to be useful. Sadly it looks like shipit is no more under active development. Last commit more than a year ago. You can write a PR and hope.