Add create-layer sub command to create filesystem changeset

[coolljt0725]

This PR use a well tested pkg from github.com/containers/storage(which is forked from docker/docker) from https://github.com/opencontainers/image-spec/pull/280 here is an example:

$ oci-image-tool create-layer
One or two filesystems are required
Usage:
  oci-image-tool create-layer [child] [parent] [flags]

Flags:
  -h, --help   help for create-layer

$ tree rootfs-1
rootfs-1
├── bin
│   ├── my-app
│   └── my-tool
└── etc
    └── my-tool-cfg

2 directories, 3 files
[lei@fedora test]$ oci-image-tool create-layer rootfs-1
[lei@fedora test]$ ls
rootfs-1  rootfs-1.tar
[lei@fedora test]$ vim rootfs-1.tar
" tar.vim version v29
" Browsing tarfile /home/lei/test/rootfs-1.tar
" Select a file with cursor and press ENTER

bin/
bin/my-app
bin/my-tool
etc/
etc/my-tool-cfg
~

[lei@fedora test]$ cp -par rootfs-1 rootfs-1-s
[lei@fedora test]$ rm rootfs-1-s/bin/my-tool
[lei@fedora test]$ touch rootfs-1-s/bin/my-tool-1
[lei@fedora test]$ mkdir rootfs-1-s/etc/my-app-cfg.d
[lei@fedora test]$ touch rootfs-1-s/etc/my-app-cfg.d/cfg
[lei@fedora test]$ tree rootfs-1-s
rootfs-1-s
├── bin
│   ├── my-app
│   └── my-tool-1
└── etc
    ├── my-app-cfg.d
    │   └── cfg
    └── my-tool-cfg

3 directories, 4 files

[lei@fedora test]$ oci-image-tool create-layer rootfs-1-s/ rootfs-1
[lei@fedora test]$ ls
rootfs-1  rootfs-1-s  rootfs-1-s.tar  rootfs-1.tar
[lei@fedora test]$ vim rootfs-1-s.tar
" tar.vim version v29
" Browsing tarfile /home/lei/test/rootfs-1-s.tar
" Select a file with cursor and press ENTER

bin/
bin/.wh.my-tool
bin/my-tool-1
etc/
etc/my-app-cfg.d/
etc/my-app-cfg.d/cfg

[wking]

On Sat, Sep 17, 2016 at 09:05:40PM -0700, Lei Jitang wrote:

• Run make update-deps to update dependency … M vendor/github.com/opencontainers/runtime-spec/specs-go/config.go (44) M vendor/github.com/opencontainers/runtime-spec/specs-go/state.go (2) M vendor/github.com/opencontainers/runtime-spec/specs-go/version.go (2)

This pulls in the Linux → *Linux change from opencontainers/runtime-spec#502. I think we want to make that bump an explicit step, so we can catch up (e.g. updating runtimeSpec, which currently uses s.Linux without initializing it because it couldn't have been nil before).

[wking]

On Sat, Sep 17, 2016 at 09:05:40PM -0700, Lei Jitang wrote:

This PR use a well tested pkg from github.com/containers/storage(which is forked from docker/docker)

Pulling in containers/storage is much better than pulling in all of Docker. But it's still doing a lot more than we need (just pkg/archive). Maybe the containers folks would be interested in moving that out into a containers/archive repository?

here is an example:

$ oci-image-tool create-layer

Now just oci-create-layer. And you should add an oci-create-layer.1.md to go with your new command.

[coolljt0725]

@wking

This pulls in the Linux → *Linux change from opencontainers/runtime-spec#502. I think we want to make that bump an explicit step, so we can catch up (e.g. updating runtimeSpec, which currently uses s.Linux without initializing it because it couldn't have been nil before).

I just want to avoid updating other packages when run make update-deps to vendor the github.com/containers/storage, I want the commit cb84b39b4cd14eb3a16693334f817c89ccb062de only contains the packages it relay on, so I run make update-deps first, then develop my commit. Maybe there is a way to update a specified package without updating all the packages?

Now just oci-create-layer. And you should add an oci-create-layer.1.md to go with your new command.

This is under working

[wking]

On Sat, Sep 17, 2016 at 11:20:18PM -0700, Lei Jitang wrote:

@wking

This pulls in the Linux → *Linux change from opencontainers/runtime-spec#502. I think we want to make that bump an explicit step, so we can catch up (e.g. updating runtimeSpec, which currently uses s.Linux without initializing it because it couldn't have been nil before).

I just want to avoid updating other packages when run make update-deps to vendor the github.com/containers/storage, I want the commit cb84b39b4cd14eb3a16693334f817c89ccb062de only contains the packages it relay on, so I run make update-deps first, then develop my commit. Maybe there is a way to update a specified package without updating all the packages?

I think you can just roll back the packages you don't want to bump with ‘git checkout HEAD -- vendor/path/to/revert’ and manually resetting portions of glide.lock. See 2b694a6a for an example of me doing that.

[coolljt0725]

@wking thanks :)

[runcom]

@coolljt0725 can't you just require a specific version of a library in glide.yaml? What's the point if having a field named version if we have to git reset and checkout what we don't need..

[wking]

On Sun, Sep 18, 2016 at 12:23:48AM -0700, Antonio Murdaca wrote:

@coolljt0725 can't you just require a specific version of a library in glide.yaml?

Ah, there's a config file for this thing? ;). That's definitely a better approach than what I was doing.

[glestaris]

I am very up for sharing a single unpacking implementation! I am not sure if this implementation should live outside of an opencontainers repository though. I would hope to see the image-tools becoming more of community maintained reference implementation in unpacking single layers.

Also, having to pull the whole containers/storage and its dependencies in image-tools makes me a bit sad as it's way more than what we need.

[vbatts]

On 20/09/16 05:53 -0700, George Lestaris wrote:

I am very up for sharing a single unpacking implementation! I am not sure if this implementation should live outside of an opencontainers repository though. I would hope to see the image-tools becoming more of community maintained reference implementation in unpacking single layers.

This is the intention of having this repo in the opencontainers org.

Also, having to pull the whole containers/storage and its dependencies in image-tools makes me a bit sad as it's way more than what we need.

perhaps there is a way to trim down that vendoring, such that only the package used is pulled in?

[glestaris]

@vbatts let me try to become more specific. Once this PR is accepted we will be using github.com/containers/storage (or a subset of that) to make a changeset (layer). This will lead us in a situation where building a layer is done through a github.com/containers/storage implementation and unpacking a layer is done through the unpack method that currently lives in image/manifest.go.

This is unfortunately asymmetric. So I am worried that the natural next step would be to use github.com/containers/storage to deal with unpacking as well.

I may be overthinking this but it feels strange to see the layer creation be implemented in a different place than the layer unpacking.

[wking]

On Tue, Sep 20, 2016 at 09:53:40AM -0700, George Lestaris wrote:

This is unfortunately asymmetric. So I am worried that the natural next step would be to use github.com/containers/storage to deal with unpacking as well.

I don't think it really matters where the reference implementation lives, as long as we're not pulling in lots of unrelated stuff. The test suites on the other hand, should be defined locally, so we can distinguish between “the tested (un)packer is buggy” and “containers/storage is buggy” when certifying third-party (un)packers.

I may be overthinking this but it feels strange to see the layer creation be implemented in a different place than the layer unpacking.

I agree.

[coolljt0725]

@glestaris I'm agree what you said, I working on implementing the packing without vendoring third-party package

[coolljt0725]

update without vendorring third-party package except logrus, most code are from docker/docker/pkg/archive, it is well tested.

[vbatts]

please rebase and address the golint reportings

[cyphar]

I've been playing with this for a while, it works pretty nice (especially with the CAS PR). I'm going to write a PR after this is merged to use go-mtree.

[zhouhao3]

Can you rebase this pr?