intel
Reduce size of Simics CI images or use another way to boot/modify images
Currently Simics CI images are 20Gb in size. It takes around half an hour to scp it to computers connected by VPN, which is a recommended way to modify the image (scp to the developer's machine and run in VirtualBox).
Either images should be reduced in size or non-gui tools like Simics(prefferably), qemu, KVM, vagrant or other libvirt-based tools should be used to boot or modify images.
Both actions (reducing an image and switching to another tool) are preferable than one of them.
in my case much faster than half an hour, please try does my method work for you: I copy image to my office host (not remote host over VPN), then I run VirtualBox on office host, redirecting UI over X connection to home (thats over VPN). That way, image copy happens over fast link, and GUI is over VPN, that is usable for me at least. But overall, yes, if we can reduce/avoid copying, would be good idea. As I explained before, I have had little success with qemu (idea: use it without GUI and scripted, use libvirt tools etc) thats why I went to path using VirtualBox
I'm not sure I have working X on my office desktop. My client machine is Mac, so I need to find out how to do it. Probably would need to install X Server or VNC client. Not sure I want to spend time on this.
Looking at the image content I'd say that most of the packages there are not needed for production CI usage. Ubuntu 20.04 docker image takes 72.8Mb of space. I understand that we may need more than that, but 1-2 Gb would be enough for our needs, I believe.
Regarding the tool to use for updating the image, I'd prefer to use Simics if that's possible. This would be consistent with production usage. Simics has plenty of ways to do that: https://compas.cs.stonybrook.edu/~mabbasidinan/simics-user-guide-unix/topic31.html
Simics has plenty of ways to do that
good pointer, thanks, yes I agree simics method is a good candidate. Before I learned about those (already some time back) I tried to solve few pain points: I eliminated need to copy image to user's land, but retained Vbox part. Every simics instance currently has set of scripts to 1) copy image to vbox-runner 2) start vbox session in vbox runner (one will have ssh access) 3) copy image back to simics runner. But this sub-system is in limbo state because my initial plan was to run vbox sessions in one of VM hosts, and I later discovered I can not run vbox and qemu at same time on same host, and that VM host got used as qemu VMs host when I expanded the system, so ATM there is no host to run those vbox sessions (unless I stop temporarily some of instances). Also, as we did not have change needs for some time already, I was thinking is it all worth running a whole subsystem with dedicated server and config. Small changes which I recently made, can be carried without booting, by loop-mounting.
Looking at the image content I'd say that most of the packages there are not needed
Thats very true. When first simics instances started, the initial image was a full GUI-based system running graphical desktop. I manually stripped it down, removed some packages and made it running without starting GUI runlevel processes. It also had big local docker images present, I think I removed some. But there is still significant amount of unneeded content. The cleaner way for CI would be to build such image from empty and add only relevant part. But in current case, we have relied on provided images which are runnable in simics.
I think I should try out image modifying-saving idea using simics, and revisit my base method (described 2 comments above) so that I replace VBox part with simics part. That would simplify it a lot and eliminate need for extra server.
I'd try docker save before Simics. It should be more straightforward. There are plenty of minimal base images on the dockerhub for any linux distro.
I am not sure I follow now. How can docker save help to maintain existing simics base image?
It can help to create it I guess. That's what you're going to do as far as I understood.
No I haven't mentioned using docker, nor planned. Creating a bootable image has been out of my reach so far, there is EFI partition which I dont know where it comes from or how to create. So far I have always relied on images provided by OZ. I tried however to boot such image in qemu but always failed, as none of tried bios variants worked for me. VBox OTOH (recommended by OZ) "just boots" it. So I am not sure I can easily reach a working combination with my own creation. But Oleg may know better.
ok, I probably misunderstood the task. I was in the impression that you want to build a root filesystem from scratch or from a minimal base image.
are you willing to work on this
yes, I plan to take a look at those simics-based methods after we get k8s (and possibly other) upgrades done in bmaas. BTW, this issue could fit better in bmaas repo than here?
@okartau - Great, could you create a new issue for the bmaas project and let me know and I can then close this one?