disable all stores option

[adriansev]

Hi! It would be really useful a command to invalidate all stores (including local one) not only the remote ones.

[TimoWilken]

Hi Adrian! There are already some similar features in alibuild, would any of the following suffice for your use case?

1. If you only want to rebuild a few specific packages, you can add force_rebuild: true to their YAML frontmatter and they will be rebuilt always, even if they were built before.
2. If you want to essentially start over and not reuse any previously built packages, you could use aliBuild build's --work-dir option to start from scratch in another, new "sw" directory.
3. aliBuild clean --aggressive (I think) cleans up sw/$architecture/store, which contains the tarballs you'd get from the remote store. It leaves installed packages alone, though, only removing the tarballs.

On 19 April 2022 09:25:00 CEST, Adrian Sevcenco @.***> wrote:

Hi! It would be really useful a command to invalidate all stores (including local one) not only the remote ones.

-- Reply to this email directly or view it on GitHub: https://github.com/alisw/alibuild/issues/755 You are receiving this because you are subscribed to this thread.

Message ID: @.***>

[adriansev]

Hi @TimoWilken ! ok, now that i tried again aliBuild clean --aggressive i remembered why i switch to manual deletion of TARS content: the clean procedure just delete the TARS/$(alibuild architecture)/store but all directories with invalid symlinks remain .. maybe it would be more efficient to just delete ${ALIBUILD_WORK_DIR}/TARS/$(alibuild architecture) ?

but this would delete ALL archives, and one could have production/analysis builds (when one do analysis) and also, testing dev things when one just want to have a clean rebuild.

1. is a no go because that would involve private branches of alidist
2. is cumbersome to keep switching the ALIBUILD_WORK_DIR variable
3. see my first point

maybe it would be useful not only to me to have a build option --no-store that would skip any already found tars (local or remote). this would have the advantage of being per build

moreover, if one gets the order of packages, for a given {current dir, -z} tuple, one could build up to package of interest, then build the desired package with --no-store

and now that i mentioned above: is it possible the aliDoctor to present the exact order of building? should i make another issue for this? Thanks a lot!

[TimoWilken]

ok, now that i tried again aliBuild clean --aggressive i remembered why i switch to manual deletion of TARS content: the clean procedure just delete the TARS/$(alibuild architecture)/store but all directories with invalid symlinks remain

The invalid symlinks are harmless. They are ignored by alibuild for the purposes of finding a tarball to reuse. You shouldn't have to remove them manually.

moreover, if one gets the order of packages, for a given {current dir, -z} tuple, one could build up to package of interest, then build the desired package with --no-store

With this workflow, couldn't you build up to (but not including) the package of interest using the remote store, and then build the package itself with --no-remote-store? Then nothing for that package will be fetched.

Locally, I'm not sure just ignoring the tarballs makes sense -- when we build a package, we install it under $ALIBUILD_WORK_DIR/$(aliBuild architecture)/$package_name/ and reuse that instead of the tarball in later builds -- so just deleting the tarball doesn't mean you'd rebuild the package. For that, you'd have to use force_rebuild: true in the alidist recipe.

Perhaps what you'd need is a command-line switch with the same effect as specifying force_rebuild: true in a recipe? So that you could do, for instance, aliBuild build --defaults o2 --force-rebuild O2,O2Physics O2Physics.

and now that i mentioned above: is it possible the aliDoctor to present the exact order of building?

When #754 is merged, you could do, for example:

aliBuild build --plugin templating -a slc7_x86-64 --defaults o2 O2 2>/dev/null <<\EOF
{% for pkg in build_order -%}
  {{ pkg }}
{% endfor %}
EOF

Though I'm not sure the build order is stable across subsequent invocations of aliBuild, especially when not building the same packages both times.

[adriansev]

ok, now that i tried again aliBuild clean --aggressive i remembered why i switch to manual deletion of TARS content: the clean procedure just delete the TARS/$(alibuild architecture)/store but all directories with invalid symlinks remain

The invalid symlinks are harmless. They are ignored by alibuild for the purposes of finding a tarball to reuse. You shouldn't have to remove them manually.

ok, got it

moreover, if one gets the order of packages, for a given {current dir, -z} tuple, one could build up to package of interest, then build the desired package with --no-store

With this workflow, couldn't you build up to (but not including) the package of interest using the remote store, and then build the package itself with --no-remote-store? Then nothing for that package will be fetched.

actually i thought the same, but i had the surprise of the build being taken from local cache (forgot to delete the TARS content) this is why i thought that a command that would disable ANY store (local or remote) could be handy .. but given the length of this conversation it's already clear that this feature request is not considered worthy of development time, which is fine...

Locally, I'm not sure just ignoring the tarballs makes sense -- when we build a package, we install it under $ALIBUILD_WORK_DIR/$(aliBuild architecture)/$package_name/ and reuse that instead of the tarball in later builds -- so just deleting the tarball doesn't mean you'd rebuild the package. For that, you'd have to use force_rebuild: true in the alidist recipe.

actually it will rebuild the package .. if i delete xjalienfs tar and i rebuild xjalienfs (or xrootd) not development package, just the upstream versions, it WILL rebuild the package with the latest found recipe (as opposed to being taken from local store as already built)

Perhaps what you'd need is a command-line switch with the same effect as specifying force_rebuild: true in a recipe? So that you could do, for instance, aliBuild build --defaults o2 --force-rebuild O2,O2Physics O2Physics.

YES! actually this is what i actually need: a way to rebuilt a specific package (or more)

and now that i mentioned above: is it possible the aliDoctor to present the exact order of building?

When #754 is merged, you could do, for example:

aliBuild build --plugin templating -a slc7_x86-64 --defaults o2 O2 2>/dev/null <<\EOF
{% for pkg in build_order -%}
  {{ pkg }}
{% endfor %}
EOF

Though I'm not sure the build order is stable across subsequent invocations of aliBuild, especially when not building the same packages both times.

oh, i see, i thought that the order is deterministic Many thanks!!

[TimoWilken]

actually it will rebuild the package .. if i delete xjalienfs tar and i rebuild xjalienfs (or xrootd) not development package, just the upstream versions, it WILL rebuild the package with the latest found recipe (as opposed to being taken from local store as already built)

Hm, this sounds like a bug. Did you delete the matching symlink too, or just the tarball under sw/TARS/$arch/store?

YES! actually this is what i actually need: a way to rebuilt a specific package (or more)

Great! Could you test #756 to see whether it does the right thing for you?

[adriansev]

actually it will rebuild the package .. if i delete xjalienfs tar and i rebuild xjalienfs (or xrootd) not development package, just the upstream versions, it WILL rebuild the package with the latest found recipe (as opposed to being taken from local store as already built)

Hm, this sounds like a bug. Did you delete the matching symlink too, or just the tarball under sw/TARS/$arch/store?

i do not think so .. i delete the complete content of TAR (well, not anymore with the new feature) and i use --no-remote-store so, the package that is requested to be built and it have not previous version stored then it will be rebuilt.

YES! actually this is what i actually need: a way to rebuilt a specific package (or more)

Great! Could you test #756 to see whether it does the right thing for you?

yup, it's perfect!! thanks a lot!!