NixOS
machine configs ignore overlay
I've defined a nixpkgs overlay but nixops isn't picking it up.
I'm using the method where I set NIX_PATH / -I to MYDIR where MYDIR contains MYDIR/nixpkgs-overlays/default.nix with contents like these. Because nix searches in $NIX_PATH/nixpkgs-overlays/default.nix by default, that overlay is picked up.
So when I pkgs = import <nixpkgs> {};, that pkgs contains my overrides.
But the pkgs passed to machine configs like machine1 = { pkgs, config, ... }: ... do NOT get passed my overrides, they get passed an un-overridden nixpkgs.
Why is that?
I can work around it by setting
machine1 = { pkgs, config, ... }: {
nixpkgs.config.packageOverrides = pkgs: (import <nixpkgs> {});
...
}
but it seems overlays are supposed to work without doing this manually for every machine.
It seems I can fix it by changing this https://github.com/NixOS/nixops/blob/b7d549143451c9cfae6644db89a495de67df1407/nix/eval-machine-info.nix#L35-L66
to have
{ name = machineName;
value = import <nixpkgs/nixos/lib/eval-config.nix> {
pkgs = import <nixpkgs> {}; # new line addded here
modules =
modules ++
...
But not sure if that makes any issues. The relevant code for eval-config.nix doesn't even seem to know if this pkgs argument should be removed:
, # !!! is this argument needed any more? The pkgs argument can
# be set modularly anyway.
pkgs ? null
@nh2
are you using overlays? They seem to not work for me for nixops unless I patch this: https://github.com/NixOS/nixops/issues/893
@cleverca22
i believe that is a nixos issue, not a nixops issue
you must set nixpkgs.overlays = [ (import ./overlay1.nix) ]; in the nixos config
@nh2
I must set that somewhere or that should be set somewhere in nixpkgs?
@cleverca22
you have to set it within the config for a machine in nixops
{
machine1 = { ... }: {
deployment.targetEnv = "ec2";
nixpkgs.overlays = [ (import ./overlay1.nix) ];
};
}
@nh2
Hmm. I had hoped that using overlays I could add overrides for my entire nixpkgs, so that is not the case and there's a separation between "nixops-evaluation-host-side pkgs" and "machine-host-side pkgs" in regards to overlays?
@cleverca22
there is also a special default machine in nixops that applies to everything
{
default = { ... }: {
nixpkgs.overlays = [ (import ./overlay1.nix) ];
};
}
now every machine in the deployment will have that overlay
@nh2
you mean defaults with s?
(at least that's what I'm using)
@cleverca22
ah, yeah, it says defaults in my netbook deployment
@nh2
hmm OK. But why is it like this, is this desired? I'm thinking of a use case, why it shouldn't automatically take my overlays
purity, nixos should use the config/overlays defined in the nixos config, not whatever is in $HOME of the current user
the code responsible for loading the pkgs arg in nixos is at https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/misc/nixpkgs.nix
anything assigned to _module.args is passed to every module in the OS eval
line 64 is where the default nixpkgs comes from
ah, the pkgs argument you found in eval-config overrides that, causing both nixpkgs.config and nixpkgs.overlays to do nothing
but setting it under defaults.nixpkgs.overlays allows it to compose against machine1.nixpkgs.overlays, so you still have more flexible changes
@nh2
I still find the argumentation slightly confusing though:
When it comes to the evaluation of import <nixpkgs>, that nixpkgs is taken from NIX_PATH.
In https://nixos.org/nixpkgs/manual/#sec-overlays-install it says
If the
overlaysargument is not provided explicitly, we look for overlays in a path. The path is determined as follows: ... Otherwise, if the Nix path entry<nixpkgs-overlays>exists, we look for overlays at that path
So in my case it picks <nixpkgs> from NIX_PATH but not <nixpkgs-overlays> -- that seems to be against the above description.
@cleverca22
nixops may also be messing with NIX_PATH during the eval, try inserting builtins.trace (builtins.toJSON builtins.nixPath) "foo" into the codepath somewhere so you can see what the nixPath is
@nh2
OK one sec
@cleverca22
builtins.nixPath will contain the result of merging NIX_PATH with all the -I flags
and the "foo" can be removed, since trace will return whatever is given to it, like in haskell
and we have no IO monad, so it has to be something nixops will want to read
@nh2
it seems to not mess with it in this case, builtins.nixPath is the same both on the top-level as well as inside the machine config, namely in my case
[{"path":"/nix/store/z7g5gqdl9lv2zj35ylqg0jnhf29ijsxs-nixops-1.5.2pre0_abcdef/share/nix/nixops","prefix":"nixops"},{"path":".","prefix":""},{"path":"../../nix-channel","prefix":""},{"path":"/nix/store/b4s1gxiis1ryvybnjhdjvgc5sr1nq0ys-nix-1.11.15/share/nix/corepkgs","prefix":"nix"}]
@cleverca22
ok, so it will search for <nixpkgs-overlays> in the current directory and ../../nix-channel/nixpkgs-overlays
@nh2
yes that is right
@cleverca22
and does it exist in the current directory?
@nh2
no, only in ../../nix-channel
@cleverca22
ah, you have a channel called nixpkgs-overlays? ls -lh ../../nix-channel/{,nixpkgs-overlays}
@nh2
I just called the dir that I put in NIX_PATH "nix-channel"
@cleverca22
ah what does the above ls say?
@nh2
% ls -lah nix-channel
total 16K
drwxrwxr-x 4 niklas niklas 4.0K Mar 4 01:03 ./
drwxr-xr-x 37 niklas niklas 4.0K Mar 3 23:56 ../
drwxrwxr-x 8 niklas niklas 4.0K Mar 3 15:14 nixpkgs/
drwxrwxr-x 2 niklas niklas 4.0K Mar 4 01:03 nixpkgs-overlays/
% ls nix-channel/nixpkgs-overlays
default.nix
@cleverca22
https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/impure.nix#L53
ok, so all files matching `*.nix` and all directories matching `*/default.nix` will be passed to import, to build up the overlay list
that should work
@nh2
@cleverca what would happen if we changed nixops as I proposed in that issue, and then somebody set nixpkgs.overlays = [ ]; inside a machine config? Would that remove the overlays loaded from NIX_PATH I added with my change?
Because if yes, maybe then what I propose there is a better default: They would be there by default as described in the nixpkgs manual, and if somebody wanted to NOT have those auto-added-from-NIX_PATH overlays on one of their machines, they could set nixpkgs.overlays = []
@cleverca22
nixpkgs would entirely ignore all config the users set at nixpkgs.overlays and nixpkgs.config causing confusing problems and the result would depend on the environment, rather then what is defined in the nix files
@nh2
but right now doesn't the result also depend on the environment, given that it picks <nixpkgs> from NIX_PATH?
@cleverca22
thats a much more expected thing, that can be overriden with nixops modify deployment.nix -I nixpkgs=https://....
@nh2
nixpkgs would entirely ignore all config the users set at nixpkgs.overlays
hmm you're right, I just tried it and if I do what I said, then setting nixpkgs.overlays = [ ]; has no effect
@cleverca22
you could even nixpkgs.overlays = throw "foo"; i think
since its ignored to that level
@nh2
no that throws though
@cleverca22
ah, but that stack trace may say why
@nh2
maybe it does evaluate it but concatenate the results, so it's what I added ++ []?
@nh2 (uploaded uploads file) and commented: that's when I do nixpkgs.overlays = throw "foo"; while having pkgs = import <nixpkgs> {}; set in eval-machine-info.nix
@cleverca22
ahh, i think i see why
@nh2
yeah I think it's because of https://github.com/NixOS/nixpkgs/blob/1dcd022f01b251b1656f349dcf749c0890de2799/nixos/modules/misc/nixpkgs.nix#L78
@nh2
type = types.listOf overlayType; so default concatenates
@cleverca22
its evaling both versions of pkgs, from the normal place, and your override, then its passing both sets of pkgs to the mkMerge system
@nh2
right
@cleverca22
but it wont concat the overlays
its the pkgs that are being merged
and ...
https://github.com/NixOS/nixpkgs/blob/4eb7945515b260efa55e44e783f4b4936de9f6a4/nixos/lib/eval-config.nix#L40
if you supply a pkgs the way you said, it will be an mkForce
so after it evals both, it throws out the one that obeys nixpkgs.overlays
if you modify a package within it with a second overlay, you should find that it doesnt take effect
@nh2
wait, I don't follow. Why wouldn't it concat the overlays? Given that overlays = mkOption and type = types.listOf overlayType;, shouldn't the options merging code think "it's a list, the default merge operations for lists is concatenation"?
@cleverca22
it doesnt merge <nixpkgs-overlays> and nixpkgs.overlays
it creates 2 entirely seperate instances of nixpkgs, one from each
misc/nixpkgs.nix line 64 will import the 1st nixpkgs, which obeys nixpkgs.overlays and nixpkgs.config, and its at the default priority, because its in the default key of mkOption
@nh2
ah hmm
@cleverca22
lib/eval-config.nix line 40, then sets the same _module.args.pkgs option with mkForce
so the 1st nixpkgs is entirely thrown out
@nh2
and if we set nixpkgs.overlays = [ ... some overlay import logic as described in the nixpkgs manual ... ] among these lines
https://github.com/NixOS/nixops/blob/b7d549143451c9cfae6644db89a495de67df1407/nix/eval-machine-info.nix#L54-L60
would that work and make sense?
so that it's an actual option that would get merged with concat, and could be overridden by the user for a machine with mkForce?
@cleverca22
there is no way to remove those, and mkForce removes all other versions, so i cant concat between all of my own files
so i either have to live with whatever you added, or put all of my own overlays in a single file
@nh2
hmm OK, that doesn't sound good
@cleverca22
for example, i started this nixos module: https://github.com/cleverca22/nixos-configs/blob/e534f8706fae4f6178194418dab2cf1944a972eb/qemu.nix that will add its own overlay, which pulls in some extra packages it needs and i plan to have several modules like that, each with their own overlay
@nh2
then I think best is to add a section to the nixops manual that explains this, that only nixpkgs is taken from NIX_PATH for the machines, and that you should define nixpkgs.overlays accordingly per machine if you want the overlays to apply
@cleverca22
the same is true for nixos itself, so maybe add it to the nixos manual, and have a note in nixops pointing to that?
@nh2
Yes sounds good. The section in the nixpkgs manual hits a bit at it with> On a NixOS system the value of the nixpkgs.overlays option, if present, is passed to the system Nixpkgs directly as an argument. Note that this does not affect the overlays for non-NixOS operations (e.g. nix-env), which are looked up independently. but I think it could be clearer.
So the summary and remainder for this issue is to improve docs in nixops and nixpkgs accordingly.
Namely, you have to give nixpkgs.overlays = [ ... ] explicitly for your nixops machine (or do it in defaults if you want it for all machines in a network).
I'm still new to NixOS and nixops, and I'm trying to run a customized version of zfs with nixops. I found this bug report, but I'm still getting an error about
nixops deploy -d zfs
building all machine configurations...
error: attribute 'extend' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/boot/kernel.nix:39:31
(use '--show-trace' to show detailed location information)
error: unable to build all machine configurations
when I deploy.
Here's the "logical" part of my nixops config:
{
network.description = "zfs test";
zfstest =
{ config, pkgs, ... }:
{
boot.zfs.enableUnstable = true;
boot.supportedFilesystems = [ "zfs" ];
boot.kernelPackages = pkgs.linuxPackages_4_15;
nixpkgs.overlays = [(
self: super:
{
zfsUnstable = super.zfsUnstable.overrideAttrs (oldAttrs : {
src = super.fetchFromGitHub {
owner = "rbrewer123";
repo = "zfs";
#rev = "b4555c777a0be3c0dba29662d278c57099c60a87";
#sha256 = "03k1maiz70jbk45619yqcf3larhvgilaxnvkid3972knm7ycgjsh";
rev = "de4a0c31b5d269e39114181465ddbba448a4db3f";
sha256 = "0ghpn9lsjdpx2qh4wk48vdd22sa1qv4hw49m0p4lv6ap5h14bx0v";
};
});
linuxPackages_4_15.zfsUnstable = super.linuxPackages_4_15.zfsUnstable.overrideAttrs (oldAttrs : {
src = super.fetchFromGitHub {
owner = "rbrewer123";
repo = "zfs";
#rev = "b4555c777a0be3c0dba29662d278c57099c60a87";
#sha256 = "03k1maiz70jbk45619yqcf3larhvgilaxnvkid3972knm7ycgjsh";
rev = "de4a0c31b5d269e39114181465ddbba448a4db3f";
sha256 = "0ghpn9lsjdpx2qh4wk48vdd22sa1qv4hw49m0p4lv6ap5h14bx0v";
};
});
}
)];
};
}
Using those overlays alone, I'm able to build the userspace and kernel space portions of zfs with nix build. But when I put them in nixops as shown it doesn't work. Do you have any ideas what's wrong?
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/can-we-add-integrations-to-discourse/156/1
