fleetdm
Method to install Orbit in Nix/NixOS
Goal
Provide a recommended way to get Orbit installed within NixOS (or just using the nix package manager).
Background
Nix is a functional language that is used by NixOS (a Linux Distribution) and nixpkgs (the default package set and repos) to declaratively manage packages and the entire OS.
How?
In a pure NixOS installation, both packages and their configurations are specified in the Nix language. This means that a binary can not be generated from fleetctl and distributed to the node. Rather, there needs to be a Nix expression that knows how to download a package (whether from source and/or compiled) along with configuration passed to it on how to instantiate and configure it.
I'm not sure the best way to even recommend using Orbit on NixOS. Currently, there isn't even a supported version of OSQuery in nixpkgs (the last maintainer stopped updating around 2020 and so they removed it). Using this last known working commit, I am easily able to get 3.3.2 installed (from 2019): https://github.com/NixOS/nixpkgs/blob/f3282c8d1e0ce6ba5d9f6aeddcfad51d879c7a4a/pkgs/tools/system/osquery/default.nix.
My first thought would be to have a generic orbit package that could be built and upon start up must have the type of configuration passed to it that is specific if running go run directly.
Another thing to consider is that since everything is meant to be declaratively specified within Nix, auto updating is generally frowned upon. We'd want to disable that in Fleet nixpkg by default.
I'm willing to do most of the work, working on the actual Nix code, but will need guidance on what direction to take.
I have confirmed that I can get osquery 3.3.2 on NixOS to connect to our fleet. However, none of the queries that I've tried succeed, so I assume that I'll need a new version of osquery (since fleetctl produces orbit installer with osquery 5.4.0).
@senorsmile have you made progress on using orbit with nixOS? Would you like to collaborate on moving this forward? How can I help?
What is being said in that Slack convo? I do not have access.
I am also trying to make Orbit run on a NixOS install, but haven't had any success yet.
@dherder i'm removing this FR from the board for now as we do not have engineering capacity to take it on in the upcoming sprint. if you haven't already, @Patagonia121 can help coordinate collaboration with the customer who offered to work on this
What is being said in that Slack convo? I do not have access.
I am also trying to make Orbit run on a NixOS install, but haven't had any success yet.
@NickBouwhuis we are in the process of collaborating with a nixOS contributor on getting the Fleet agent working on nixOS, please stay tuned for updates; early stages right now. But, if you are looking for telemetry only, it does appear that the osquery nixos package is being updated and is working well: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/tools/system/osquery/info.json
Fwiw, our team at GitLab was able to get a few devices running fleetd. Here's the config repo they're using https://gitlab.com/proglottis/gitlab-team-nix
@jficz and @zhardie This is how I got Fleet working on NixOS based heavily on what gitlab wrote.
https://github.com/AdrielVelazquez/fleet-nixos
The only thing missing from Gitlab was this portion of
orbit-fhs = pkgs.buildFHSEnv {
name = "orbit-fhs";
# Add dependencies needed by the orbit executable AND its children (like sudo)
targetPkgs = pkgs: [
pkgs.stdenv.cc.cc
pkgs.glibc
pkgs.zlib
# Add PAM and sudo to the environment
pkgs.pam
pkgs.sudo
];
runScript = "${cfg.package}/bin/orbit --fleet-desktop=false";
};
And executing it after some chmod changes
serviceConfig = {
# Execute orbit within the FHS environment
ExecStart = "${orbit-fhs}/bin/orbit-fhs";
ExecStartPre = pkgs.writeShellScript "orbit-init" ''
mkdir -p /opt/orbit
cp "${cfg.package}/opt/orbit/certs.pem" \
"${cfg.package}/opt/orbit/osquery.flags" \
"${cfg.package}/opt/orbit/tuf-metadata.json" \
/opt/orbit
chmod 600 /opt/orbit/tuf-metadata.json
'';
Removing customer-montague as they are moving towards standardizing on Ubuntu.
Moved the original issue description here:
Goal
Provide a recommended way to get Orbit installed within NixOS (or just using the nix package manager).
Background
Nix is a functional language that is used by NixOS (a Linux Distribution) and nixpkgs (the default package set and repos) to declaratively manage packages and the entire OS.
How?
In a pure NixOS installation, both packages and their configurations are specified in the Nix language. This means that a binary can not be generated from fleetctl and distributed to the node. Rather, there needs to be a Nix expression that knows how to download a package (whether from source and/or compiled) along with configuration passed to it on how to instantiate and configure it.
I'm not sure the best way to even recommend using Orbit on NixOS. Currently, there isn't even a supported version of OSQuery in nixpkgs (the last maintainer stopped updating around 2020 and so they removed it). Using this last known working commit, I am easily able to get 3.3.2 installed (from 2019): https://github.com/NixOS/nixpkgs/blob/f3282c8d1e0ce6ba5d9f6aeddcfad51d879c7a4a/pkgs/tools/system/osquery/default.nix.
My first thought would be to have a generic orbit package that could be built and upon start up must have the type of configuration passed to it that is specific if running go run directly.
Another thing to consider is that since everything is meant to be declaratively specified within Nix, auto updating is generally frowned upon. We'd want to disable that in Fleet nixpkg by default.
I'm willing to do most of the work, working on the actual Nix code, but will need guidance on what direction to take.
I used @AdrielVelazquez repo as a starting point and wound up turning it into https://github.com/adamcik/fleet-nixos
- Packages are built from source, instead of importing the deb from
fleetctl - All the update features are disabled, it uses system ca certs, and doesn't have any tuf files etc.
- This includes hardcoded config (via existing flags/env) + patches (just adding extra flags/env) to play nice and put things in
/var/lib/orbit+/var/logs/orbitnot/opt - This also includes using the nixos version of osquery without relying on a hardcoded path
- Wraps a bunch of the flags/envs from
.../cmd/orbit.goin options for the service - Makes sure the user service for
fleet-desktophas an identifier to use - Makes sure
xdg-openis available tofleet-desktopand has a "full" user path (personally I had to add~/.local/binfor reasons, so that is hardocded for now). - Lastly I got this working with
nix-updateso it should be trivial to bump new versions. E.g. I saw 1.46.0 just dropped, and the script works fine, I'm just waiting for the right go version to be available in unstable (at least I think that the error I saw)
I've tested this with our self hosted fleet instance at work, and it seems to work just fine (except for our rules all being ubuntu centric for now). Hopefully this is good enough that it could be cleaned up and pushed into nixpkgs, perhaps some of the flags can turn into upstream patches as well. When it comes to packaging in nix I'm learning as I go, so if anyone spots anything that could be improved feedback is welcome.
But that being said, I've managed to achieve what I was aiming for to get things working, at least enough to scratch my itch, and will probably not put any more time into this.
@mason-buettner discussed with dale and taking this off the help-customers project for now since you've done the testing and determined it's not compatible at the moment. we'll come back to this in the future if Fleet provides official support for nix