how to / best practices for setting up and tracking a Python project with nix flakes

rutrum, 3 months ago

There’s two ways to create a dev-environment, which is how you should start. Don’t worry about building a project with nix until you get a dev shell with all your (runtime) dependencies. You can do this by creating a shell.nix or use flakes and have an output for devShells.default. You can use mkShell to accomplish this. I would give you a reference config, but I can’t do that at the moment. If you need help with this later reach out and I’ll get something together.

After that you can work on building it. This involves making a derivation that contains the runtime dependencies of your code, as well as the source itself. If its python, this would include, likely, the entirety of your project. This could be a little tricky, but a lot of people package python code, so there’s things in nixpkgs meant for packages python projects. This article might be helpful, or out-dated: nixos.wiki/wiki/Python but it should point to some useful things in nixpkgs to build python projects.

Then, you can focus on containerization, if you really need it. I don’t know how much you know about nix, but building in nix inherently isolates dependencies away from one another, so if you get the process above working you know it is reproducible. But unlike a container, the above build structure only works on a machine with nix. So if you want to distribute your project as a container, while utiltizing your nix build process, you can look into something like dockerTools from nixpkgs: nix.dev/…/building-and-running-docker-images.html

Let me know if you have more questions.

ComradeKhoumrag, 3 months ago

Thanks, I think I finally got it, but I think I’ll need to switch to flakes if I want to be able to install say numpy==1.10.0 instead of latest (is there a way to specify which python package version should be installed in mkShell?)

Does this explanation make sense?

<span style="color:#323232;">{ pkgs ? import <nixpkgs> {} }:
</span><span style="color:#323232;">pkgs.mkShell {
</span><span style="color:#323232;">  packages = with pkgs; [
</span><span style="color:#323232;">    (python310Full.withPackages 
</span><span style="color:#323232;">                        (python-pkgs: with python-pkgs;
</span><span style="color:#323232;">                          [numpy pandas requests]))];
</span><span style="color:#323232;">  #inputsFrom = [ pkgs.python310Full ];
</span><span style="color:#323232;">}
</span>

To derive this ephemeral shell…

use a package manager defaulted from nix packages: {pkgs ? import <nixpkgs> {} }:

Make an ephemeral shell with programs located in the package manager: pkgs.mkShell {

Define the set of installed packages in the ephemeral shell

<span style="color:#323232;"> packages = with pkgs; [
</span><span style="color:#323232;">    (python310Full.withPackages # Include python310 in the set of installed packages
</span><span style="color:#323232;">                        (python-pkgs: with python-pkgs; # not entirely sure what's going on here, is python-pkgs a predefined arg? fails with nativeBuildInputs if this isn't here 
</span><span style="color:#323232;">                          [numpy pandas requests]))]; # the pip packages to include
</span>