Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create nix flake with default package and dev shell #285

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ivandimitrov8080
Copy link

This adds flake support for Nix

@EricLBuehler EricLBuehler added documentation Improvements or additions to documentation new feature New feature or request labels May 10, 2024
Copy link
Owner

@EricLBuehler EricLBuehler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding this! This removes the necessity of building with cargo manually if all you want is a quickstart, right? It seems super cool.

I just wanted to confirm that we have a way to let the user specify the feature flags (cuda, etc) and also ensure that we are building in --release for maximum optimizations.

**Running the new Phi 3 model with 128K context window**

`cargo run --release --features ... -- -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3`

`nix run github:EricLBuehler/mistral.rs -- -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3`
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we specify features and ensure that there is a release build here?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nix is a tool for reproducible builds and deployments. You define a build and it does the same thing every single time. Here's what the build command looks like now:

cargo build --release --message-format json-render-diagnostics --locked

It still uses cargo under the hood to compile the project.

This specific flake uses ipetkov/crane to build this. Here's what the crane library provides as build overrides.

This means that for each feature you could have a different package definition with a different cargoExtraArgs field specifying the feature set and a default package enabling all the features.

You don't have the option to specify build arguments through the command line. This is something nix users have learned to live with.

python3
pkg-config
openssl
];
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EricLBuehler Essentially, if you add cargoExtraArgs = "--features ..." inside the buildPackage function, you'd be enabling the features you want. buildInputs specifies runtime dependencies, checkPhaseCargoCommand runs the tests (it's empty because this gets built in an isolated environment without internet access for reproducibility and the tests want to download stuff), pname specifies the package name and the default binary name for nix run, and src specifies the source code.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explaining! Is there a way for this cargoExtraArgs parameter to be set when the user runs the command? I think it is important to allow the cuda/metal/etc feature activation with this.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EricLBuehler The only way that this can be achieved is by providing a different package definition for each feature so that the user can choose which package they want to build. This can be done with recursion over a list of features with the only difference between the packages being the feature set. I'm not that good at nix to know how to do it right off the bat but it seems like a cool problem that I might try if you give me a list of features.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EricLBuehler Usually what is done is a function is provided in the nix language where you input the feature set and build the package based on that. This is easier to implement but it doesn't solve the problem of having that accessible through the command line since it's nix.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ivandimitrov8080, can you make this work by having different nix flake files? I'm not familiar with nix though.

The features which I think are important to expose are:

  • cuda
  • metal
  • mkl
  • accelerate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation new feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants