-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
create nix flake with default package and dev shell #285
base: master
Are you sure you want to change the base?
create nix flake with default package and dev shell #285
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for adding this! This removes the necessity of building with cargo
manually if all you want is a quickstart, right? It seems super cool.
I just wanted to confirm that we have a way to let the user specify the feature flags (cuda, etc) and also ensure that we are building in --release
for maximum optimizations.
**Running the new Phi 3 model with 128K context window** | ||
|
||
`cargo run --release --features ... -- -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3` | ||
|
||
`nix run github:EricLBuehler/mistral.rs -- -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we specify features and ensure that there is a release build here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nix is a tool for reproducible builds and deployments. You define a build and it does the same thing every single time. Here's what the build command looks like now:
cargo build --release --message-format json-render-diagnostics --locked
It still uses cargo under the hood to compile the project.
This specific flake uses ipetkov/crane to build this. Here's what the crane library provides as build overrides.
This means that for each feature you could have a different package definition with a different cargoExtraArgs
field specifying the feature set and a default package enabling all the features.
You don't have the option to specify build arguments through the command line. This is something nix users have learned to live with.
python3 | ||
pkg-config | ||
openssl | ||
]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@EricLBuehler Essentially, if you add cargoExtraArgs = "--features ..."
inside the buildPackage
function, you'd be enabling the features you want. buildInputs
specifies runtime dependencies, checkPhaseCargoCommand
runs the tests (it's empty because this gets built in an isolated environment without internet access for reproducibility and the tests want to download stuff), pname
specifies the package name and the default binary name for nix run
, and src
specifies the source code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for explaining! Is there a way for this cargoExtraArgs
parameter to be set when the user runs the command? I think it is important to allow the cuda/metal/etc feature activation with this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@EricLBuehler The only way that this can be achieved is by providing a different package definition for each feature so that the user can choose which package they want to build. This can be done with recursion over a list of features with the only difference between the packages being the feature set. I'm not that good at nix to know how to do it right off the bat but it seems like a cool problem that I might try if you give me a list of features.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@EricLBuehler Usually what is done is a function is provided in the nix language where you input the feature set and build the package based on that. This is easier to implement but it doesn't solve the problem of having that accessible through the command line since it's nix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ivandimitrov8080, can you make this work by having different nix flake files? I'm not familiar with nix though.
The features which I think are important to expose are:
- cuda
- metal
- mkl
- accelerate
This adds flake support for Nix