Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[LFX Contribution]: Integrating MLX framework as a WasmEdge NN Backend #3330

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

guptaaryan16
Copy link

#3266

Thanks for giving me such an important feature to work on, and I have been working on the issue for a while and feels like it can be taken for an initial review now.

Now my work is divided into two parts:

  1. Implementing a CPP inference Example for MLX using the current CPP API:
    I have implemented a simple NN with the current CPP API of MLX, keeping in mind the needs for the plugin and the general reference to other projects like PyTorch C++ Fronted API.
    Currently the API supports creating layers like BatchNorm, transformers among others and is able to perform basic inference . I am working on more classes and models and will try to complete them as soon as possible.
    Current features:
  • Creating model classes for the LLMs and other NNs
  • Get parameters listed in a Hashmap and print them, Also working on loading weights from a SafeTensors file
  • Completed some classes based on the nn::Module class and working on more

For implementation, you can check: https://github.com/guptaaryan16/mlx/blob/Cpp_api/examples/cpp/nn_inference_example.cpp
https://github.com/guptaaryan16/mlx/blob/Cpp_api/examples/cpp/attention.cpp

  1. Implementing mlx.cpp and other files for wasi-nn plugin
    I have been studying the implementation of other plugins, especially ggml and pytorch, and it has helped me to implement some functionality like loading weights from SafeTensors and GGUF format for now.
    Most of the functionality further will be dependent on the implementation of the CPP frontend for the MLX library, which I plan to now complete as soon as possible.

My Notes and Info About Completed Milestones

Currently, I have faced multiple problems due my current knowledge in C++ and low level design for the CPP based NN ( for loading LLMs) and thus I have not been able to move as fast as I have listed in my milestones. Still I believe that after a solid baseline implementation of a NN class(which is almost done now) and further a LLAMA model, I will be able to complete the plugin as soon as possible.
From now on, I will be using the current implemented NN example to create something similar to llama.cpp for MLX (will try to keep it within the library itself to simplify loading of models) and thus will complete more parts for the plugin within a few weeks.

Thank you for helping me till now and hope that I complete the project as soon as possible.

cc @hydai @awni

Copy link
Member

juntao commented Apr 8, 2024

Hello, I am a code review bot on flows.network. Here are my reviews of code commits in this PR.


@github-actions github-actions bot added c-Plugin An issue related to WasmEdge Plugin c-WASI-NN labels Apr 8, 2024
@guptaaryan16 guptaaryan16 marked this pull request as draft April 8, 2024 15:05
@bmorphism
Copy link

wow, this is pretty brutal @juntao -- respect!

@guptaaryan16 guptaaryan16 force-pushed the wasi-nn-mlx branch 2 times, most recently from ecf74e4 to 84da6d7 Compare May 22, 2024 12:18
Signed-off-by: Aryan Gupta <guptaaryan16@gmail.com>
Signed-off-by: Aryan Gupta <guptaaryan16@gmail.com>
Signed-off-by: Aryan Gupta <guptaaryan16@gmail.com>
Signed-off-by: Aryan Gupta <guptaaryan16@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c-CMake c-Plugin An issue related to WasmEdge Plugin c-WASI-NN
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants