Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to readme and added application notes #168 #178

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
46 changes: 46 additions & 0 deletions APPLICATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Application Notes

This Application Notes is targeted at both model packagers and application developers and is for information that is not directly relevant to users who are just simply trying to use llamafiles in a standalone manner. Instead it is for developers who want their models to better integrate with other developers models or applications. (Think of this as an informal ad-hoc community standards page)

## Finding Llamafiles

While we do not have a package manager for llamafiles, applications developers are recommended
to search for AI models tagged as `llamafile` in Hugging Face AI repository.
Be sure to display the publishing user or organisation and to sort by trending.

Within a llamafile repository entry in Hugging Face, there may be multiple `*.llamafile` files
to choose from. The current convention to describe each sub entries of llamafiles is to
insert a table in the model card surrounded by html comment start and end marker named `*-provided-files`.

For example a model card for a llamafile should have this section that you can parse:

```markdown
<!-- README_llamafile.md-provided-files start -->
## Provided files

| Name | Quant method | Bits | Size | Max RAM required | Use case |
| ---- | ---- | ---- | ---- | ---- | ----- |
| [phi-2.Q2_K.llamafile](https://huggingface.co/jartine/phi-2-llamafile/blob/main/phi-2.Q2_K.llamafile) | Q2_K | 2 | 1.17 GB| 3.67 GB | smallest, significant quality loss - not recommended for most purposes |
... further llamafile entries here ...

<!-- README_llamafile.md-provided-files end -->
```

## Llamafile Naming Convention

Llamafiles follows the same naming convention as gguf but instead of `.gguf` its `.llamafile`. Consult [gguf naming convention]([https://github.com/ggerganov/ggml/blob/master/docs/gguf.md#specification](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md#gguf-naming-convention)) for specifics.


## Installing A Llamafile And Making It Accessible To Other Local Applications

Llamafiles are designed to be standalone and portable, eliminating the need for a traditional installation. For optimal discovery and integration with local application scripts/programs, we recommend the following search paths:

- **System-wide Paths**:
- `/usr/share/llamafile` (Linux/MacOS/BSD): Ideal for developers creating packages, commonly accessed via package managers like `apt get install` in Debian-based Linux OSes.
- `/opt/llamafile` (Linux/MacOS/BSD): Positioned in the `/opt` directory, suitable for installers downloaded directly from the web.
- `C:\llamafile` (Windows): A direct path for Windows systems.

- **User-specific Path**:
- `~/.llamafile` (Linux/MacOS/BSD): Located in the user's home directory, facilitating user-specific configurations in line with Unix-like conventions.

For applications or scripts referencing the Llamafile path, setting the environment variable `$LLAMAFILE_PATH` to a singular path can enhance configuration simplicity and system consistency.
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,23 @@ chmod +x llava-v1.5-7b-q4.llamafile

**Having trouble? See the "Gotchas" section below.**

## Installing A Llamafile And Making It Accessible To Other Local Applications

Llamafiles are designed to be standalone and portable, eliminating the need for a traditional installation. For optimal discovery and integration with local application scripts/programs, we recommend the following install paths:

- **System-wide Paths**:
- `/opt/llamafile` (Linux/MacOS/BSD)
- `C:\llamafile` (Windows)

- **User-specific Path**:
- `~/.llamafile` (Linux/MacOS/BSD)

- **Additional Search Locations**: These path serves as a reference for applications or scripts that might expect to find the Llamafile here. However, direct installations to this directory are discouraged unless you know what you are doing.
- `/usr/share/llamafile` (Linux/MacOS/BSD)

For applications or scripts referencing the Llamafile path, setting the environment variable `$LLAMAFILE_PATH` to a singular path.


### JSON API Quickstart

When llamafile is started, in addition to hosting a web
Expand Down