Skip to content

An all-new Language Model That Processes Ultra-Long Sequences of 100,000+ Ultra-Fast

License

Notifications You must be signed in to change notification settings

kyegomez/Andromeda

Repository files navigation

Multi-Modality

Andromeda: Ultra-Fast and Ultra-Intelligent SOTA Language Model 🚀🌌

Open Bounties Rewarded Bounties GitHub issues GitHub forks GitHub stars GitHub license Share on Twitter Share on Facebook Share on LinkedIn Discord Share on Reddit Share on Hacker News Share on Pinterest Share on WhatsApp

Welcome to Andromeda, The Fastest, Most Creative, and Reliable Language Model Ever Built, train your own verison, conduct inference, and finetune your own verison with simple plug in and play scripts get started in 10 seconds:

Features

  • 💼 Handle Ultra Long Sequences (32,000-200,000+ context lengths)
  • ⚡ Ultra Fast Processing (32,000+ tokens in under 100ms)
  • 🎓 Superior Reasoning Capabilities

🎯 Principles

  • Efficiency: Optimize with techniques like attention flashing, rotary position encodings, and deep normalization.
  • Flexibility: Adapt to various tasks and domains for wide applications.
  • Scalability: Designed to scale with resources and data sizes.
  • Community-Driven: Thrives on contributions from the open-source community.

💻 Install

python3.11 -m pip install --upgrade andromeda-torch

Usage

  • Forward pass with random inputs
import torch

from andromeda.configs import Andromeda1Billion

model = Andromeda1Billion()

x = torch.randint(0, 256, (1, 1024)).cuda()

out = model(x)  # (1, 1024, 20000)
print(out)
  • Tokenized inputs
from andromeda_torch import Tokenizer
from andromeda_torch.configs import Andromeda1Billion

model = Andromeda1Billion()
tokenizer = Tokenizer()

encoded_text = tokenizer.encode("Hello world!")
out = model(encoded_text)
print(out)

📚 Training

  1. Set the environment variables:

    • ENTITY_NAME: Your wandb project name
    • OUTPUT_DIR: Directory to save the weights (e.g., ./weights)
    • MASTER_ADDR: For distributed training
    • MASTER_PORT For master port distributed training
    • RANK- Number of nodes services
    • WORLD_SIZE Number of gpus
  2. Configure the training:

    • Accelerate Config
    • Enable Deepspeed 3
    • Accelerate launch train_distributed_accelerate.py

For more information, refer to the Training SOP.


Todo

  • Add Yarn Embeddings from zeta

📈 Benchmarks

Speed

  • Andromeda utilizes one of the most reliable Attentions ever, flash attention 2.0 Triton. It consumes 50x less memory than GPT-3 and 10x less than LLAMA.

AndromedaBanner

  • We can speed this up even more with dynamic sparse flash attention 2.0.

License

Apache License