Not able to run Zephyr 7B Gemma with 4 80GB A100s

[TJ-Solergibert]

I'm not able to run Zephyr 7B Gemma with 4 80GB A100s. I get the following error:

RuntimeError: The size of tensor a (0) must match the size of tensor b (24576) at non-singleton dimension 1

After running:

ACCELERATE_LOG_LEVEL=info accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_sft.py recipes/zephyr-7b-gemma/sft/config_full.yaml

As can be seen, I've just modified num_processes and I tested zero3_init_flag: false

compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
  deepspeed_multinode_launcher: standard
  offload_optimizer_device: none
  offload_param_device: none
  zero3_init_flag: false
  zero3_save_16bit_model: true
  zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

I've seen this related issue, (#57), but none of the solutions work.

Hope we find a solution soon for the members of the 4 GPU cluster club! 🤗

[TJ-Solergibert]

I've just find out that it works IF YOU INSTALL the dependencies as point 1 of this post. I've run the following to set up the environment:

pip install "torch==2.1.2" tensorboard
python -m pip install .
pip uninstall transformer-engine # I got errors, I'm working with A100s
pip install  --upgrade \
  "transformers==4.38.2" \
  "datasets==2.16.1" \
  "accelerate==0.26.1" \
  "evaluate==0.4.1" \
  "bitsandbytes==0.42.0" \
  "trl==0.7.11" \
  "peft==0.8.2"

pip install ninja packaging
MAX_JOBS=4 pip install flash-attn --no-build-isolation --upgrade

And the complete list of dependencies:

absl-py                   2.0.0
accelerate                0.26.1
aiohttp                   3.8.5
aiosignal                 1.3.1
alignment-handbook        0.4.0.dev0
annotated-types           0.5.0
apex                      0.1
appdirs                   1.4.4
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
asttokens                 2.4.0
astunparse                1.6.3
async-timeout             4.0.3
attrs                     23.1.0
audioread                 3.0.1
backcall                  0.2.0
beautifulsoup4            4.12.2
bitsandbytes              0.42.0
bleach                    6.0.0
blis                      0.7.11
cachetools                5.3.1
catalogue                 2.0.10
certifi                   2023.7.22
cffi                      1.16.0
charset-normalizer        3.2.0
click                     8.1.6
cloudpathlib              0.15.1
cloudpickle               2.2.1
cmake                     3.27.6
comm                      0.1.4
confection                0.1.3
contourpy                 1.1.1
cubinlinker               0.3.0+2.gce0680b
cuda-python               12.2.0rc5+5.g84845d1
cudf                      23.8.0
cugraph                   23.8.0
cugraph-dgl               23.8.0
cugraph-service-client    23.8.0
cugraph-service-server    23.8.0
cuml                      23.8.0
cupy-cuda12x              12.1.0
cycler                    0.12.1
cymem                     2.0.8
Cython                    3.0.3
dask                      2023.7.1
dask-cuda                 23.8.0
dask-cudf                 23.8.0
datasets                  2.16.1
debugpy                   1.8.0
decorator                 5.1.1
deepspeed                 0.12.2
defusedxml                0.7.1
dill                      0.3.7
distributed               2023.7.1
dm-tree                   0.1.8
docker-pycreds            0.4.0
docstring-parser          0.15
einops                    0.7.0
evaluate                  0.4.1
exceptiongroup            1.1.3
execnet                   2.0.2
executing                 2.0.0
expecttest                0.1.3
fastjsonschema            2.18.1
fastrlock                 0.8.1
filelock                  3.12.4
flash-attn                2.5.6
fonttools                 4.43.1
frozenlist                1.4.0
fsspec                    2023.6.0
gast                      0.5.4
gitdb                     4.0.11
GitPython                 3.1.40
google-auth               2.23.2
google-auth-oauthlib      0.4.6
graphsurgeon              0.4.6
grpcio                    1.59.0
hf_transfer               0.1.6
hjson                     3.1.0
huggingface-hub           0.21.4
hypothesis                5.35.1
idna                      3.4
importlib-metadata        6.8.0
iniconfig                 2.0.0
intel-openmp              2021.4.0
ipykernel                 6.25.2
ipython                   8.16.1
ipython-genutils          0.2.0
ipywidgets                8.1.1
jedi                      0.19.1
Jinja2                    3.1.2
joblib                    1.3.2
json5                     0.9.14
jsonschema                4.19.1
jsonschema-specifications 2023.7.1
jupyter                   1.0.0
jupyter_client            8.3.1
jupyter-console           6.6.3
jupyter_core              5.3.2
jupyter-tensorboard       0.2.0
jupyterlab                2.3.2
jupyterlab-pygments       0.2.2
jupyterlab-server         1.2.0
jupyterlab-widgets        3.0.9
jupytext                  1.15.2
kiwisolver                1.4.5
langcodes                 3.3.0
librosa                   0.9.2
lit                       17.0.6
llvmlite                  0.40.1
locket                    1.0.0
Markdown                  3.4.4
markdown-it-py            3.0.0
MarkupSafe                2.1.3
matplotlib                3.8.0
matplotlib-inline         0.1.6
mdit-py-plugins           0.4.0
mdurl                     0.1.2
mistune                   3.0.2
mkl                       2021.1.1
mkl-devel                 2021.1.1
mkl-include               2021.1.1
mock                      5.1.0
mpmath                    1.3.0
msgpack                   1.0.5
multidict                 6.0.4
multiprocess              0.70.15
munch                     4.0.0
murmurhash                1.0.10
nbclient                  0.8.0
nbconvert                 7.9.2
nbformat                  5.9.2
nest-asyncio              1.5.8
networkx                  2.6.3
ninja                     1.11.1.1
notebook                  6.4.10
numba                     0.57.1+1.g5fba9aa8f
numpy                     1.26.4
nvfuser                   0.0.20+gitunknown
nvidia-cublas-cu11        11.10.3.66
nvidia-cublas-cu12        12.1.3.1
nvidia-cuda-cupti-cu11    11.7.101
nvidia-cuda-cupti-cu12    12.1.105
nvidia-cuda-nvrtc-cu11    11.7.99
nvidia-cuda-nvrtc-cu12    12.1.105
nvidia-cuda-runtime-cu11  11.7.99
nvidia-cuda-runtime-cu12  12.1.105
nvidia-cudnn-cu11         8.5.0.96
nvidia-cudnn-cu12         8.9.2.26
nvidia-cufft-cu11         10.9.0.58
nvidia-cufft-cu12         11.0.2.54
nvidia-curand-cu11        10.2.10.91
nvidia-curand-cu12        10.3.2.106
nvidia-cusolver-cu11      11.4.0.1
nvidia-cusolver-cu12      11.4.5.107
nvidia-cusparse-cu11      11.7.4.91
nvidia-cusparse-cu12      12.1.0.106
nvidia-dali-cuda120       1.30.0
nvidia-nccl-cu11          2.14.3
nvidia-nccl-cu12          2.18.1
nvidia-nvjitlink-cu12     12.4.99
nvidia-nvtx-cu11          11.7.91
nvidia-nvtx-cu12          12.1.105
nvidia-pyindex            1.0.9
nvtx                      0.2.5
oauthlib                  3.2.2
onnx                      1.14.0
opencv                    4.7.0
packaging                 23.1
pandas                    1.5.3
pandocfilters             1.5.0
parso                     0.8.3
partd                     1.4.0
pathy                     0.10.2
peft                      0.8.2
pexpect                   4.8.0
pickleshare               0.7.5
Pillow                    9.2.0
pip                       23.3.2
platformdirs              3.11.0
pluggy                    1.3.0
ply                       3.11
polygraphy                0.49.0
pooch                     1.7.0
preshed                   3.0.9
prettytable               3.9.0
prometheus-client         0.17.1
prompt-toolkit            3.0.39
protobuf                  3.20.2
psutil                    5.9.4
ptxcompiler               0.8.1+1.g2cb1b35
ptyprocess                0.7.0
pure-eval                 0.2.2
py-cpuinfo                9.0.0
pyarrow                   11.0.0
pyarrow-hotfix            0.6
pyasn1                    0.5.0
pyasn1-modules            0.3.0
pybind11                  2.11.1
pybind11-global           2.11.1
pycocotools               2.0+nv0.7.3
pycparser                 2.21
pydantic                  1.10.13
pydantic_core             2.10.1
Pygments                  2.16.1
pylibcugraph              23.8.0
pylibcugraphops           23.8.0
pylibraft                 23.8.0
pynvml                    11.4.1
pyparsing                 3.1.1
pytest                    7.4.2
pytest-flakefinder        1.1.0
pytest-rerunfailures      12.0
pytest-shard              0.1.2
pytest-xdist              3.3.1
python-dateutil           2.8.2
python-hostlist           1.23.0
pytorch-quantization      2.1.2
pytz                      2023.3
PyYAML                    6.0.1
pyzmq                     25.1.1
qtconsole                 5.5.1
QtPy                      2.4.1
raft-dask                 23.8.0
referencing               0.30.2
regex                     2023.10.3
requests                  2.31.0
requests-oauthlib         1.3.1
resampy                   0.4.2
responses                 0.18.0
rich                      13.7.1
rmm                       23.8.0
rpds-py                   0.10.4
rsa                       4.9
safetensors               0.4.2
scikit-learn              1.2.0
scipy                     1.11.1
seaborn                   0.13.1
Send2Trash                1.8.2
sentencepiece             0.1.99
sentry-sdk                1.39.1
setproctitle              1.3.3
setuptools                69.0.3
shtab                     1.7.1
six                       1.16.0
smart-open                6.4.0
smmap                     5.0.1
sortedcontainers          2.4.0
soundfile                 0.12.1
soupsieve                 2.5
spacy                     3.7.1
spacy-legacy              3.0.12
spacy-loggers             1.0.5
sphinx-glpi-theme         0.3
srsly                     2.4.8
stack-data                0.6.3
sympy                     1.12
tabulate                  0.9.0
tbb                       2021.10.0
tblib                     2.0.0
tensorboard               2.9.0
tensorboard-data-server   0.6.1
tensorboard-plugin-wit    1.8.1
tensorrt                  8.6.1
terminado                 0.17.1
thinc                     8.2.1
threadpoolctl             3.2.0
thriftpy2                 0.4.16
tinycss2                  1.2.1
tokenizers                0.15.2
toml                      0.10.2
tomli                     2.0.1
toolz                     0.12.0
torch                     2.1.2
tornado                   6.3.3
tqdm                      4.66.1
traitlets                 5.9.0
transformers              4.38.2
treelite                  3.2.0
treelite-runtime          3.2.0
triton                    2.1.0
trl                       0.7.11
typer                     0.9.0
types-dataclasses         0.6.6
typing_extensions         4.7.1
tyro                      0.7.3
ucx-py                    0.33.0
uff                       0.6.9
urllib3                   1.26.16
wandb                     0.16.1
wasabi                    1.1.2
wcwidth                   0.2.8
weasel                    0.3.2
webencodings              0.5.1
Werkzeug                  3.0.0
wheel                     0.41.2
widgetsnbextension        4.0.9
xdoctest                  1.0.2
xgboost                   1.7.5
xxhash                    3.4.1
yarl                      1.9.2
zict                      3.0.0
zipp                      3.16.2