[Bug]: lora dreambooth reported an error

[baicai99]

What happened?

Yesterday, our system was still running normally, but today when we tried to use it, we encountered an error report, after a preliminary investigation, we suspect that this may be due to some packages installed in the environment caused by dependency conflicts.

Colab cell output

📂 Connecting to Google Drive...
Mounted at /content/drive

💿 Checking dataset...
📁MyDrive/Loras/bangbangv3/dataset
📈 Found 181 images with 10 repeats, equaling 1810 steps.
📉 Divide 1810 steps by 2 batch size to get 905.0 steps per epoch.
🔮 There will be 10 epochs, for around 9050 total training steps.

🏭 Installing dependencies...

Cloning into '/content/kohya-trainer'...
remote: Enumerating objects: 5921, done.
remote: Counting objects: 100% (2786/2786), done.
remote: Compressing objects: 100% (359/359), done.
remote: Total 5921 (delta 2619), reused 2433 (delta 2427), pack-reused 3135
Receiving objects: 100% (5921/5921), 9.37 MiB | 14.79 MiB/s, done.
Resolving deltas: 100% (4207/4207), done.
HEAD is now at 9a67e0d Merge pull request #610 from lubobill1990/patch-1
38 packages can be upgraded. Run 'apt list --upgradable' to see them.
The following additional packages will be installed:
  libaria2-0 libc-ares2
The following NEW packages will be installed:
  aria2 libaria2-0 libc-ares2
0 upgraded, 3 newly installed, 0 to remove and 38 not upgraded.
Need to get 1,513 kB of archives.
After this operation, 5,441 kB of additional disk space will be used.
Selecting previously unselected package libc-ares2:amd64.
(Reading database ... 121752 files and directories currently installed.)
Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.3_amd64.deb ...
Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...
Selecting previously unselected package libaria2-0:amd64.
Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...
Unpacking libaria2-0:amd64 (1.36.0-1) ...
Selecting previously unselected package aria2.
Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...
Unpacking aria2 (1.36.0-1) ...
Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...
Setting up libaria2-0:amd64 (1.36.0-1) ...
Setting up aria2 (1.36.0-1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.4) ...
/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link

Processing /content/kohya-trainer
  Preparing metadata (setup.py) ... done
Collecting accelerate==0.15.0 (from -r requirements.txt (line 1))
  Downloading accelerate-0.15.0-py3-none-any.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 3.0 MB/s eta 0:00:00
Collecting transformers==4.26.0 (from -r requirements.txt (line 2))
  Downloading transformers-4.26.0-py3-none-any.whl (6.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 62.7 MB/s eta 0:00:00
Collecting ftfy==6.1.1 (from -r requirements.txt (line 3))
  Downloading ftfy-6.1.1-py3-none-any.whl (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 7.2 MB/s eta 0:00:00
Requirement already satisfied: albumentations==1.3.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 4)) (1.3.1)
Requirement already satisfied: opencv-python==4.8.0.76 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 5)) (4.8.0.76)
Collecting einops==0.6.0 (from -r requirements.txt (line 6))
  Downloading einops-0.6.0-py3-none-any.whl (41 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.6/41.6 kB 6.1 MB/s eta 0:00:00
Collecting diffusers==0.10.2 (from -r requirements.txt (line 7))
  Downloading diffusers-0.10.2-py3-none-any.whl (503 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 503.1/503.1 kB 49.8 MB/s eta 0:00:00
Collecting pytorch-lightning==1.9.0 (from -r requirements.txt (line 8))
  Downloading pytorch_lightning-1.9.0-py3-none-any.whl (825 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 825.8/825.8 kB 69.0 MB/s eta 0:00:00
Collecting bitsandbytes==0.41.3.post2 (from -r requirements.txt (line 9))
  Downloading bitsandbytes-0.41.3.post2-py3-none-any.whl (92.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.6/92.6 MB 8.0 MB/s eta 0:00:00
Collecting tensorflow==2.14.0 (from -r requirements.txt (line 10))
  Downloading tensorflow-2.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (489.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 489.8/489.8 MB 2.7 MB/s eta 0:00:00
Collecting safetensors==0.4.1 (from -r requirements.txt (line 11))
  Downloading safetensors-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 76.2 MB/s eta 0:00:00
Requirement already satisfied: toml==0.10.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (0.10.2)
Collecting voluptuous==0.13.1 (from -r requirements.txt (line 13))
  Downloading voluptuous-0.13.1-py3-none-any.whl (29 kB)
Collecting lion_pytorch==0.0.6 (from -r requirements.txt (line 14))
  Downloading lion_pytorch-0.0.6-py3-none-any.whl (4.2 kB)
Collecting dadaptation==3.1 (from -r requirements.txt (line 15))
  Downloading dadaptation-3.1.tar.gz (13 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting prodigyopt==1.0 (from -r requirements.txt (line 16))
  Downloading prodigyopt-1.0-py3-none-any.whl (5.5 kB)
Collecting lycoris_lora==0.1.4 (from -r requirements.txt (line 17))
  Downloading lycoris_lora-0.1.4.tar.gz (37 kB)
  Preparing metadata (setup.py) ... done
Collecting xformers==0.0.22.post7 (from -r requirements.txt (line 18))
  Downloading xformers-0.0.22.post7-cp310-cp310-manylinux2014_x86_64.whl (211.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.8/211.8 MB 2.4 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (1.25.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (24.0)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (5.9.5)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (6.0.1)
Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.15.0->-r requirements.txt (line 1)) (2.2.1+cu121)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (3.13.1)
Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (0.20.3)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (2023.12.25)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (2.31.0)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.26.0->-r requirements.txt (line 2))
  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 94.4 MB/s eta 0:00:00
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.26.0->-r requirements.txt (line 2)) (4.66.2)
Requirement already satisfied: wcwidth>=0.2.5 in /usr/local/lib/python3.10/dist-packages (from ftfy==6.1.1->-r requirements.txt (line 3)) (0.2.13)
Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (1.11.4)
Requirement already satisfied: scikit-image>=0.16.1 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (0.19.3)
Requirement already satisfied: qudida>=0.0.4 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (0.0.4)
Requirement already satisfied: opencv-python-headless>=4.1.1 in /usr/local/lib/python3.10/dist-packages (from albumentations==1.3.1->-r requirements.txt (line 4)) (4.9.0.80)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from diffusers==0.10.2->-r requirements.txt (line 7)) (7.0.2)
Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from diffusers==0.10.2->-r requirements.txt (line 7)) (9.4.0)
Requirement already satisfied: fsspec[http]>2021.06.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (2023.6.0)
Collecting torchmetrics>=0.7.0 (from pytorch-lightning==1.9.0->-r requirements.txt (line 8))
  Downloading torchmetrics-1.3.1-py3-none-any.whl (840 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 840.4/840.4 kB 68.2 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (4.10.0)
Collecting lightning-utilities>=0.4.2 (from pytorch-lightning==1.9.0->-r requirements.txt (line 8))
  Downloading lightning_utilities-0.10.1-py3-none-any.whl (24 kB)
Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.4.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.6.3)
Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (24.3.7)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.5.4)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.2.0)
Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.9.0)
Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (16.0.6)
Requirement already satisfied: ml-dtypes==0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.2.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.3.0)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (3.20.3)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (67.7.2)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.16.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (2.4.0)
Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.14.1)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (0.36.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.14.0->-r requirements.txt (line 10)) (1.62.1)
Collecting tensorboard<2.15,>=2.14 (from tensorflow==2.14.0->-r requirements.txt (line 10))
  Downloading tensorboard-2.14.1-py3-none-any.whl (5.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 95.3 MB/s eta 0:00:00
Collecting tensorflow-estimator<2.15,>=2.14.0 (from tensorflow==2.14.0->-r requirements.txt (line 10))
  Downloading tensorflow_estimator-2.14.0-py2.py3-none-any.whl (440 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 440.7/440.7 kB 47.1 MB/s eta 0:00:00
Collecting keras<2.15,>=2.14.0 (from tensorflow==2.14.0->-r requirements.txt (line 10))
  Downloading keras-2.14.0-py3-none-any.whl (1.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 64.0 MB/s eta 0:00:00
Collecting torch>=1.4.0 (from accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading torch-2.1.0-cp310-cp310-manylinux1_x86_64.whl (670.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 670.2/670.2 MB 2.0 MB/s eta 0:00:00
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (1.12)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (3.2.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (3.1.3)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 35.0 MB/s eta 0:00:00
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 54.4 MB/s eta 0:00:00
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 55.2 MB/s eta 0:00:00
Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 2.1 MB/s eta 0:00:00
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 3.0 MB/s eta 0:00:00
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 7.3 MB/s eta 0:00:00
Collecting nvidia-curand-cu12==10.3.2.106 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 11.2 MB/s eta 0:00:00
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 7.6 MB/s eta 0:00:00
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 3.1 MB/s eta 0:00:00
Collecting nvidia-nccl-cu12==2.18.1 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 MB 4.4 MB/s eta 0:00:00
Collecting nvidia-nvtx-cu12==12.1.105 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 16.3 MB/s eta 0:00:00
Collecting triton==2.1.0 (from torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.2/89.2 MB 8.0 MB/s eta 0:00:00
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1))
  Downloading nvidia_nvjitlink_cu12-12.4.99-py3-none-manylinux2014_x86_64.whl (21.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.1/21.1 MB 71.9 MB/s eta 0:00:00
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.43.0)
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (3.9.3)
Requirement already satisfied: scikit-learn>=0.19.1 in /usr/local/lib/python3.10/dist-packages (from qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (1.2.2)
Requirement already satisfied: imageio>=2.4.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (2.31.6)
Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (2024.2.12)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image>=0.16.1->albumentations==1.3.1->-r requirements.txt (line 4)) (1.5.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (2.27.0)
Collecting google-auth-oauthlib<1.1,>=0.5 (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10))
  Downloading google_auth_oauthlib-1.0.0-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.5.2)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.0.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.26.0->-r requirements.txt (line 2)) (2024.2.2)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->diffusers==0.10.2->-r requirements.txt (line 7)) (3.17.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0->-r requirements.txt (line 8)) (4.0.3)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (5.3.3)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (1.4.0)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.19.1->qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.19.1->qudida>=0.0.4->albumentations==1.3.1->-r requirements.txt (line 4)) (3.3.0)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (2.1.5)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.4.0->accelerate==0.15.0->-r requirements.txt (line 1)) (1.3.0)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (0.5.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow==2.14.0->-r requirements.txt (line 10)) (3.2.2)
Building wheels for collected packages: dadaptation, lycoris_lora, library
  Building wheel for dadaptation (pyproject.toml) ... done
  Created wheel for dadaptation: filename=dadaptation-3.1-py3-none-any.whl size=22788 sha256=711cb41973d95c7ce0e4bd199fcb74b3dec8cff2c9eb399bcc1d56ce581e7dc2
  Stored in directory: /root/.cache/pip/wheels/34/25/92/b7708738900a69a166f41de56c776039b5a05599b36075c34f
  Building wheel for lycoris_lora (setup.py) ... done
  Created wheel for lycoris_lora: filename=lycoris_lora-0.1.4-py3-none-any.whl size=26134 sha256=7f3f78d978836dd99f155c2acd8f963970f9e5022cf1026ec4f169f727f98edd
  Stored in directory: /root/.cache/pip/wheels/65/d4/14/c7a61ec6ce0398272767c355bd02af1aa309cecd7102b22080
  Building wheel for library (setup.py) ... done
  Created wheel for library: filename=library-0.0.0-py3-none-any.whl size=84081 sha256=6955abd0aed633ab95d29096622dee016fa7b133014b2663ea3403cba9c8134d
  Stored in directory: /root/.cache/pip/wheels/d8/ef/ac/94db7d75dc3a8e81533bc2d001d79467c693ec69ed02bbee59
Successfully built dadaptation lycoris_lora library
Installing collected packages: voluptuous, tokenizers, library, bitsandbytes, triton, tensorflow-estimator, safetensors, prodigyopt, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, lightning-utilities, keras, ftfy, einops, dadaptation, nvidia-cusparse-cu12, nvidia-cudnn-cu12, transformers, nvidia-cusolver-cu12, google-auth-oauthlib, diffusers, torch, tensorboard, xformers, torchmetrics, tensorflow, lycoris_lora, lion_pytorch, accelerate, pytorch-lightning
  Attempting uninstall: tokenizers
    Found existing installation: tokenizers 0.15.2
    Uninstalling tokenizers-0.15.2:
      Successfully uninstalled tokenizers-0.15.2
  Attempting uninstall: triton
    Found existing installation: triton 2.2.0
    Uninstalling triton-2.2.0:
      Successfully uninstalled triton-2.2.0
  Attempting uninstall: tensorflow-estimator
    Found existing installation: tensorflow-estimator 2.15.0
    Uninstalling tensorflow-estimator-2.15.0:
      Successfully uninstalled tensorflow-estimator-2.15.0
  Attempting uninstall: safetensors
    Found existing installation: safetensors 0.4.2
    Uninstalling safetensors-0.4.2:
      Successfully uninstalled safetensors-0.4.2
  Attempting uninstall: keras
    Found existing installation: keras 2.15.0
    Uninstalling keras-2.15.0:
      Successfully uninstalled keras-2.15.0
  Attempting uninstall: transformers
    Found existing installation: transformers 4.38.2
    Uninstalling transformers-4.38.2:
      Successfully uninstalled transformers-4.38.2
  Attempting uninstall: google-auth-oauthlib
    Found existing installation: google-auth-oauthlib 1.2.0
    Uninstalling google-auth-oauthlib-1.2.0:
      Successfully uninstalled google-auth-oauthlib-1.2.0
  Attempting uninstall: torch
    Found existing installation: torch 2.2.1+cu121
    Uninstalling torch-2.2.1+cu121:
      Successfully uninstalled torch-2.2.1+cu121
  Attempting uninstall: tensorboard
    Found existing installation: tensorboard 2.15.2
    Uninstalling tensorboard-2.15.2:
      Successfully uninstalled tensorboard-2.15.2
  Attempting uninstall: tensorflow
    Found existing installation: tensorflow 2.15.0
    Uninstalling tensorflow-2.15.0:
      Successfully uninstalled tensorflow-2.15.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tf-keras 2.15.1 requires tensorflow<2.16,>=2.15, but you have tensorflow 2.14.0 which is incompatible.
torchaudio 2.2.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.
torchtext 0.17.1 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.
torchvision 0.17.1+cu121 requires torch==2.2.1, but you have torch 2.1.0 which is incompatible.
Successfully installed accelerate-0.15.0 bitsandbytes-0.41.3.post2 dadaptation-3.1 diffusers-0.10.2 einops-0.6.0 ftfy-6.1.1 google-auth-oauthlib-1.0.0 keras-2.14.0 library-0.0.0 lightning-utilities-0.10.1 lion_pytorch-0.0.6 lycoris_lora-0.1.4 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.18.1 nvidia-nvjitlink-cu12-12.4.99 nvidia-nvtx-cu12-12.1.105 prodigyopt-1.0 pytorch-lightning-1.9.0 safetensors-0.4.1 tensorboard-2.14.1 tensorflow-2.14.0 tensorflow-estimator-2.14.0 tokenizers-0.13.3 torch-2.1.0 torchmetrics-1.3.1 transformers-4.26.0 triton-2.1.0 voluptuous-0.13.1 xformers-0.0.22.post7

✅ Installation finished in 291 seconds.

🔄 Downloading model...

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
0c3920|OK  |   153MiB/s|//content/animefull-final-pruned-fp16.safetensors

Status Legend:
(OK):download completed.


📄 Config saved to /content/drive/MyDrive/Loras/bangbangv3/training_config.toml
📄 Dataset config saved to /content/drive/MyDrive/Loras/bangbangv3/dataset_config.toml

⭐ Starting trainer...

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /content/kohya-trainer/train_network.py:17 in <module>                                           │
│                                                                                                  │
│    14 from accelerate.utils import set_seed                                                      │
│    15 from diffusers import DDPMScheduler                                                        │
│    16                                                                                            │
│ ❱  17 import library.train_util as train_util                                                    │
│    18 from library.train_util import (                                                           │
│    19 │   DreamBoothDataset,                                                                     │
│    20 )                                                                                          │
│                                                                                                  │
│ /content/kohya-trainer/library/train_util.py:36 in <module>                                      │
│                                                                                                  │
│     33 import torch                                                                              │
│     34 from torch.nn.parallel import DistributedDataParallel as DDP                              │
│     35 from torch.optim import Optimizer                                                         │
│ ❱   36 from torchvision import transforms                                                        │
│     37 from transformers import CLIPTokenizer                                                    │
│     38 import transformers                                                                       │
│     39 import diffusers                                                                          │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torchvision/__init__.py:6 in <module>                    │
│                                                                                                  │
│     3 from modulefinder import Module                                                            │
│     4                                                                                            │
│     5 import torch                                                                               │
│ ❱   6 from torchvision import _meta_registrations, datasets, io, models, ops, transforms, util   │
│     7                                                                                            │
│     8 from .extension import _HAS_OPS                                                            │
│     9                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torchvision/_meta_registrations.py:164 in <module>       │
│                                                                                                  │
│   161                                                                                            │
│   162                                                                                            │
│   163 @torch._custom_ops.impl_abstract("torchvision::nms")                                       │
│ ❱ 164 def meta_nms(dets, scores, iou_threshold):                                                 │
│   165 │   torch._check(dets.dim() == 2, lambda: f"boxes should be a 2d tensor, got {dets.dim()   │
│   166 │   torch._check(dets.size(1) == 4, lambda: f"boxes should have 4 elements in dimension    │
│   167 │   torch._check(scores.dim() == 1, lambda: f"scores should be a 1d tensor, got {scores.   │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_ops.py:253 in inner                        │
│                                                                                                  │
│   250 │   """                                                                                    │
│   251 │                                                                                          │
│   252 │   def inner(func):                                                                       │
│ ❱ 253 │   │   custom_op = _find_custom_op(qualname, also_check_torch_library=True)               │
│   254 │   │   custom_op.impl_abstract(_stacklevel=3)(func)                                       │
│   255 │   │   return func                                                                        │
│   256                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1076 in _find_custom_op         │
│                                                                                                  │
│   1073 │   │   raise RuntimeError(                                                               │
│   1074 │   │   │   f"Could not find custom op \"{qualname}\". Did you register it via "          │
│   1075 │   │   │   f"the torch._custom_ops API?")                                                │
│ ❱ 1076 │   overload = get_op(qualname)                                                           │
│   1077 │   result = custom_op_from_existing(overload)                                            │
│   1078 │   return result                                                                         │
│   1079                                                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1062 in get_op                  │
│                                                                                                  │
│   1059 │   │   error_not_found()                                                                 │
│   1060 │   opnamespace = getattr(torch.ops, ns)                                                  │
│   1061 │   if not hasattr(opnamespace, name):                                                    │
│ ❱ 1062 │   │   error_not_found()                                                                 │
│   1063 │   packet = getattr(opnamespace, name)                                                   │
│   1064 │   if not hasattr(packet, 'default'):                                                    │
│   1065 │   │   error_not_found()                                                                 │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/_custom_op/impl.py:1052 in error_not_found         │
│                                                                                                  │
│   1049                                                                                           │
│   1050 def get_op(qualname):                                                                     │
│   1051 │   def error_not_found():                                                                │
│ ❱ 1052 │   │   raise ValueError(                                                                 │
│   1053 │   │   │   f"Could not find the operator {qualname}. Please make sure you have "         │
│   1054 │   │   │   f"already registered the operator and (if registered from C++) "              │
│   1055 │   │   │   f"loaded it via torch.ops.load_library.")                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Could not find the operator torchvision::nms. Please make sure you have already 
registered the operator and (if registered from C++) loaded it via torch.ops.load_library.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/bin/accelerate:8 in <module>                                                          │
│                                                                                                  │
│   5 from accelerate.commands.accelerate_cli import main                                          │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py:45 in main         │
│                                                                                                  │
│   42 │   │   exit(1)                                                                             │
│   43 │                                                                                           │
│   44 │   # Run                                                                                   │
│ ❱ 45 │   args.func(args)                                                                         │
│   46                                                                                             │
│   47                                                                                             │
│   48 if __name__ == "__main__":                                                                  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:1104 in launch_command     │
│                                                                                                  │
│   1101 │   elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA  │
│   1102 │   │   sagemaker_launcher(defaults, args)                                                │
│   1103 │   else:                                                                                 │
│ ❱ 1104 │   │   simple_launcher(args)                                                             │
│   1105                                                                                           │
│   1106                                                                                           │
│   1107 def main():                                                                               │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:567 in simple_launcher     │
│                                                                                                  │
│    564 │   process = subprocess.Popen(cmd, env=current_env)                                      │
│    565 │   process.wait()                                                                        │
│    566 │   if process.returncode != 0:                                                           │
│ ❱  567 │   │   raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)       │
│    568                                                                                           │
│    569                                                                                           │
│    570 def multi_gpu_launcher(args):                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/usr/bin/python3', 'train_network.py', 
'--dataset_config=/content/drive/MyDrive/Loras/bangbangv3/dataset_config.toml', 
'--config_file=/content/drive/MyDrive/Loras/bangbangv3/training_config.toml']' returned non-zero 
exit status 1.

Which colab and model(s) were you using when the error occurred?

https://github.com/camenduru/stable-diffusion-webui-colab/tree/training

Which Public WebUI Colab URL were you using when the error occurred?

remote.moe

If you used HiRes mode when the error occurred, please provide the Hires info

No response