Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training time too long #184

Open
newton2149 opened this issue Sep 6, 2023 · 0 comments
Open

Training time too long #184

newton2149 opened this issue Sep 6, 2023 · 0 comments

Comments

@newton2149
Copy link

Loading train data:   6%|█▊                          | 25/395 [01:08<08:03,  1.31s/it]INFO:ljs_base:Train Epoch: 6 [6%]
INFO:ljs_base:[2.675187349319458, 1.905822992324829, 2.1355035305023193, 27.08325958251953, 1.6539193391799927, 1.324233889579773, 2000, 0.0001997751124671936]
DEBUG:matplotlib:matplotlib data path: /opt/conda/envs/vits/lib/python3.11/site-packages/matplotlib/mpl-data
DEBUG:matplotlib:CONFIGDIR=/home/ubuntu/.config/matplotlib
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is linux
INFO:ljs_base:Saving model and optimizer state at iteration 6 to ./logs/ljs_base/G_2000.pth
INFO:ljs_base:Saving model and optimizer state at iteration 6 to ./logs/ljs_base/D_2000.pth
Loading train data:  26%|███████                    | 103/395 [02:50<04:48,  1.01it/s]

I am trying to train with the LJ Speech Dataset in aws ec2 instance.

Using p3.2xlarge instance It is kind of very slow. Is there any kind of optimization I can do?

And while training how much time did it take for you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant