Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support get optimizer and lr_schedulers from deepspeed config #19860

Open
npuichigo opened this issue May 11, 2024 · 0 comments
Open

Support get optimizer and lr_schedulers from deepspeed config #19860

npuichigo opened this issue May 11, 2024 · 0 comments
Labels
feature Is an improvement or enhancement needs triage Waiting to be triaged by maintainers

Comments

@npuichigo
Copy link

npuichigo commented May 11, 2024

Description & Motivation

Currently it's not supported. However in huggingface/accelerate, users can specify DummyOptimizer and DummyScheduler to indicate they want them to be replaced by those from deepspeed config.

Pitch

Lightning should have some way to let people use optimizers from deepspeed config, like OneBitAdam or something else that cannot be manually instantiated.

self._deepspeed_engine, optimizer = self._initialize_engine(module, optimizers[0])

Alternatives

No response

Additional context

No response

cc @Borda

@npuichigo npuichigo added feature Is an improvement or enhancement needs triage Waiting to be triaged by maintainers labels May 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Is an improvement or enhancement needs triage Waiting to be triaged by maintainers
Projects
None yet
Development

No branches or pull requests

1 participant