-
Notifications
You must be signed in to change notification settings - Fork 247
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature Request: Return all losses with multi-loss methods e.g., VICReg #1422
Comments
Another example with multiple component losses is TiCo https://arxiv.org/abs/2206.10698 |
A third example with multiple component losses is |
Yes, this is a common issue. We currently only return a single loss because it slightly simplifies the code and it allows you to exchange loss functions without having to make other code changes (you don't have to handle aggregation of the different parts). For VICReg you can compute the components individually:
See also #1161 For TiCo and HypersphereLoss we should definitely add the possibility to calculate the components individually. Out of curiosity, when logging the individual parts would you log them with or without the loss weights? For example in VICReg each loss part has a weight ( |
I was previously implementing all this SSL stuff myself for a research project, then discovered Lightly and thought "Awesome! Why reimplement the wheel? I'll just switch to Lightly." I can of course compute the components individually, but I would prefer to either use Lightly or not, rather than mix and match. The pattern I was using that works well is that each loss function should return a dictionary by default, and then one can access the combined loss e.g.:
Without the weights, definitely. |
Some SSL methods have a total loss that is a weighted combination of several component losses. For instance, VICReg is a sum of three loss terms: invariance, variance, covariance.
It would be really nice if all the component losses were returned by the
VICRegLoss
function.The minimal use case is logging/monitoring. For instance, if the VICReg loss becomes unstable, the user can't necessarily tell which of the three component losses is misbehaving.
The text was updated successfully, but these errors were encountered: