Which miner to use for ProxyAnchor (and ProxyNCA) loss?

[AabyWan]

Greetings, Kevin (and the PML community),

First, thank you for contributing to the field and making this framework available. Your work has made my first steps into metric learning much more accessible and made me move from Keras to Torch earlier than I thought possible. Many thanks!

Moving to the question, which may be relatively simple, which miner would you use together with the ProxyAnchor loss?
Similarly, if I wanted to compare between ProxyAnchor and ProxyNCA, would I swap the loss function?

I took the following approach, which does work programmatically, but I'm curious if I am following the original work or making up a new approach 😇.

distance = distances.CosineSimilarity()
reducer = reducers.DivisorReducer()

loss_fn = losses.ProxyAnchorLoss(
    num_classes=N_CLASSES, 
    embedding_size=N_EMBEDDINGS, 
    margin=MARGIN, 
    alpha=ALPHA,
    distance=distance,
    reducer=reducer)

# original paper scale learning rate by 100 for proxies
loss_fn_optim = torch.optim.AdamW(loss_fn.parameters(), lr=LR*100)    

miner = miners.TripletMarginMiner(
    margin=MARGIN, 
    distance=distance, 
    type_of_triplets="semihard")

I then pass that to my trainer function below:

def train(model, miner, dataloader, optimizer, loss_fn, device, loss_fn_optim=None):
    # make sure the model is in train mode 
    model.train()
    running_loss = 0.0

    for i, batch in enumerate(dataloader):
        inputs, labels = batch[0].to(device), batch[1].to(device)
        optimizer.zero_grad()
        outputs = model(inputs)

        indices_tuple = miner(outputs, labels)
        loss = loss_fn(outputs, labels, indices_tuple)
        running_loss += loss.item()
        
        loss.backward()
        optimizer.step()

        # if the loss func req an optim, such as ProxyAnchorLoss.
        if loss_fn_optim : loss_fn_optim.step()
    
    return running_loss

[lemuelatQSA]

I may be wrong but... what's the point of using a miner on proxy-based loss? You are not comparing data points against each other, you're comparing data points to class proxies

[KevinMusgrave]

This library does something unorthodox when using a miner with a classification loss. It weights each sample's loss by the frequency that it appears in the mined pairs.

@AabyWan If you want to follow the original papers, do not use a miner with ProxyAnchor or ProxyNCA.

[lemuelatQSA]

Maybe we can update the documentation specifying this behaviour (if it's not already there)

[KevinMusgrave]

I agree. I have an old issue reminding me to do this: #202 😄

[AabyWan]

Brilliant, I have updated my implementation accordingly and results are slightly better without the miner 😇
Many thanks both of you, much appreciated 🏆

While not a loss, I found notes such as "To implement the loss function described in the paper, use this miner in combination with NTXentLoss(temperature=0.1)" really useful. Perhaps adding a similar note to the two proxy losses could help others.