Train with known triplets

[swarmt]

Hi all!

I feel like I am missing something obvious here, appreciate any guidance.

I have a dataset that contains known image pairs. I want to train a model to minimise the cosine similarity between them.

How should I set up my data loader and training loop?

I think ideally I want to be able to specify the 3 images in the triplet or specify both the anchor positive and anchor negative pairs.

[KevinMusgrave]

Pairs or triplets are formed based on labels. So when you load a batch of image pairs, give each image in the pair the same label.

For example, let's say you have a batch of 4 images, where images 0 and 1 are a pair and images 2 and 3 are a pair:

from pytorch_metric_learning.losses import ContrastiveLoss

loss_fn = ContrastiveLoss()

embeddings = model(images)
labels = torch.tensor([0,0,1,1])
loss = loss_fn(embeddings, labels)

If each image in each batch has exactly 1 positive, then you can accomplish the same thing with SelfSupervisedLoss without creating labels:

from pytorch_metric_learning.losses import ContrastiveLoss, SelfSupervisedLoss

loss_fn = SelfSupervisedLoss(ContrastiveLoss())

anchor_embs = model(anchors)
positive_embs = model(positives)
loss = loss_fn(anchor_embs, positive_embs)

[KevinMusgrave]

There's no special integration of AccuracyCalculator with SelfSupervisedLoss. So you'll have to create labels that make sense "globally", i.e. every anchor-positive pair gets its own unique label within the validation set.

[swarmt]

I think I understand. Thanks for all the help Kevin and thanks for the fantastic library.

[swarmt]

Hey Kevin can I please check with you if this is correct?

My dataloader returns anchor, positive, negative tensors for each __getitem__

Where my mining_func is EmbeddingsAlreadyPackagedAsTriplets()

    def compute_loss(self, batch):
        anchor, positive, negative = batch
        anchor_embeddings = self(anchor)
        positive_embeddings = self(positive)
        negative_embeddings = self(negative)
        embeddings = torch.cat([anchor_embeddings, positive_embeddings, negative_embeddings], dim=0)
        labels = torch.arange(embeddings.size(0))  # set labels to something, doesn't really matter
        triplets = self.mining_func(embeddings, labels)
        return self.loss_func(embeddings, labels, triplets)

[swarmt]

Or alternatively without the miner

    def compute_loss(self, batch):
        anchor, positive, negative = batch
        combined = torch.cat([anchor, positive, negative], dim=0)
        embeddings = self(combined)
        batch_size = anchor.size(0)
        anchors = torch.arange(0, batch_size)
        positives = torch.arange(batch_size, 2 * batch_size)
        negatives = torch.arange(2 * batch_size, 3 * batch_size)
        indices_tuple = (anchors, positives, negatives)
        return self.loss_func(embeddings, labels=None, indices_tuple=indices_tuple)

[KevinMusgrave]

If I remember correctly, the EmbeddingsAlreadyPackagedAsTriplets expects a slightly different format, so I would just create the indices_tuple yourself as you've done in your second snippet.

Your second snippet looks correct.