Duplicate dividing in relative positional encoding

[songweige]

Hey @lucidrains, thanks for keeping these models implemented. In line 88

video-diffusion-pytorch/video_diffusion_pytorch/video_diffusion_pytorch.py

Lines 84 to 88 in f55f1b0

	num_buckets //= 2
	ret += (n < 0).long() * num_buckets
	n = torch.abs(n)
	max_exact = num_buckets // 2

you have max_exact as the half of num_buckets, whose value was already halved in line 84.

I think that is duplicated and should be changed to identity:

 max_exact = num_buckets

[oxjohanndiep]

I suggest you read the paper "On Scalar Embedding of Relative Positions in Attention Models". In that paper, they explain the implemented bucketing function.