🐛 Bug

To Reproduce

Steps to reproduce the behavior (always include the command you ran):

1. Run cmd '....'
2. See error

Code sample

I think the conditional logic needs to verify whether size is present in the map to cache its computation.
However, the current condition doesn't make sense.

Expected behavior

The condition needs to verify if it is a member of self._features_size_map.

Environment

• fairseq Version (e.g., 1.0 or main):
• PyTorch Version (e.g., 1.0)
• OS (e.g., Linux):
• How you installed fairseq (pip, source):
• Build command you used (if compiling from source):
• Python version:
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information:

Additional context