Can horovd process more shards than workers

[dr-graviton]

Hi!

I understand that when using Horovod in a multi node environment, I should have each shard dedicated per node. Based on the size of my compute node, I've split the data into 4 shards - based on the max that can fit on my node.

I'm wondering if it's possible to have more shards than the # of nodes? For example, if I have 2 nodes, can each download & process a dataset that has 4 shards? In my use case, each shard is big enough to fit on a single worker, but I do not want to have 4 nodes. Can I make do with 2 workers that process the shards in something like a round robin approach?

Thanks