4 x Raspberry Pi 5 8GB [Mixtral 8x7B]

[b4rtaz]

Model: Mixtral 8x7B: 46.7B total parameters, 12.9B / token (source)
Weights: Q40 (24.89GB)
Buffer: Q80
Switch: TP-Link LS1008G Switch
Version: 0.3.0 + patch

Metric	4 x Raspberry Pi 5 8GB
Avg generation time	564.83 ms
Avg inference time	397.19 ms
Avg transfer time	165.47 ms