8x Intel Xeon E3-1246 v3 ($25.00, 4c/8t AVX2) 8/16 GB + Arista DCS 7050QX-32S (40 Gbp/s) [Llama 2 70B]

[spyd3rweb]

Provider: homelab (k8s v1.24.14)
Pod/VolcanoJob resource limits: cpu: [7, 8], memory: [8Gi, 16Gi]

Average Single Token Generation Time

Llama 7B Q40 Weights Q80 Buffer

Metric	4x (1x 8t, 3x 7t; 16GB)	8x (1x 8t, 7x 7t; 8GB)
Avg generation time	990.50 ms	700.25 ms
Avg inference time	732.69 ms	475.75 ms
Avg transfer time	257.12 ms	223.12 ms

4x Pod (1x root: 8t/16GB, 3x worker 7t/16GB) k8s-pod-network (40Gb/s)

pods

spyder@b01:~/repos/distributed-llama$ kubectl get pod -n text-gen -o wide
NAME                          READY   STATUS      RESTARTS   AGE     IP               NODE      NOMINATED NODE   READINESS GATES
dllama-root-0                 0/1     Completed   0          2m51s   192.168.31.32    cloud19   <none>           <none>
dllama-worker-0               0/1     Error       0          3m24s   192.168.27.0     cloud16   <none>           <none>
dllama-worker-1               0/1     Error       0          3m24s   192.168.21.54    cloud18   <none>           <none>
dllama-worker-2               0/1     Error       0          3m24s   192.168.25.126   cloud15   <none>           <none>

logs

spyder@b01:~/repos/distributed-llama$ kubectl logs -f -n text-gen dllama-root-0
./distributed-llama/main inference --model ./models/dllama_meta-llama_Llama-2-70b_q40/dllama_meta-llama_Llama-2-70b_q40.bin --tokenizer ./models/dllama_meta-llama_Llama-2-70b_q40/tokenizer.bin --weights-float-type q40 --buffer-float-type q80 --prompt "Hello world" --steps 16 --nthreads 8 --workers 192.168.27.0:9998 192.168.21.54:9998 192.168.25.126:9998
💡 dim: 8192
💡 hiddenDim: 28672
💡 nLayers: 80
💡 nHeads: 64
💡 nKvHeads: 8
💡 vocabSize: 32000
💡 seqLen: 2048
💡 nSlices: 4
⏩ Loaded 39706066944 bytes
🔶 G  968 ms I  717 ms T  251 ms S 3046611 kB R   3442 kB Hello
🔶 G  967 ms I  685 ms T  282 ms S  11475 kB R   3442 kB  world
🔶 G 1061 ms I  675 ms T  386 ms S  11475 kB R   3442 kB !
🔶 G  905 ms I  686 ms T  217 ms S  11475 kB R   3442 kB  «
🔶 G  993 ms I  692 ms T  300 ms S  11475 kB R   3442 kB  Can
🔶 G 1070 ms I  847 ms T  222 ms S  11475 kB R   3442 kB cer
🔶 G  925 ms I  746 ms T  178 ms S  11475 kB R   3442 kB .
🔶 G 1254 ms I  937 ms T  316 ms S  11475 kB R   3442 kB  C
🔶 G  906 ms I  692 ms T  214 ms S  11475 kB R   3442 kB AN
🔶 G  960 ms I  787 ms T  173 ms S  11475 kB R   3442 kB C
🔶 G 1057 ms I  794 ms T  262 ms S  11475 kB R   3442 kB ER
🔶 G  995 ms I  674 ms T  320 ms S  11475 kB R   3442 kB .
🔶 G  978 ms I  699 ms T  278 ms S  11475 kB R   3442 kB  C
🔶 G  924 ms I  708 ms T  215 ms S  11475 kB R   3442 kB AN
🔶 G  929 ms I  681 ms T  248 ms S  11475 kB R   3442 kB C
🔶 G  956 ms I  703 ms T  252 ms S  11475 kB R   3442 kB ER
Generated tokens:    16
Avg generation time: 990.50 ms
Avg inference time:  732.69 ms
Avg transfer time:   257.12 ms

spyder@b01:~/repos/distributed-llama$ kubectl logs -f -n text-gen dllama-worker-0
./distributed-llama/main worker --port 9998 --nthreads 7
Listening on 0.0.0.0:9998...
Client connected
💡 sliceIndex: 1
💡 nSlices: 4
⏩ Received 120324096 bytes for block 0 (74828 kB/s)
⏩ Received 120324096 bytes for block 1 (82301 kB/s)
⏩ Received 120324096 bytes for block 2 (103997 kB/s)
⏩ Received 120324096 bytes for block 3 (105363 kB/s)
⏩ Received 120324096 bytes for block 4 (107432 kB/s)
⏩ Received 120324096 bytes for block 5 (88800 kB/s)
⏩ Received 120324096 bytes for block 6 (114485 kB/s)
⏩ Received 120324096 bytes for block 7 (83616 kB/s)
⏩ Received 120324096 bytes for block 8 (93274 kB/s)
⏩ Received 120324096 bytes for block 9 (92202 kB/s)
⏩ Received 120324096 bytes for block 10 (101711 kB/s)
⏩ Received 120324096 bytes for block 11 (105270 kB/s)
⏩ Received 120324096 bytes for block 12 (107528 kB/s)
⏩ Received 120324096 bytes for block 13 (108011 kB/s)
⏩ Received 120324096 bytes for block 14 (88800 kB/s)
⏩ Received 120324096 bytes for block 15 (105640 kB/s)
⏩ Received 120324096 bytes for block 16 (96646 kB/s)
⏩ Received 120324096 bytes for block 17 (75156 kB/s)
⏩ Received 120324096 bytes for block 18 (96029 kB/s)
⏩ Received 120324096 bytes for block 19 (114594 kB/s)
⏩ Received 120324096 bytes for block 20 (104087 kB/s)
⏩ Received 120324096 bytes for block 21 (106106 kB/s)
⏩ Received 120324096 bytes for block 22 (103817 kB/s)
⏩ Received 120324096 bytes for block 23 (113835 kB/s)
⏩ Received 120324096 bytes for block 24 (96646 kB/s)
⏩ Received 120324096 bytes for block 25 (109286 kB/s)
⏩ Received 120324096 bytes for block 26 (93202 kB/s)
⏩ Received 120324096 bytes for block 27 (92486 kB/s)
⏩ Received 120324096 bytes for block 28 (90674 kB/s)
⏩ Received 120324096 bytes for block 29 (106860 kB/s)
⏩ Received 120324096 bytes for block 30 (106860 kB/s)
⏩ Received 120324096 bytes for block 31 (111825 kB/s)
⏩ Received 120324096 bytes for block 32 (107624 kB/s)
⏩ Received 120324096 bytes for block 33 (89996 kB/s)
⏩ Received 120324096 bytes for block 34 (91501 kB/s)
⏩ Received 120324096 bytes for block 35 (92202 kB/s)
⏩ Received 120324096 bytes for block 36 (101028 kB/s)
⏩ Received 120324096 bytes for block 37 (83212 kB/s)
⏩ Received 120324096 bytes for block 38 (98064 kB/s)
⏩ Received 120324096 bytes for block 39 (107721 kB/s)
⏩ Received 120324096 bytes for block 40 (90266 kB/s)
⏩ Received 120324096 bytes for block 41 (103997 kB/s)
⏩ Received 120324096 bytes for block 42 (96259 kB/s)
⏩ Received 120324096 bytes for block 43 (94077 kB/s)
⏩ Received 120324096 bytes for block 44 (119964 kB/s)
⏩ Received 120324096 bytes for block 45 (113728 kB/s)
⏩ Received 120324096 bytes for block 46 (85762 kB/s)
⏩ Received 120324096 bytes for block 47 (112558 kB/s)
⏩ Received 120324096 bytes for block 48 (115585 kB/s)
⏩ Received 120324096 bytes for block 49 (113087 kB/s)
⏩ Received 120324096 bytes for block 50 (80431 kB/s)
⏩ Received 120324096 bytes for block 51 (92061 kB/s)
⏩ Received 120324096 bytes for block 52 (112243 kB/s)
⏩ Received 120324096 bytes for block 53 (99195 kB/s)
⏩ Received 120324096 bytes for block 54 (95193 kB/s)
⏩ Received 120324096 bytes for block 55 (98869 kB/s)
⏩ Received 120324096 bytes for block 56 (96336 kB/s)
⏩ Received 120324096 bytes for block 57 (118897 kB/s)
⏩ Received 120324096 bytes for block 58 (114594 kB/s)
⏩ Received 120324096 bytes for block 59 (117389 kB/s)
⏩ Received 120324096 bytes for block 60 (89660 kB/s)
⏩ Received 120324096 bytes for block 61 (110086 kB/s)
⏩ Received 120324096 bytes for block 62 (115363 kB/s)
⏩ Received 120324096 bytes for block 63 (104812 kB/s)
⏩ Received 120324096 bytes for block 64 (114923 kB/s)
⏩ Received 120324096 bytes for block 65 (106012 kB/s)
⏩ Received 120324096 bytes for block 66 (78082 kB/s)
⏩ Received 120324096 bytes for block 67 (106765 kB/s)
⏩ Received 120324096 bytes for block 68 (113728 kB/s)
⏩ Received 120324096 bytes for block 69 (109286 kB/s)
⏩ Received 120324096 bytes for block 70 (101883 kB/s)
⏩ Received 120324096 bytes for block 71 (108011 kB/s)
⏩ Received 120324096 bytes for block 72 (102143 kB/s)
⏩ Received 120324096 bytes for block 73 (102403 kB/s)
⏩ Received 120324096 bytes for block 74 (104087 kB/s)
⏩ Received 120324096 bytes for block 75 (82021 kB/s)
⏩ Received 120324096 bytes for block 76 (97984 kB/s)
⏩ Received 120324096 bytes for block 77 (99937 kB/s)
⏩ Received 120324096 bytes for block 78 (96568 kB/s)
⏩ Received 120324096 bytes for block 79 (93637 kB/s)
Error receiving data: socket closed

top nodes

spyder@b01:~/repos/distributed-llama$ kubectl top nodes
NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
cloud11   539m         6%     19441Mi         62%
cloud12   638m         7%     20193Mi         64%
cloud13   730m         9%     23623Mi         75%
cloud14   377m         4%     17147Mi         54%
cloud15   4195m        52%    19033Mi         60%
cloud16   5423m        67%    19343Mi         61%
cloud17   239m         2%     14150Mi         45%
cloud18   5339m        66%    19620Mi         62%
cloud19   6817m        85%    3153Mi          10%
cloud24   239m         0%     17475Mi         13%

8x Pod (1x root: 8t/8GB, 7x worker 7t/8GB) k8s-pod-network (40Gb/s)

pods

spyder@b01:~/repos/distributed-llama$ kubectl get pod -n text-gen -o wide
NAME                          READY   STATUS      RESTARTS   AGE     IP               NODE      NOMINATED NODE   READINESS GATES
dllama-root-0                 0/1     Completed   0          2m41s   192.168.31.3     cloud19   <none>           <none>
dllama-worker-0               0/1     Error       0          3m22s   192.168.30.60    cloud12   <none>           <none>
dllama-worker-1               0/1     Error       0          3m22s   192.168.31.100   cloud13   <none>           <none>
dllama-worker-2               0/1     Error       0          3m22s   192.168.27.6     cloud16   <none>           <none>
dllama-worker-3               0/1     Error       0          3m22s   192.168.21.55    cloud18   <none>           <none>
dllama-worker-4               0/1     Error       0          3m22s   192.168.24.252   cloud14   <none>           <none>
dllama-worker-5               0/1     Error       0          3m22s   192.168.25.127   cloud15   <none>           <none>
dllama-worker-6               0/1     Error       0          3m22s   192.168.21.84    cloud17   <none>           <none>

logs

spyder@b01:~/repos/distributed-llama$ kubectl logs -f -n text-gen dllama-root-0
./distributed-llama/main inference --model ./models/dllama_meta-llama_Llama-2-70b_q40/dllama_meta-llama_Llama-2-70b_q40.bin --tokenizer ./models/dllama_meta-llama_Llama-2-70b_q40/tokenizer.bin --weights-float-type q40 --buffer-float-type q80 --prompt "Hello world" --steps 16 --nthreads 8 --workers 192.168.30.60:9998 192.168.31.100:9998 192.168.27.6:9998 192.168.21.55:9998 192.168.24.252:9998 192.168.25.127:9998 192.168.21.84:9998
💡 dim: 8192
💡 hiddenDim: 28672
💡 nLayers: 80
💡 nHeads: 64
💡 nKvHeads: 8
💡 vocabSize: 32000
💡 seqLen: 2048
💡 nSlices: 8
⏩ Loaded 39706066944 bytes
🔶 G  647 ms I  442 ms T  204 ms S 3569849 kB R   4016 kB Hello
🔶 G  768 ms I  493 ms T  260 ms S  28857 kB R   4016 kB  world
🔶 G  721 ms I  463 ms T  257 ms S  28857 kB R   4016 kB !
🔶 G  818 ms I  489 ms T  329 ms S  28857 kB R   4016 kB  Welcome
🔶 G  576 ms I  442 ms T  133 ms S  28857 kB R   4016 kB  to
🔶 G  669 ms I  437 ms T  231 ms S  28857 kB R   4016 kB  Word
🔶 G  636 ms I  471 ms T  164 ms S  28857 kB R   4016 kB Press
🔶 G  666 ms I  459 ms T  207 ms S  28857 kB R   4016 kB .
🔶 G  758 ms I  467 ms T  290 ms S  28857 kB R   4016 kB  This
🔶 G  589 ms I  442 ms T  147 ms S  28857 kB R   4016 kB  is
🔶 G  630 ms I  455 ms T  175 ms S  28857 kB R   4016 kB  your
🔶 G  590 ms I  463 ms T  127 ms S  28857 kB R   4016 kB  first
🔶 G  627 ms I  472 ms T  155 ms S  28857 kB R   4016 kB  post
🔶 G  962 ms I  696 ms T  266 ms S  28857 kB R   4016 kB .
🔶 G  903 ms I  456 ms T  447 ms S  28857 kB R   4016 kB  Edit
🔶 G  644 ms I  465 ms T  178 ms S  28857 kB R   4016 kB  or
Generated tokens:    16
Avg generation time: 700.25 ms
Avg inference time:  475.75 ms
Avg transfer time:   223.12 ms

spyder@b01:~/repos/distributed-llama$ kubectl logs -f -n text-gen dllama-worker-0
./distributed-llama/main worker --port 9998 --nthreads 7
Listening on 0.0.0.0:9998...
Client connected
💡 sliceIndex: 1
💡 nSlices: 8
⏩ Received 60162048 bytes for block 0 (28392 kB/s)
⏩ Received 60162048 bytes for block 1 (38029 kB/s)
⏩ Received 60162048 bytes for block 2 (48207 kB/s)
⏩ Received 60162048 bytes for block 3 (51465 kB/s)
⏩ Received 60162048 bytes for block 4 (50556 kB/s)
⏩ Received 60162048 bytes for block 5 (52133 kB/s)
⏩ Received 60162048 bytes for block 6 (44664 kB/s)
⏩ Received 60162048 bytes for block 7 (48912 kB/s)
⏩ Received 60162048 bytes for block 8 (46782 kB/s)
⏩ Received 60162048 bytes for block 9 (50727 kB/s)
⏩ Received 60162048 bytes for block 10 (47372 kB/s)
⏩ Received 60162048 bytes for block 11 (46066 kB/s)
⏩ Received 60162048 bytes for block 12 (40269 kB/s)
⏩ Received 60162048 bytes for block 13 (54103 kB/s)
⏩ Received 60162048 bytes for block 14 (45508 kB/s)
⏩ Received 60162048 bytes for block 15 (50052 kB/s)
⏩ Received 60162048 bytes for block 16 (47597 kB/s)
⏩ Received 60162048 bytes for block 17 (41577 kB/s)
⏩ Received 60162048 bytes for block 18 (49072 kB/s)
⏩ Received 60162048 bytes for block 19 (50813 kB/s)
⏩ Received 60162048 bytes for block 20 (51641 kB/s)
⏩ Received 60162048 bytes for block 21 (57407 kB/s)
⏩ Received 60162048 bytes for block 22 (51509 kB/s)
⏩ Received 60162048 bytes for block 23 (55757 kB/s)
⏩ Received 60162048 bytes for block 24 (42397 kB/s)
⏩ Received 60162048 bytes for block 25 (52589 kB/s)
⏩ Received 60162048 bytes for block 26 (47149 kB/s)
⏩ Received 60162048 bytes for block 27 (56121 kB/s)
⏩ Received 60162048 bytes for block 28 (51421 kB/s)
⏩ Received 60162048 bytes for block 29 (53430 kB/s)
⏩ Received 60162048 bytes for block 30 (50472 kB/s)
⏩ Received 60162048 bytes for block 31 (53194 kB/s)
⏩ Received 60162048 bytes for block 32 (39816 kB/s)
⏩ Received 60162048 bytes for block 33 (53525 kB/s)
⏩ Received 60162048 bytes for block 34 (51202 kB/s)
⏩ Received 60162048 bytes for block 35 (40269 kB/s)
⏩ Received 60162048 bytes for block 36 (42608 kB/s)
⏩ Received 60162048 bytes for block 37 (57026 kB/s)
⏩ Received 60162048 bytes for block 38 (53620 kB/s)
⏩ Received 60162048 bytes for block 39 (50599 kB/s)
⏩ Received 60162048 bytes for block 40 (42338 kB/s)
⏩ Received 60162048 bytes for block 41 (55245 kB/s)
⏩ Received 60162048 bytes for block 42 (51028 kB/s)
⏩ Received 60162048 bytes for block 43 (52960 kB/s)
⏩ Received 60162048 bytes for block 44 (48479 kB/s)
⏩ Received 60162048 bytes for block 45 (49475 kB/s)
⏩ Received 60162048 bytes for block 46 (40431 kB/s)
⏩ Received 60162048 bytes for block 47 (49968 kB/s)
⏩ Received 60162048 bytes for block 48 (52269 kB/s)
⏩ Received 60162048 bytes for block 49 (43850 kB/s)
⏩ Received 60162048 bytes for block 50 (49273 kB/s)
⏩ Received 60162048 bytes for block 51 (48596 kB/s)
⏩ Received 60162048 bytes for block 52 (53860 kB/s)
⏩ Received 60162048 bytes for block 53 (48793 kB/s)
⏩ Received 60162048 bytes for block 54 (43407 kB/s)
⏩ Received 60162048 bytes for block 55 (51730 kB/s)
⏩ Received 60162048 bytes for block 56 (43251 kB/s)
⏩ Received 60162048 bytes for block 57 (53668 kB/s)
⏩ Received 60162048 bytes for block 58 (41151 kB/s)
⏩ Received 60162048 bytes for block 59 (44269 kB/s)
⏩ Received 60162048 bytes for block 60 (50093 kB/s)
⏩ Received 60162048 bytes for block 61 (50472 kB/s)
⏩ Received 60162048 bytes for block 62 (54643 kB/s)
⏩ Received 60162048 bytes for block 63 (47223 kB/s)
⏩ Received 60162048 bytes for block 64 (51686 kB/s)
⏩ Received 60162048 bytes for block 65 (51775 kB/s)
⏩ Received 60162048 bytes for block 66 (54693 kB/s)
⏩ Received 60162048 bytes for block 67 (53573 kB/s)
⏩ Received 60162048 bytes for block 68 (51333 kB/s)
⏩ Received 60162048 bytes for block 69 (43407 kB/s)
⏩ Received 60162048 bytes for block 70 (44697 kB/s)
⏩ Received 60162048 bytes for block 71 (49598 kB/s)
⏩ Received 60162048 bytes for block 72 (53241 kB/s)
⏩ Received 60162048 bytes for block 73 (48557 kB/s)
⏩ Received 60162048 bytes for block 74 (49886 kB/s)
⏩ Received 60162048 bytes for block 75 (47521 kB/s)
⏩ Received 60162048 bytes for block 76 (39895 kB/s)
⏩ Received 60162048 bytes for block 77 (50010 kB/s)
⏩ Received 60162048 bytes for block 78 (47002 kB/s)
⏩ Received 60162048 bytes for block 79 (48440 kB/s)
Error receiving data: socket closed

top nodes

spyder@b01:~/repos/distributed-llama$ kubectl top nodes
NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
cloud11   477m         5%     19513Mi         62%
cloud12   7361m        92%    24757Mi         79%
cloud13   7493m        93%    28517Mi         91%
cloud14   4598m        57%    17023Mi         54%
cloud15   7371m        92%    23604Mi         75%
cloud16   5044m        63%    19217Mi         61%
cloud17   4506m        56%    14231Mi         45%
cloud18   6375m        79%    24121Mi         77%
cloud19   7966m        99%    10219Mi         32%
cloud24   275m         0%     17467Mi         13%

CPU Info

CPU: Intel Xeon E3-1246 v3

root@cloud16:~# lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              2
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           60
Model name:                      Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
Stepping:                        3
CPU MHz:                         1575.820
CPU max MHz:                     3900.0000
CPU min MHz:                     800.0000
BogoMIPS:                        6984.26
Virtualization:                  VT-x
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        1 MiB
L3 cache:                        8 MiB
NUMA node0 CPU(s):               0-7
Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:             Mitigation; Microcode
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht
                                 tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid
                                 aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x
                                 2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ss
                                 bd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invp
                                 cid xsaveopt dtherm ida arat pln pts md_clear flush_l1d

[b4rtaz]

Great results @spyd3rweb! Do you have logs from any worker? I'm curious about the transfer speed.

[spyd3rweb]

Just realized the worker also takes the nthread argument, updated to show the workers using 7 for the two runs. Unfortunately, it's an active cluster so can't max out the nodes to 8t, especially the 3 master/control-plane nodes. Definitely cpu limited on the transfer speed.

I have 7 of the worker nodes which also have a separate 56 Gbp/s rdma-ipoib network, I've never done any rdma development but I might look into what it would take to modify dllama to see how the performance changes when the memory access/transfer is offloaded to the RDMA-capable Host Channel Adapters (HCA). I'm using old Mellanox ConnectX-3 Pros VPI (~$35).