[Root] Ryzen 3600, [Worker1] Ryzen 3600, [Network] 1 Gigabit #41
Replies: 6 comments 18 replies
-
|
I was really surprised to that 8 Raspberry Pi 8B's were faster than my two Ryzen 3600's |
Beta Was this translation helpful? Give feedback.
-
|
Hello @DifferentialityDevelopment!
This was a bug, I fixed it here.
This is "expected". The application utilizes CPU fully, so if you set too high value for
AVX is not fully supported yet (only Q40 × Q80 matmul), but maybe this problem requires some investigation. |
Beta Was this translation helpful? Give feedback.
-
|
@DifferentialityDevelopment what kind of buffer have you used in this test ( |
Beta Was this translation helpful? Give feedback.
-
|
Q80 I think? |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Will try again tonight will the latest release, still going over the code to try and understand how it all works :) |
Beta Was this translation helpful? Give feedback.
-
|
With the latest commit I went up to 2.6 tokens per second from 2.15 🔥 🔶 G 380 ms I 207 ms T 172 ms S 1917438 kB R 442 kB Hello |
Beta Was this translation helpful? Give feedback.
-
|
For certain!, it's a dirt cheap gigabit switch. |
Beta Was this translation helpful? Give feedback.
-
|
If I run it without any workers sudo nice -n -20 ./main inference --steps 20 --prompt "Hello World! " --model ~/Meta-Llama-3-8B-Instruct-Distributed/dllama_original_q40.bin --tokenizer ~/Meta-Llama-3-8B-Instruct-Distributed/dllama-llama3-tokenizer.t --weights-float-type q40 --buffer-float-type q80 --nthreads 8 |
Beta Was this translation helpful? Give feedback.
-
|
The inference time is higher, but it just goes to show the transfer time also needs to be considered when running a model using distributed-llama. |
Beta Was this translation helpful? Give feedback.
-
|
I see in your logs: This gives 31% speed up. Here I got 34% between 1 node and 2 nodes. It's very interesting what takes ~15% the most, maybe the last layer 🤔. |
Beta Was this translation helpful? Give feedback.
-
Let's calculate the transfer time theoretically.llama3 8BSince the transfer is full-duplex, there's no interference between uplink and downlink. So, the average transfer time should be 4.08ms. However, your result is llama2 7BFor comparison, I summarize a similar model (llama2 7B) using different devices: VMsIn this discussion, the Network Bandwidth is So, the network utilization ratio is merely 3%. RaspberryPi
Also, the result of the Raspberry Pi cluster is calculated to be 9.0%, 48.0%, 14.1% for 2,4,8 devices.
SummaryI think the network utilization, average around 11%, ranging from |
Beta Was this translation helpful? Give feedback.
-
Maybe the bottleneck is not the switch, it's the root node instead? |
Beta Was this translation helpful? Give feedback.
-
|
I have 6 CPU cores and 12 threads and I ran it with 8 threads, CPU usage hanged around 80% +/- |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I agree with the |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! I'll try that out. |
Beta Was this translation helpful? Give feedback.
-
|
Windows -> Windows send_results [ ID] Interval Transfer Bitrate WSL -> WSL send_results [ ID] Interval Transfer Bitrate Retr |
Beta Was this translation helpful? Give feedback.
-
|
Interestingly, if I use --parallel 4 in WSL -> WSL iperf3 -c 192.168.1.3 -p 9990 --parallel 4 [ 5] 1.00-2.00 sec 37.5 MBytes 315 Mbits/sec 0 2.03 MBytes [ 5] 2.00-3.00 sec 35.0 MBytes 294 Mbits/sec 0 2.03 MBytes [ 5] 3.00-4.00 sec 35.0 MBytes 294 Mbits/sec 0 2.03 MBytes [ 5] 4.00-5.00 sec 33.8 MBytes 283 Mbits/sec 0 2.03 MBytes [ 5] 5.00-6.00 sec 36.2 MBytes 304 Mbits/sec 0 2.03 MBytes [ 5] 6.00-7.00 sec 30.0 MBytes 252 Mbits/sec 0 2.03 MBytes [ 5] 7.00-8.00 sec 27.5 MBytes 231 Mbits/sec 0 2.03 MBytes [ 5] 8.00-9.00 sec 26.2 MBytes 220 Mbits/sec 0 2.03 MBytes [ 5] 9.00-10.00 sec 32.5 MBytes 273 Mbits/sec 0 2.03 MBytes [ ID] Interval Transfer Bitrate Retr Then the throughput increases again to almost 1 gigabit speed |
Beta Was this translation helpful? Give feedback.
-
|
If I run it with sudo nice -n 20 iperf3 -c 192.168.1.3 -p 9990 Connecting to host 192.168.1.3, port 9990 [ ID] Interval Transfer Bitrate Retr |
Beta Was this translation helpful? Give feedback.
-
|
That's weird. Since a single socket can only enjoy half bandwidth while four threads can enjoy the total bandwidth simultaneously, then why don't programme the multi-thread sockets? https://github.com/b4rtaz/distributed-llama/blob/main/src%2Fsocket.cpp#L151-L179 @b4rtaz |
Beta Was this translation helpful? Give feedback.
-
|
@zhengpeirong very interesting. 🤔 |
Beta Was this translation helpful? Give feedback.
-
Distributed Llama Version: 0.3.1
Model: Llama 3 8B Q40 (huggingface)
Switch: Tenda Gigabit Switch (Generic)
Couple of things I discovered during my test, similiar to how workers must be a multiple of 2 ie 2, 4, 8 etc
Threads are subject to the same condition, if you specify an odd number of threads it will crash with:
Assertion `size % nThreads == 0' failed.
Another thing I noticed that's counter intuitive is that while increasing thread count gave noticeable improvement in tokens/s, the speed would tank if the amount of threads exceeded the amount of logical processors available to the machine
It maxed out the cpu usage but the tokens/s tanked really hard, which I found very weird, for reference I let it run just to post the results of that.
For reference both machines have a Ryzen 3600 with 6 cores and 12 logical processors.
As a first step I've gotten it to run locally, I was surprised to find that the inference speed was so similar to Raspberry Pi 4B's, despite the hardware being so much faster.
Beta Was this translation helpful? Give feedback.
All reactions