You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
b4rtaz@raspberrypi3:~/distributed-llama $ ./main inference --prompt "The Eiffel Tower is" --weights-float-type q40 --buffer-float-type q80 --nthreads 4 --model ../dllama_meta-llama-3-8b_q40.bin --tokenizer ../dllama-llama3-tokenizer.t --steps 64 --workers 10.0.0.4:9998 10.0.0.1:9998 10.0.0.2:9998
💡 arch: llama2
💡 dim: 4096
💡 hiddenDim: 14336
💡 nLayers: 32
💡 nHeads: 32
💡 nKvHeads: 8
💡 vocabSize: 128256
💡 seqLen: 2048
💡 nSlices: 4
💡 ropeTheta: 500000.0
📄 bosId: 128000
📄 eosId: 128001
...
⏩ Loaded 6323781632 bytes
🔶 G 363 ms I 298 ms T 59 ms S 2877687 kB R 714 kB The
🔶 G 311 ms I 257 ms T 54 ms S 2295 kB R 714 kB E
🔶 G 356 ms I 298 ms T 57 ms S 2295 kB R 714 kB iff
🔶 G 371 ms I 303 ms T 68 ms S 2295 kB R 714 kB el
🔶 G 397 ms I 295 ms T 102 ms S 2295 kB R 714 kB Tower
🔶 G 416 ms I 312 ms T 103 ms S 2295 kB R 714 kB is
🔶 G 383 ms I 296 ms T 85 ms S 2295 kB R 714 kB a
🔶 G 365 ms I 293 ms T 71 ms S 2295 kB R 714 kB
🔶 G 355 ms I 288 ms T 66 ms S 2295 kB R 714 kB 324
🔶 G 325 ms I 258 ms T 66 ms S 2295 kB R 714 kB -m
🔶 G 316 ms I 259 ms T 55 ms S 2295 kB R 714 kB etre
🔶 G 325 ms I 261 ms T 63 ms S 2295 kB R 714 kB (
🔶 G 316 ms I 259 ms T 55 ms S 2295 kB R 714 kB 1
🔶 G 312 ms I 258 ms T 51 ms S 2295 kB R 714 kB ,
🔶 G 313 ms I 258 ms T 52 ms S 2295 kB R 714 kB 063
🔶 G 373 ms I 266 ms T 106 ms S 2295 kB R 714 kB ft
🔶 G 334 ms I 265 ms T 68 ms S 2295 kB R 714 kB )
🔶 G 345 ms I 265 ms T 76 ms S 2295 kB R 714 kB wrought
🔶 G 314 ms I 256 ms T 56 ms S 2295 kB R 714 kB iron
🔶 G 311 ms I 260 ms T 49 ms S 2295 kB R 714 kB lattice
🔶 G 333 ms I 271 ms T 61 ms S 2295 kB R 714 kB tower
🔶 G 312 ms I 268 ms T 42 ms S 2295 kB R 714 kB on
🔶 G 372 ms I 283 ms T 87 ms S 2295 kB R 714 kB the
🔶 G 307 ms I 262 ms T 44 ms S 2295 kB R 714 kB Champ
🔶 G 380 ms I 262 ms T 116 ms S 2295 kB R 714 kB de
🔶 G 326 ms I 262 ms T 62 ms S 2295 kB R 714 kB Mars
🔶 G 312 ms I 263 ms T 48 ms S 2295 kB R 714 kB in
🔶 G 332 ms I 264 ms T 67 ms S 2295 kB R 714 kB Paris
🔶 G 316 ms I 263 ms T 51 ms S 2295 kB R 714 kB ,
🔶 G 340 ms I 266 ms T 72 ms S 2295 kB R 714 kB France
🔶 G 337 ms I 260 ms T 76 ms S 2295 kB R 714 kB .
🔶 G 321 ms I 259 ms T 59 ms S 2295 kB R 714 kB Built
🔶 G 326 ms I 261 ms T 64 ms S 2295 kB R 714 kB in
🔶 G 315 ms I 260 ms T 54 ms S 2295 kB R 714 kB
🔶 G 327 ms I 259 ms T 66 ms S 2295 kB R 714 kB 188
🔶 G 312 ms I 257 ms T 54 ms S 2295 kB R 714 kB 9
🔶 G 322 ms I 263 ms T 54 ms S 2295 kB R 714 kB as
🔶 G 321 ms I 259 ms T 61 ms S 2295 kB R 714 kB the
🔶 G 330 ms I 261 ms T 68 ms S 2295 kB R 714 kB entrance
🔶 G 329 ms I 257 ms T 70 ms S 2295 kB R 714 kB arch
🔶 G 313 ms I 260 ms T 52 ms S 2295 kB R 714 kB to
🔶 G 313 ms I 259 ms T 53 ms S 2295 kB R 714 kB the
🔶 G 324 ms I 260 ms T 63 ms S 2295 kB R 714 kB
🔶 G 329 ms I 266 ms T 62 ms S 2295 kB R 714 kB 188
🔶 G 318 ms I 260 ms T 56 ms S 2295 kB R 714 kB 9
🔶 G 327 ms I 264 ms T 61 ms S 2295 kB R 714 kB World
🔶 G 315 ms I 262 ms T 51 ms S 2295 kB R 714 kB 's
🔶 G 315 ms I 264 ms T 50 ms S 2295 kB R 714 kB Fair
🔶 G 337 ms I 266 ms T 70 ms S 2295 kB R 714 kB ,
🔶 G 328 ms I 262 ms T 65 ms S 2295 kB R 714 kB it
🔶 G 314 ms I 263 ms T 50 ms S 2295 kB R 714 kB has
🔶 G 320 ms I 267 ms T 52 ms S 2295 kB R 714 kB become
🔶 G 328 ms I 254 ms T 72 ms S 2295 kB R 714 kB both
🔶 G 321 ms I 264 ms T 56 ms S 2295 kB R 714 kB a
🔶 G 352 ms I 301 ms T 50 ms S 2295 kB R 714 kB global
🔶 G 318 ms I 260 ms T 57 ms S 2295 kB R 714 kB cultural
🔶 G 324 ms I 267 ms T 56 ms S 2295 kB R 714 kB icon
🔶 G 328 ms I 269 ms T 58 ms S 2295 kB R 714 kB of
🔶 G 315 ms I 260 ms T 54 ms S 2295 kB R 714 kB France
🔶 G 319 ms I 269 ms T 49 ms S 2295 kB R 714 kB and
🔶 G 320 ms I 263 ms T 56 ms S 2295 kB R 714 kB one
🔶 G 329 ms I 267 ms T 61 ms S 2295 kB R 714 kB of
🔶 G 319 ms I 269 ms T 49 ms S 2295 kB R 714 kB the
🔶 G 317 ms I 267 ms T 49 ms S 2295 kB R 714 kB most
Generated tokens: 64
Avg generation time: 331.47 ms
Avg inference time: 267.62 ms
Avg transfer time: 62.34 ms
2 x Raspberry Pi 5 8GB
💡 arch: llama2
💡 dim: 4096
💡 hiddenDim: 14336
💡 nLayers: 32
💡 nHeads: 32
💡 nKvHeads: 8
💡 vocabSize: 128256
💡 seqLen: 2048
💡 nSlices: 2
💡 ropeTheta: 500000.0
📄 bosId: 128000
📄 eosId: 128001
...
⏩ Loaded 6323781632 bytes
🔶 G 542 ms I 468 ms T 70 ms S 1917574 kB R 476 kB The
🔶 G 416 ms I 355 ms T 61 ms S 646 kB R 476 kB E
🔶 G 425 ms I 358 ms T 67 ms S 646 kB R 476 kB iff
🔶 G 480 ms I 366 ms T 114 ms S 646 kB R 476 kB el
🔶 G 424 ms I 354 ms T 70 ms S 646 kB R 476 kB Tower
🔶 G 458 ms I 359 ms T 99 ms S 646 kB R 476 kB is
🔶 G 419 ms I 357 ms T 60 ms S 646 kB R 476 kB the
🔶 G 419 ms I 360 ms T 57 ms S 646 kB R 476 kB most
🔶 G 434 ms I 358 ms T 75 ms S 646 kB R 476 kB visited
🔶 G 453 ms I 363 ms T 88 ms S 646 kB R 476 kB paid
🔶 G 461 ms I 359 ms T 101 ms S 646 kB R 476 kB monument
🔶 G 432 ms I 355 ms T 75 ms S 646 kB R 476 kB in
🔶 G 450 ms I 356 ms T 92 ms S 646 kB R 476 kB the
🔶 G 518 ms I 388 ms T 128 ms S 646 kB R 476 kB world
🔶 G 473 ms I 366 ms T 105 ms S 646 kB R 476 kB and
🔶 G 417 ms I 353 ms T 63 ms S 646 kB R 476 kB is
🔶 G 420 ms I 355 ms T 63 ms S 646 kB R 476 kB one
🔶 G 451 ms I 355 ms T 95 ms S 646 kB R 476 kB of
🔶 G 417 ms I 357 ms T 59 ms S 646 kB R 476 kB the
🔶 G 418 ms I 359 ms T 58 ms S 646 kB R 476 kB main
🔶 G 419 ms I 356 ms T 62 ms S 646 kB R 476 kB landmarks
🔶 G 426 ms I 355 ms T 70 ms S 646 kB R 476 kB of
🔶 G 464 ms I 361 ms T 102 ms S 646 kB R 476 kB the
🔶 G 426 ms I 356 ms T 68 ms S 646 kB R 476 kB city
🔶 G 436 ms I 360 ms T 75 ms S 646 kB R 476 kB of
🔶 G 461 ms I 361 ms T 99 ms S 646 kB R 476 kB Paris
🔶 G 477 ms I 394 ms T 81 ms S 646 kB R 476 kB ,
🔶 G 464 ms I 361 ms T 98 ms S 646 kB R 476 kB France
🔶 G 425 ms I 356 ms T 68 ms S 646 kB R 476 kB .
🔶 G 422 ms I 355 ms T 66 ms S 646 kB R 476 kB It
🔶 G 420 ms I 354 ms T 64 ms S 646 kB R 476 kB is
🔶 G 436 ms I 355 ms T 80 ms S 646 kB R 476 kB located
🔶 G 461 ms I 361 ms T 99 ms S 646 kB R 476 kB on
🔶 G 461 ms I 359 ms T 100 ms S 646 kB R 476 kB the
🔶 G 431 ms I 356 ms T 73 ms S 646 kB R 476 kB Champ
🔶 G 404 ms I 356 ms T 47 ms S 646 kB R 476 kB de
🔶 G 461 ms I 361 ms T 99 ms S 646 kB R 476 kB Mars
🔶 G 430 ms I 365 ms T 64 ms S 646 kB R 476 kB in
🔶 G 426 ms I 355 ms T 70 ms S 646 kB R 476 kB the
🔶 G 420 ms I 358 ms T 60 ms S 646 kB R 476 kB seventh
🔶 G 437 ms I 354 ms T 82 ms S 646 kB R 476 kB arr
🔶 G 461 ms I 358 ms T 102 ms S 646 kB R 476 kB ond
🔶 G 420 ms I 360 ms T 58 ms S 646 kB R 476 kB issement
🔶 G 421 ms I 357 ms T 62 ms S 646 kB R 476 kB of
🔶 G 427 ms I 359 ms T 67 ms S 646 kB R 476 kB Paris
🔶 G 419 ms I 365 ms T 53 ms S 646 kB R 476 kB .
🔶 G 465 ms I 369 ms T 94 ms S 646 kB R 476 kB The
🔶 G 435 ms I 353 ms T 81 ms S 646 kB R 476 kB tower
🔶 G 431 ms I 367 ms T 62 ms S 646 kB R 476 kB is
🔶 G 400 ms I 359 ms T 39 ms S 646 kB R 476 kB the
🔶 G 501 ms I 364 ms T 135 ms S 646 kB R 476 kB tallest
🔶 G 420 ms I 361 ms T 57 ms S 646 kB R 476 kB building
🔶 G 420 ms I 351 ms T 67 ms S 646 kB R 476 kB in
🔶 G 422 ms I 362 ms T 59 ms S 646 kB R 476 kB Paris
🔶 G 466 ms I 371 ms T 94 ms S 646 kB R 476 kB and
🔶 G 473 ms I 366 ms T 106 ms S 646 kB R 476 kB the
🔶 G 465 ms I 364 ms T 99 ms S 646 kB R 476 kB most
🔶 G 444 ms I 362 ms T 81 ms S 646 kB R 476 kB -
🔶 G 454 ms I 365 ms T 87 ms S 646 kB R 476 kB visited
🔶 G 471 ms I 360 ms T 110 ms S 646 kB R 476 kB paid
🔶 G 464 ms I 364 ms T 98 ms S 646 kB R 476 kB monument
🔶 G 474 ms I 363 ms T 110 ms S 646 kB R 476 kB in
🔶 G 431 ms I 358 ms T 72 ms S 646 kB R 476 kB the
🔶 G 515 ms I 407 ms T 107 ms S 646 kB R 476 kB world
Generated tokens: 64
Avg generation time: 444.27 ms
Avg inference time: 362.73 ms
Avg transfer time: 80.11 ms
1 x Raspberry Pi 5 8GB
💡 arch: llama2
💡 dim: 4096
💡 hiddenDim: 14336
💡 nLayers: 32
💡 nHeads: 32
💡 nKvHeads: 8
💡 vocabSize: 128256
💡 seqLen: 2048
💡 nSlices: 1
💡 ropeTheta: 500000.0
📄 bosId: 128000
📄 eosId: 128001
...
⏩ Loaded 6323781632 bytes
🔶 G 623 ms I 616 ms T 3 ms S 0 kB R 0 kB The
🔶 G 529 ms I 529 ms T 0 ms S 0 kB R 0 kB E
🔶 G 573 ms I 566 ms T 7 ms S 0 kB R 0 kB iff
🔶 G 574 ms I 564 ms T 10 ms S 0 kB R 0 kB el
🔶 G 568 ms I 560 ms T 8 ms S 0 kB R 0 kB Tower
🔶 G 573 ms I 565 ms T 8 ms S 0 kB R 0 kB is
🔶 G 577 ms I 567 ms T 8 ms S 0 kB R 0 kB
🔶 G 560 ms I 549 ms T 10 ms S 0 kB R 0 kB 320
🔶 G 614 ms I 603 ms T 10 ms S 0 kB R 0 kB meters
🔶 G 566 ms I 557 ms T 8 ms S 0 kB R 0 kB tall
🔶 G 575 ms I 564 ms T 10 ms S 0 kB R 0 kB .
🔶 G 530 ms I 529 ms T 0 ms S 0 kB R 0 kB It
🔶 G 531 ms I 530 ms T 0 ms S 0 kB R 0 kB is
🔶 G 562 ms I 552 ms T 9 ms S 0 kB R 0 kB
🔶 G 535 ms I 533 ms T 0 ms S 0 kB R 0 kB 320
🔶 G 531 ms I 530 ms T 0 ms S 0 kB R 0 kB meters
🔶 G 567 ms I 558 ms T 8 ms S 0 kB R 0 kB away
🔶 G 575 ms I 564 ms T 9 ms S 0 kB R 0 kB from
🔶 G 573 ms I 561 ms T 10 ms S 0 kB R 0 kB the
🔶 G 564 ms I 552 ms T 9 ms S 0 kB R 0 kB observation
🔶 G 571 ms I 561 ms T 9 ms S 0 kB R 0 kB deck
🔶 G 576 ms I 564 ms T 10 ms S 0 kB R 0 kB .
🔶 G 570 ms I 563 ms T 6 ms S 0 kB R 0 kB Find
🔶 G 566 ms I 556 ms T 8 ms S 0 kB R 0 kB the
🔶 G 574 ms I 566 ms T 6 ms S 0 kB R 0 kB angle
🔶 G 574 ms I 565 ms T 8 ms S 0 kB R 0 kB between
🔶 G 579 ms I 568 ms T 10 ms S 0 kB R 0 kB the
🔶 G 566 ms I 556 ms T 8 ms S 0 kB R 0 kB E
🔶 G 572 ms I 564 ms T 6 ms S 0 kB R 0 kB iff
🔶 G 575 ms I 564 ms T 9 ms S 0 kB R 0 kB el
🔶 G 571 ms I 563 ms T 6 ms S 0 kB R 0 kB Tower
🔶 G 532 ms I 531 ms T 0 ms S 0 kB R 0 kB and
🔶 G 532 ms I 531 ms T 0 ms S 0 kB R 0 kB the
🔶 G 532 ms I 530 ms T 0 ms S 0 kB R 0 kB observation
🔶 G 533 ms I 531 ms T 0 ms S 0 kB R 0 kB deck
🔶 G 565 ms I 557 ms T 6 ms S 0 kB R 0 kB .
🔶 G 562 ms I 552 ms T 9 ms S 0 kB R 0 kB The
🔶 G 575 ms I 573 ms T 0 ms S 0 kB R 0 kB angle
🔶 G 579 ms I 569 ms T 8 ms S 0 kB R 0 kB is
🔶 G 572 ms I 561 ms T 10 ms S 0 kB R 0 kB measured
🔶 G 580 ms I 570 ms T 9 ms S 0 kB R 0 kB to
🔶 G 569 ms I 557 ms T 10 ms S 0 kB R 0 kB the
🔶 G 569 ms I 559 ms T 8 ms S 0 kB R 0 kB left
🔶 G 532 ms I 531 ms T 0 ms S 0 kB R 0 kB of
🔶 G 534 ms I 532 ms T 0 ms S 0 kB R 0 kB the
🔶 G 573 ms I 566 ms T 5 ms S 0 kB R 0 kB E
🔶 G 567 ms I 557 ms T 8 ms S 0 kB R 0 kB iff
🔶 G 576 ms I 566 ms T 9 ms S 0 kB R 0 kB el
🔶 G 573 ms I 563 ms T 9 ms S 0 kB R 0 kB Tower
🔶 G 579 ms I 569 ms T 8 ms S 0 kB R 0 kB .
🔶 G 582 ms I 571 ms T 9 ms S 0 kB R 0 kB <|end_of_text|>
Generated tokens: 52
Avg generation time: 564.31 ms
Avg inference time: 556.67 ms
Avg transfer time: 6.17 ms
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Distributed Llama Version: 0.3.1
Model: Llama 3 8B Q40 (huggingface)
Switch: TP-Link LS1008G Switch
4 x Raspberry Pi 5 8GB
2 x Raspberry Pi 5 8GB
1 x Raspberry Pi 5 8GB
Beta Was this translation helpful? Give feedback.
All reactions