server not responding #4222
Comments
|
The mixtral model is quite large, and it looks like your GPU has 80G available, so I'm guessing this is a cloud instance. We've seen that cloud instances can have slow I/O and take quite a long time to load larger models. We currently have a 10m timeout set in the code, but have heard some users report it taking longer than 10m to fully load larger models in some cases. Did you see it hit the 10m timeout eventually? You might also want to open another terminal into your instance and check performance metrics. |
|
Hello @dhiltgen, thank you for your answer, ollama now detects the GPU but inference is extremely slow (about 2 words in 10 minutes). The setup I have is a A100-80GB GPU virtualized with nvidia GRID, with driver version 525 and CUDA 12. GPU usage is at 0%, but it is technically using it, otherwise it would be NA. Would you have any guidance on how to proceed ? Thanks again |
|
I would like to add that it's best to completely disable any swap memory. This has tremendously helped me so far. |
|
@thomassrour it sounds like it did eventually load, but took a long time to do so. Once it loaded, you didn't see performance you were expecting. Can you share the server log (or check for the line that looks something like this Also, if you upgrade to 0.1.38 (just shipped) we've added an |
|
Actually most users that run into issue of not using enough layers can occur if num_batch size is too small or too high. |
What is the issue?
Hello,
I have trouble reaching my ollama container. I have tried using the images for 0.1.32 and 0.1.33, as some users reported bugs 0.1.33 but it doesn't work on either. Here is the output of docker logs, when trying mixtral (I have also tried llama3, same result) :
time=2024-05-07T07:33:21.130Z level=INFO source=images.go:817 msg="total blobs: 10"
time=2024-05-07T07:33:21.134Z level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-05-07T07:33:21.135Z level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)"
time=2024-05-07T07:33:21.136Z level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama3873501864/runners
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx2
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cuda_v11
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/rocm_v60002
time=2024-05-07T07:33:25.457Z level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]"
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-05-07T07:33:25.457Z level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-05-07T07:33:25.457Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
time=2024-05-07T07:33:25.457Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3873501864/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers//libcudart.so /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]"
time=2024-05-07T07:33:25.462Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0]"
wiring cudart library functions in /tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0
dlsym: cudaSetDevice
dlsym: cudaDeviceSynchronize
dlsym: cudaDeviceReset
dlsym: cudaMemGetInfo
dlsym: cudaGetDeviceCount
dlsym: cudaDeviceGetAttribute
dlsym: cudaDriverGetVersion
CUDA driver version: 12-0
time=2024-05-07T07:33:25.495Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-05-07T07:33:25.495Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[0] CUDA totalMem 85895020544
[0] CUDA freeMem 79269199872
time=2024-05-07T07:33:25.661Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.0"
releasing cudart library
time=2024-05-07T07:33:30.257Z level=DEBUG source=gguf.go:57 msg="model = &llm.gguf{containerGGUF:(llm.containerGGUF)(0xc000426c80), kv:llm.KV{}, tensors:[]llm.Tensor(nil), parameters:0x0}"
time=2024-05-07T07:33:30.522Z level=DEBUG source=gguf.go:193 msg="general.architecture = llama"
time=2024-05-07T07:33:30.528Z level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-05-07T07:33:30.528Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so"
time=2024-05-07T07:33:30.528Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3873501864/runners/cuda/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers//libcudart.so /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]"
time=2024-05-07T07:33:30.529Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0]"
wiring cudart library functions in /tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0
dlsym: cudaSetDevice
dlsym: cudaDeviceSynchronize
dlsym: cudaDeviceReset
dlsym: cudaMemGetInfo
dlsym: cudaGetDeviceCount
dlsym: cudaDeviceGetAttribute
dlsym: cudaDriverGetVersion
CUDA driver version: 12-0
time=2024-05-07T07:33:30.530Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-05-07T07:33:30.530Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[0] CUDA totalMem 85895020544
[0] CUDA freeMem 79269199872
time=2024-05-07T07:33:30.674Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.0"
releasing cudart library
time=2024-05-07T07:33:30.719Z level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-05-07T07:33:30.719Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
time=2024-05-07T07:33:30.719Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3873501864/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers//libcudart.so /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]"
time=2024-05-07T07:33:30.720Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0]"
wiring cudart library functions in /tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0
dlsym: cudaSetDevice
dlsym: cudaDeviceSynchronize
dlsym: cudaDeviceReset
dlsym: cudaMemGetInfo
dlsym: cudaGetDeviceCount
dlsym: cudaDeviceGetAttribute
dlsym: cudaDriverGetVersion
CUDA driver version: 12-0
time=2024-05-07T07:33:30.721Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-05-07T07:33:30.721Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[0] CUDA totalMem 85895020544
[0] CUDA freeMem 79269199872
time=2024-05-07T07:33:30.869Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.0"
releasing cudart library
time=2024-05-07T07:33:30.916Z level=INFO source=server.go:127 msg="offload to gpu" reallayers=33 layers=33 required="26042.6 MiB" used="26042.6 MiB" available="75597.0 MiB" kv="256.0 MiB" fulloffload="184.0 MiB" partialoffload="935.0 MiB"
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx2
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cuda_v11
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/rocm_v60002
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx2
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cuda_v11
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/rocm_v60002
time=2024-05-07T07:33:30.916Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-07T07:33:30.918Z level=DEBUG source=server.go:259 msg="LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/tmp/ollama3873501864/runners/cuda_v11"
time=2024-05-07T07:33:30.918Z level=INFO source=server.go:264 msg="starting llama server" cmd="/tmp/ollama3873501864/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-e9e56e8bb5f0fcd4860675e6837a8f6a94e659f5fa7dce6a1076279336320f2b --ctx-size 2048 --batch-size 512 --embedding --log-format json --n-gpu-layers 33 --verbose --port 33787"
time=2024-05-07T07:33:30.919Z level=INFO source=server.go:389 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"WARN","line":2494,"msg":"server.cpp is not built with verbose logging.","tid":"140005422133248","timestamp":1715067210}
time=2024-05-07T07:33:30.970Z level=DEBUG source=server.go:420 msg="server not yet available" error="health resp: Get "http://127.0.0.1:33787/health\": dial tcp 127.0.0.1:33787: connect: connection refused"
{"build":1,"commit":"7593639","function":"main","level":"INFO","line":2819,"msg":"build info","tid":"140005422133248","timestamp":1715067210}
{"function":"main","level":"INFO","line":2822,"msg":"system info","n_threads":4,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | ","tid":"140005422133248","timestamp":1715067210,"total_threads":8}
llama_model_loader: loaded meta data with 26 key-value pairs and 995 tensors from /root/.ollama/models/blobs/sha256-e9e56e8bb5f0fcd4860675e6837a8f6a94e659f5fa7dce6a1076279336320f2b (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = mistralai
llama_model_loader: - kv 2: llama.context_length u32 = 32768
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
llama_model_loader: - kv 4: llama.block_count u32 = 32
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 9: llama.expert_count u32 = 8
llama_model_loader: - kv 10: llama.expert_used_count u32 = 2
llama_model_loader: - kv 11: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 12: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 13: general.file_type u32 = 2
llama_model_loader: - kv 14: tokenizer.ggml.model str = llama
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32000] = ["", "
", "", "<0x00>", "<...llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,58980] = ["▁ t", "i n", "e r", "▁ a", "h e...
llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 24: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv 25: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type f16: 32 tensors
llama_model_loader: - type q4_0: 833 tensors
llama_model_loader: - type q8_0: 64 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 8
llm_load_print_meta: n_expert_used = 2
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8x7B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 46.70 B
llm_load_print_meta: model size = 24.62 GiB (4.53 BPW)
llm_load_print_meta: general.name = mistralai
llm_load_print_meta: BOS token = 1 '
''llm_load_print_meta: EOS token = 2 '
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: GRID A100D-80C, compute capability 8.0, VMM: no
time=2024-05-07T07:33:31.220Z level=DEBUG source=server.go:420 msg="server not yet available" error="server not responding"
llm_load_tensors: ggml ctx size = 0.96 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CUDA_Host buffer size = 70.31 MiB
llm_load_tensors: CUDA0 buffer size = 25145.55 MiB
..................................................time=2024-05-07T07:35:23.693Z level=DEBUG source=server.go:420 msg="server not yet available" error="health resp: Get "http://127.0.0.1:33787/health": dial tcp 127.0.0.1:33787: i/o timeout"
time=2024-05-07T07:35:23.894Z level=DEBUG source=server.go:420 msg="server not yet available" error="server not responding"
`
Thanks for your help
OS
Docker
GPU
Nvidia
CPU
No response
Ollama version
0.1.32, 0.1.33
The text was updated successfully, but these errors were encountered: