Feature/support older intel mac book pro with gcc 13 #1085

nezda · 2023-12-27T11:37:41Z

Type of Change

Initiated discussion #1081 to while trying to get this project to compile on a 2019 Intel-based MacBook Pro.

Currently segfaults when running inference on mistralai/Mistral-7B-v0.1 quantized as WeightOnlyQuantConfig(ggml=True)

feature or bug fix or documentation or others
API changed or not

Description

detail description
JIRA ticket: xxx

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

2019 Intel-based MacBook Pro 2.3 GHz 8-Core Intel Core i9 on macOS 13.5 (22G74) using brew install gcc (13.2.0), export CC=/usr/local/opt/gcc/bin/gcc-13, export CXX=/usr/local/opt/gcc/bin/g++-13 and building with pip install -v .

Dependency Change?

any library dependency introduced or removed

…en/products/docs/accelerator-engines/what-is-intel-amx.html) anyway per intel#1081 (comment) Signed-off-by: Luke Nezda <lnezda@gmail.com>

…ツ)_/¯ Signed-off-by: Luke Nezda <lnezda@gmail.com>

Signed-off-by: Luke Nezda <lnezda@gmail.com>

…ntel#1081 (comment) Signed-off-by: Luke Nezda <lnezda@gmail.com>

…in this scope; did you mean 'SYS_sysctlbyname'? ld: library not found for -lrt collect2: error: ld returned 1 exit status [2/164] Building CXX object models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o FAILED: models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o /usr/local/opt/gcc/bin/g++-13 -DNE_GELU_USE_VEC -DNE_SIMD_VEC_DOT_F16 -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/core/. -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas -O3 -DNDEBUG -std=c++17 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk -fPIC -mf16c -mfma -mavx -mavx2 -fopenmp -MD -MT models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o -MF models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o.d -o models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o -c /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp: In function 'int32_t get_num_physical_cores()': /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp:36:16: error: 'sysctlbyname' was not declared in this scope; did you mean 'SYS_sysctlbyname'? 36 | int result = sysctlbyname("hw.perflevel0.physicalcpu", &num_physical_cores, &len, NULL, 0); | ^~~~~~~~~~~~ | SYS_sysctlbyname [3/164] Building CXX object models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o FAILED: models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o /usr/local/opt/gcc/bin/g++-13 -DNE_GELU_USE_VEC -DNE_SIMD_VEC_DOT_F16 -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/core/. -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas -O3 -DNDEBUG -std=c++17 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk -fPIC -mf16c -mfma -mavx -mavx2 -fopenmp -MD -MT models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o -MF models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o.d -o models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o -c /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp: In function 'int32_t get_num_physical_cores()': /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp:36:16: error: 'sysctlbyname' was not declared in this scope; did you mean 'SYS_sysctlbyname'? 36 | int result = sysctlbyname("hw.perflevel0.physicalcpu", &num_physical_cores, &len, NULL, 0); | ^~~~~~~~~~~~ | SYS_sysctlbyname Signed-off-by: Luke Nezda <lnezda@gmail.com>

Signed-off-by: Luke Nezda <lnezda@gmail.com>

…ents Signed-off-by: Luke Nezda <lnezda@gmail.com>

Signed-off-by: Luke Nezda <lnezda@gmail.com>

…s too Signed-off-by: Luke Nezda <lnezda@gmail.com>

…n_hidden_size, rope_scale, rope_theta - tried to fix write_vocab_only but Params here missing lots too Signed-off-by: Luke Nezda <lnezda@gmail.com>

DDEle

Great work!

DDEle · 2023-12-29T16:46:41Z

intel_extension_for_transformers/llm/runtime/graph/scripts/convert_llama.py

@zhenwei-intel Can you check it and check if there are similar problems in other models?

DDEle · 2023-12-29T16:47:10Z

intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py

@intellinjun Can you check it and check if there are similar problems in other models?

requirements.txt

intel_extension_for_transformers/llm/runtime/graph/scripts/requirements/common.txt

examples/huggingface/pytorch/question-answering/pruning/group_lasso/modeling.py

DDEle · 2023-12-29T16:58:14Z

docker/README.md

-docker pull intel/ai-tools:itrex-devel-1.3.0
+docker pull intel/ai-tools:itrex-1.3.0-devel


Thanks for the correction. I'm not familiar with docker but it seems that you are right. To be confirmed by @tylertitsworth?

DDEle · 2023-12-29T17:11:09Z

intel_extension_for_transformers/llm/library/jblas/jblas/kernel_jit.h

-    db(reinterpret_cast<uintptr_t>(nullptr), sizeof(intptr_t));  // case 0 should never occur
+    db(nullptr, sizeof(intptr_t));  // case 0 should never occur


copying your error msg

/Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/kernel_jit.h:1231:7: error: call of overloaded 'db(uintptr_t, long unsigned int)' is ambiguous 1231 | db(reinterpret_cast<uintptr_t>(nullptr), sizeof(intptr_t)); // case 0 should never occur | ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In file included from /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/jit_base.h:19, from /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/jit_blas_storage.h:15: /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/xbyak/xbyak.h:1133:14: note: candidate: 'void Xbyak::CodeArray::db(uint64_t, size_t)' 1133 | void db(uint64_t code, size_t codeSize) | ^~ /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/xbyak/xbyak.h:1129:14: note: candidate: 'void Xbyak::CodeArray::db(const uint8_t*, size_t)' 1129 | void db(const uint8_t *code, size_t codeSize) | ^~ ninja: build stopped: subcommand failed.

I actually want the first case. Can you try reinterpret_cast<uint64_t>(nullptr) (or just UINT64_C(0) if fails)?

For curiosity, what is uintptr_t in your side?

Hmm, original code gcc rejected with above was

db(reinterpret_cast<uintptr_t>(nullptr), sizeof(intptr_t)); // case 0 should never occur

my wild guess "fix" that is committed in this branch atm

db(nullptr, sizeof(intptr_t)); // case 0 should never occur

you've just requested

db(reinterpret_cast<uint64_t>(nullptr), sizeof(intptr_t)); // case 0 should never occur

which seems to compile fine ¯\_(ツ)_/¯ : I will push the change.

I re-confirmed the original doesn't compile with my gcc 13.2.0. I also confirmed I still get segfaults after re-building everything with this change.

As for what uintptr_t is in my env, I really don't know how to answer that - I asked Bard "What is the difference between uintptr_t and uint64_t ?" and it sounded fairly pedantic - like they could differ if we had 128 bit pointers or something.

…uirements/common.txt Co-authored-by: Yi DING <yi1.ding@intel.com> Signed-off-by: Luke Nezda <lnezda@gmail.com>

Co-authored-by: Yi DING <yi1.ding@intel.com> Signed-off-by: Luke Nezda <lnezda@gmail.com>

@DDEle

…ツ)_/¯ - adjust per code review @DDEle Signed-off-by: Luke Nezda <lnezda@gmail.com>

VincyZhang · 2024-05-13T06:06:50Z

@nezda @DDEle any update about this PR?

nezda force-pushed the feature/support-older-Intel-MacBook-Pro-with-gcc-13 branch 4 times, most recently from 63bd546 to 9c57593 Compare December 29, 2023 05:02

nezda added 13 commits December 29, 2023 09:32

Intel Mac's don't support [AMX](https://www.intel.com/content/www/us/…

00a4adb

…en/products/docs/accelerator-engines/what-is-intel-amx.html) anyway per intel#1081 (comment) Signed-off-by: Luke Nezda <lnezda@gmail.com>

gcc-13 said reinterpret_cast<uintptr_t>(nullptr) was ambiguous ¯\_(…

5f4e039

…ツ)_/¯ Signed-off-by: Luke Nezda <lnezda@gmail.com>

fix docker command

1b14304

Signed-off-by: Luke Nezda <lnezda@gmail.com>

disable the build of qbits and the deprecated executor on darwin per i…

f02ed60

…ntel#1081 (comment) Signed-off-by: Luke Nezda <lnezda@gmail.com>

don't link to -rt for APPLE either per intel#1081 (reply in thread)

46b64bd

Signed-off-by: Luke Nezda <lnezda@gmail.com>

typos

8f99e30

Signed-off-by: Luke Nezda <lnezda@gmail.com>

trying to make Intel mac work... (this didn't help?)

9087bfb

Signed-off-by: Luke Nezda <lnezda@gmail.com>

hacked requirements to strip "+cpu" suffix from torch==2.1.0 requirem…

5fe2be0

…ents Signed-off-by: Luke Nezda <lnezda@gmail.com>

hacked requirements to strip "+cpu" suffix from torch==2.1.0 requirem…

aae9843

…ents Signed-off-by: Luke Nezda <lnezda@gmail.com>

added missing rope_scale param

beb027b

Signed-off-by: Luke Nezda <lnezda@gmail.com>

attempt to deal with missing Linux-only os.sched_getaffinity for maco…

aaee454

…s too Signed-off-by: Luke Nezda <lnezda@gmail.com>

I think f_norm_eps should be rms_norm_eps; guessed still missing ff…

bfec292

…n_hidden_size, rope_scale, rope_theta - tried to fix write_vocab_only but Params here missing lots too Signed-off-by: Luke Nezda <lnezda@gmail.com>

nezda force-pushed the feature/support-older-Intel-MacBook-Pro-with-gcc-13 branch from c1f89be to bfec292 Compare December 29, 2023 15:54

DDEle reviewed Dec 29, 2023

View reviewed changes

nezda and others added 3 commits December 29, 2023 11:47

Update intel_extension_for_transformers/llm/runtime/graph/scripts/req…

4ff62db

…uirements/common.txt Co-authored-by: Yi DING <yi1.ding@intel.com> Signed-off-by: Luke Nezda <lnezda@gmail.com>

on mac (darwin) strip "+cpu" suffix from torch==2.1.0 requirements

cd05487

Co-authored-by: Yi DING <yi1.ding@intel.com> Signed-off-by: Luke Nezda <lnezda@gmail.com>

gcc-13 said reinterpret_cast<uintptr_t>(nullptr) was ambiguous ¯\_(…

05b66db

…ツ)_/¯ - adjust per code review @DDEle Signed-off-by: Luke Nezda <lnezda@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/support older intel mac book pro with gcc 13 #1085

Feature/support older intel mac book pro with gcc 13 #1085

nezda commented Dec 27, 2023 •

edited

DDEle left a comment

DDEle Dec 29, 2023

DDEle Dec 29, 2023

DDEle Dec 29, 2023

DDEle Dec 29, 2023

nezda Dec 29, 2023

VincyZhang commented May 13, 2024

		docker pull intel/ai-tools:itrex-devel-1.3.0
		docker pull intel/ai-tools:itrex-1.3.0-devel

		db(reinterpret_cast<uintptr_t>(nullptr), sizeof(intptr_t)); // case 0 should never occur
		db(nullptr, sizeof(intptr_t)); // case 0 should never occur

Feature/support older intel mac book pro with gcc 13 #1085

Are you sure you want to change the base?

Feature/support older intel mac book pro with gcc 13 #1085

Conversation

nezda commented Dec 27, 2023 • edited

Type of Change

Description

Expected Behavior & Potential Risk

How has this PR been tested?

Dependency Change?

DDEle left a comment

Choose a reason for hiding this comment

DDEle Dec 29, 2023

Choose a reason for hiding this comment

DDEle Dec 29, 2023

Choose a reason for hiding this comment

DDEle Dec 29, 2023

Choose a reason for hiding this comment

DDEle Dec 29, 2023

Choose a reason for hiding this comment

nezda Dec 29, 2023

Choose a reason for hiding this comment

VincyZhang commented May 13, 2024

nezda commented Dec 27, 2023 •

edited