Improvements to the INT8 GEMM portion of the code for Power

[ChipKerchner]

These are changes to improve GEMM portion of the code for Power.

There are 2 main code changes :

1. Changing a function to a template parameter so that operations that add/sub zero are eliminated at compile time. Plus reuse a vector that has the mask instead of rebuilding each time.
2. Add processing 16 columns at a time in MlasGemmQuantCopyPackB8x8 - this should reduce potential page faults by a factor of 4 and also be faster.
3. Unroll MlasQgemmStoreVectorMMA and vectorize other variables.

[yuslepukhin]

Cc: @chenfucn

[ChipKerchner]

Questions about the lint warnings. Do all lines have to be less than 120 characters? And for statements that span multiple lines, how are they written?

[yuslepukhin]

Typically, we employ clangformat. I use a visual cue in the IDE and break them manually, then run the formatter again.

[ChipKerchner]

Which style should I be using for clang-format? microsoft?

It seems the formatter wants to change a lot of code that I did not alter.

[yuslepukhin]

Which style should I be using for clang-format? microsoft?

It seems the formatter wants to change a lot of code that I did not alter.

Your editor should pick this up automatically.
https://github.com/microsoft/onnxruntime/blob/main/.clang-format

[yufenglee]

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline

[yufenglee]

/azp run Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Android CI Pipeline

[yufenglee]

/azp run iOS CI Pipeline,ONNX Runtime React Native CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[ChipKerchner]

I use vim as my editor. I'm not sure it will pickup lint formatting.

[yuslepukhin]

I use vim as my editor. I'm not sure it will pickup lint formatting.

Most of the failures about extra space. Lots of editors show non-visible characters.

[edgchen1]

can you share performance measurements to show how much improvement there is?

[ChipKerchner]

I'm seeing about a 2.6-4X improvement for PackB

[yufenglee]

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline

[yufenglee]

/azp run Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Android CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[ChipKerchner]

Can we move forward with this PR?

[yuslepukhin]

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline

[yuslepukhin]

/azp run Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Android CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).