test_large_mmaped_weights_non_abi_compatible_cuda unit test failure

[nWEIdia]

🐛 Describe the bug

Encountered the following test failures during CI testing
==================================== RERUNS ====================================
_ AOTInductorTestNonABICompatibleCuda.test_large_mmaped_weights_non_abi_compatible_cuda _
Traceback (most recent call last):
File "/usr/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/lib/python3.10/unittest/case.py", line 591, in run
self._callTestMethod(testMethod)
File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
method()
File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 2759, in wrapper
method(*args, **kwargs)
File "/opt/pytorch/pytorch/test/inductor/test_torchinductor.py", line 9818, in new_test
return value(self)
File "/opt/pytorch/pytorch/test/inductor/test_aot_inductor.py", line 288, in test_large_mmaped_weights
self.check_model(Model(), example_inputs)
File "/opt/pytorch/pytorch/test/inductor/test_aot_inductor.py", line 107, in check_model
actual = AOTIRunnerUtil.run(
File "/opt/pytorch/pytorch/test/inductor/test_aot_inductor_utils.py", line 107, in run
optimized = AOTIRunnerUtil.load(device, so_path)
File "/opt/pytorch/pytorch/test/inductor/test_aot_inductor_utils.py", line 88, in load
return torch._export.aot_load(so_path, device)
File "/usr/local/lib/python3.10/dist-packages/torch/_export/init.py", line 405, in aot_load
runner = torch.C.aoti.AOTIModelContainerRunnerCuda(so_path, 1, device) # type: ignore[assignment, call-arg]
RuntimeError: create_func( &container_handle, num_models, device_str.c_str(), cubin_dir.empty() ? nullptr : cubin_dir.c_str()) API call failed at /opt/pytorch/pytorch/torch/csrc/inductor/aoti_runner/model_container_runner.cpp, line 49

To execute this test, run the following from the base repo dir:
python test/inductor/test_aot_inductor.py -k test_large_mmaped_weights_non_abi_compatible_cuda

cc @desertfire @chenyang78 @eqy @ptrblck @tinglvv @xwang233

Versions

nightly

[nWEIdia]

It started to show up on April 20th test, possibly a regression around these days.