-
Notifications
You must be signed in to change notification settings - Fork 6.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Model instantiation without loading from disk #8327
Comments
Thank you for the request @AndreasKaratzas
It wouldn't completely address your situation, but is it feasible for you to create the scripted model (in Python) once and for all and just load that one file every time? I'm sorry that this is not the answer you're looking for, but I'm afraid the C++ models won't be coming back. For some context, we used to support direct C++ model access in the past, but the decision was made to deprecate that API in favour of torchscript. Whether this was a user-friendly decision in the long-term is up for debate, but the main reason at the time was that maintaining both the Python and C++ backend was just too much maintenance work. Some relevant past discussion: #4375 (comment) If that helps and if it's an option for you, those C++ models were removed in https://github.com/pytorch/vision/pull/6632/files so you'll find the original C++ implementations there (some of these models may have been updated in Python since then). |
Hello @NicolasHug, Thank you for your prompt response.
This would be a nice solution generally. However, to my problem, there are several parameters that need to be taken into account. My domain primarily evolves around embedded systems, where the workload changes rapidly and frequently. In particular, regarding the case of edge data centers, there are various clients with different needs who post different queries on the edge systems. Therefore, these constrained devices must be capable of quickly serving each client. I opted for I also reviewed the pull request referenced. As pointed out, the models there are old and point to a rather old release of Overall, I think that in the case of |
馃殌 The feature
So far, the only way to use a model from
torchvision
is through loading ajit
checkpoint from the disk like so:The feature that I would like to propose is to purge away the need to have a precompiled jit file and integrate a methodology in the C++ PyTorch frontend that can easily instantiate any
torchvision.models
file as easily as in Python. For example:Motivation, pitch
There shouldn't be any dependencies between the Python frontend and the C++ frontend. Specifically, there are projects that leverage the C++ PyTorch API solely, and in that case, the developers have to invoke every time a Python script before the utilization of their framework just to create an instance of the desired model from
torchvision.models
to then use their framework. This is a timely process, particularly if there is frequent model change at runtime.Specific use case:
I am building a framework that is connected with Torch TensorRT and utilizes NVIDIA NVDLAs of Jetson boards. However, every time I query my framework for some workload, I have to first use Python and compile a jit instance to later load in my framework. This creates a huge overhead and since disk operations are most timely, it defeats the whole purpose of using C++ to accelerate the process.
The text was updated successfully, but these errors were encountered: