[QUESTION]: How to reduce pymilvus package size?

[mahaveer08]

Is there an existing issue for this?

• I have searched the existing issues

What is your question?

I need to build my script so where I used pymilvus package and I want to install locally zip it then upload to lambda but the size of folders seems 400MB so lambda doesn’t accept more than 250MB file size so how to reduce the package size and what are the supported sub packages as it installed lot of supportive packages?

Anything else?

No response

[mahaveer08]

@XuanYang-cn any inputs on this?

[XuanYang-cn]

@mahaveer08 From my test, pymilvus.tar.gz only uses 1.2MB, what else does your folders contain? It seems that 400MB contains not only pymilvus but many other things.

[mahaveer08]

List of all supported packages like below:
Pyarrow, urllib3, azure, grpc, numpy, pandas, pymilvus, setuptools etc. so these kind of packages get added when you run “pip install pymilvus -t .” While building the folder for lambda function.
So only pymilvus.tar.gz is enough to support ?

[mahaveer08]

Below packages installed after running "pip install pymilvus -t ."

Successfully installed argon2-cffi-23.1.0 argon2-cffi-bindings-21.2.0 azure-core-1.30.1 azure-storage-blob-12.19.1 boto3-1.34.79 botocore-1.34.79 certifi-2024.2.2 cffi-1.16.0 charset-normalizer-3.3.2 cryptography-42.0.5 environs-9.5.0 grpcio-1.60.0 idna-3.6 isodate-0.6.1 jmespath-1.0.1 marshmallow-3.21.1 minio-7.2.5 numpy-1.26.4 packaging-24.0 pandas-2.2.1 protobuf-5.26.1 pyarrow-15.0.2 pycparser-2.22 pycryptodome-3.20.0 pymilvus-2.4.0 python-dateutil-2.9.0.post0 python-dotenv-1.0.1 pytz-2024.1 requests-2.31.0 s3transfer-0.10.1 scipy-1.13.0 setuptools-69.2.0 six-1.16.0 typing-extensions-4.11.0 tzdata-2024.1 ujson-5.9.0 urllib3-1.26.18

Let me know what are the necessary dependencies need to install from above @XuanYang-cn ?

[XuanYang-cn]

@mahaveer08 They are all essentials for pymilvus to run.

What do you mean when you said "build the folder for lambda function"? What's the difference to use pymilvus as lambda function to normal usage by pip install pymilvus?

[mahaveer08]

I have a Python script that inserts data into a Milvus database using the pymilvus package. To utilize this script on a Lambda function, I need to compress it along with the pymilvus folder, ensuring that the folder size remains below 250MB.

[mahaveer08]

Anyhow I deployed using container image rather than zipping the pymilvus package folder.