Weaviate backup using S3 closed pipe http 499 after 1m 3s

[djjeffr]

How to reproduce this bug?

The s3.PutObjectPart is trying to upload 250-270MB parts, something in backup code is closing pipe after 1m 3s.
You will need a slow s3 to reproduce error, we are using k8s minio.

What is the expected behavior?

It should not close pipe, the time out should be configurable to match proxy time out. This type of error should produce slow backend storage service.

What is the actual behavior?

It kills the pipe after six 499 failures (1m 3s x6).

Supporting information

We just get a closed pipe error or object write error.

Server Version

1.24.3

Code of Conduct

• I have read and agree to the Weaviate's Contributor Guide and Code of Conduct