[Bug]: During the process of transferring replicas, service interruptions may occur for both search and query operations.

[zhuwenxing]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:master-20240510-e2332bdc-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

error in client

[2024-05-10T10:20:14.636Z] [2024-05-10 10:20:12 - ERROR - pymilvus.decorators]: grpc RpcError: [hybrid_search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-05-10 10:20:02.154542', 'gRPC error': '2024-05-10 10:20:12.168731'}> (decorators.py:150)
[2024-05-10T10:20:14.636Z] [2024-05-10 10:20:12 - ERROR - pymilvus.decorators]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-05-10 10:20:02.158004', 'gRPC error': '2024-05-10 10:20:12.184097'}> (decorators.py:150)
[2024-05-10T10:20:14.636Z] [2024-05-10 10:20:12 - ERROR - pymilvus.decorators]: grpc RpcError: [query], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-05-10 10:20:02.156387', 'gRPC error': '2024-05-10 10:20:12.199975'}> (decorators.py:150)
[2024-05-10T10:20:14.636Z] [2024-05-10 10:20:12 - ERROR - ci_test]: Traceback (most recent call last):
[2024-05-10T10:20:14.636Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-05-10T10:20:14.636Z]     res = func(*args, **_kwargs)
[2024-05-10T10:20:14.636Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-05-10T10:20:14.636Z]     return func(*arg, **kwargs)
[2024-05-10T10:20:14.636Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 1078, in query
[2024-05-10T10:20:14.636Z]     return conn.query(
[2024-05-10T10:20:14.636Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 154, in handler
[2024-05-10T10:20:14.637Z]     raise e from e
[2024-05-10T10:20:14.637Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler
[2024-05-10T10:20:14.637Z]     return func(*args, **kwargs)
[2024-05-10T10:20:14.637Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 175, in handler
[2024-05-10T10:20:14.637Z]     return func(self, *args, **kwargs)
[2024-05-10T10:20:14.637Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 90, in handler
[2024-05-10T10:20:14.637Z]     raise e from e
[2024-05-10T10:20:14.637Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 86, in handler
[2024-05-10T10:20:14.637Z]     return func(*args, **kwargs)
[2024-05-10T10:20:14.637Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 1484, in query
[2024-05-10T10:20:14.637Z]     response = future.result()
[2024-05-10T10:20:14.637Z]   File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 881, in result
[2024-05-10T10:20:14.637Z]     raise self
[2024-05-10T10:20:14.637Z] grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
[2024-05-10T10:20:14.637Z] 	status = StatusCode.DEADLINE_EXCEEDED
[2024-05-10T10:20:14.637Z] 	details = "Deadline Exceeded"
[2024-05-10T10:20:14.637Z] 	debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2024-05-10T10:20:12.156732815+00:00", grpc_status:4}"
[2024-05-10T10:20:14.637Z] >
[2024-05-10T10:20:14.637Z]  (api_request.py:45)

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/resource_group_test/detail/resource_group_test/8/pipeline/

log: proxy log

Anything else?

may be the same as #32899

[zhuwenxing]

/assign @weiliu1031