Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

secretflow-lite==1.3.0.dev20231212 安装后测试ray异常 #1087

Open
hy928302776 opened this issue Dec 19, 2023 · 8 comments
Open

secretflow-lite==1.3.0.dev20231212 安装后测试ray异常 #1087

hy928302776 opened this issue Dec 19, 2023 · 8 comments
Assignees
@ian-huu @zhouaihui @6fj @anakinxc

Comments

@hy928302776
Copy link

Issue Type

Bug

Source

binary

Secretflow Version

secretflow-lite==1.3.0.dev20231212

OS Platform and Distribution

mac/m1 8c16g

Python version

3.8.13

Bazel version

No response

GCC/Compiler version

No response

What happend and What you expected to happen.

在mac/m1系统中安装secretflow环境：
pip install secretflow-lite==1.3.0.dev20231212
执行测试案例，ray读取log日志异常

Reproduction code to reproduce the issue.

执行测试案例

import secretflow as sf

sf.init(parties=['alice', 'bob'], address='local')
alice_device = sf.PYU('alice')
message_from_alice = alice_device(lambda x: x)("Hello World!")
print(message_from_alice)

出现异常：

/Users/huangying/miniconda3/envs/sf/bin/python /Users/huangying/PycharmProjects/django-vue-admin-main/backend/dvadmin/test.py 
2023-12-19 14:08:39,340	ERROR services.py:1195 -- Failed to start the dashboard: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2023-12-19_14-08-38_184512_70104/logs/dashboard.log'
2023-12-19 14:08:39,340	ERROR services.py:1196 -- Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2023-12-19_14-08-38_184512_70104/logs/dashboard.log'
Traceback (most recent call last):
  File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/services.py", line 1167, in start_api_server
    with open(dashboard_log, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ray/session_2023-12-19_14-08-38_184512_70104/logs/dashboard.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/services.py", line 1178, in start_api_server
    raise Exception(err_msg + f"\nFailed to read dashboard log: {e}")
Exception: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2023-12-19_14-08-38_184512_70104/logs/dashboard.log'
2023-12-19 14:08:39,472	INFO worker.py:1538 -- Started a local Ray instance.
[2023-12-19 14:08:48,634 E 70104 27111613] core_worker_process.cc:219: Failed to get the system config from raylet because it is dead. Worker will terminate. Status: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details:  .Please see `raylet.out` for more details.

Process finished with exit code 1
@hy928302776
Copy link
Author

image

@xiushuiguande
Copy link

@hy928302776
1、
pip show ray
pip show secretflow-ray
看下是否是重复装了ray/secretflow-ray
2、检查下 /tmp/ray/session_latest/sockets 是否存在以及权限是否能正常访问；

如果没问题的话麻烦提供下 Ray 的日志，在 /tmp/ray/session_latest/logs 目录下

@hy928302776
Copy link
Author

image image

没看到日志文件：
image

console的日志是：
/Users/huangying/miniconda3/envs/sf/bin/python /Users/huangying/PycharmProjects/django-vue-admin-main/backend/dvadmin/test.py
2023-12-19 16:48:52,160 ERROR services.py:1195 -- Failed to start the dashboard: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2023-12-19_16-48-51_021816_95797/logs/dashboard.log'
2023-12-19 16:48:52,160 ERROR services.py:1196 -- Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2023-12-19_16-48-51_021816_95797/logs/dashboard.log'
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/services.py", line 1167, in start_api_server
with open(dashboard_log, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ray/session_2023-12-19_16-48-51_021816_95797/logs/dashboard.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/services.py", line 1178, in start_api_server
raise Exception(err_msg + f"\nFailed to read dashboard log: {e}")
Exception: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2023-12-19_16-48-51_021816_95797/logs/dashboard.log'
2023-12-19 16:48:52,294 INFO worker.py:1538 -- Started a local Ray instance.
[2023-12-19 16:48:52,440 E 95797 27312831] core_worker.cc:179: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

Process finished with exit code 1

@hy928302776
Copy link
Author

看错了有日志文件：
image
ka看哪个呢

@xiushuiguande
Copy link

@hy928302776 把raylet.err发一下

@hy928302776
Copy link
Author

~
1、python-core-driver-01000000ffffffffffffffffffffffffffffffffffffffffffffffff_95797.log的内容
~
"python-core-driver-01000000ffffffffffffffffffffffffffffffffffffffffffffffff_95797.log" 3L, 470B
[2023-12-19 16:48:52,298 I 95797 27312831] core_worker_process.cc:107: Constructing CoreWorkerProcess. pid: 95797
[2023-12-19 16:48:52,299 I 95797 27312831] io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2023-12-19 16:48:52,440 E 95797 27312831] core_worker.cc:179: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory
~

2、raylet.err的内容
(sf) huangying@MacBook-Pro-2 logs % cat raylet.err
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
from grpc import _compression
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
from grpc._cython import cygrpc
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/dashboard/agent.py", line 11, in
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
[2023-12-19 16:48:52,311 E 95830 27313209] (raylet) agent_manager.cc:135: The raylet exited immediately because the Ray agent failed. The raylet fate shares with the agent. This can happen because the Ray agent was unexpectedly killed or failed. See dashboard_agent.log for the root cause.
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
from grpc import _compression
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
from grpc._cython import cygrpc
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
from grpc import _compression
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
from grpc._cython import cygrpc
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
from grpc import _compression
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
from grpc._cython import cygrpc
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
from grpc import _compression
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
from grpc._cython import cygrpc
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
from grpc import _compression
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
from grpc._cython import cygrpc
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
from grpc import _compression import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in

File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
import grpc # noqa: F401
from grpc import _compression
from grpc._cython import cygrpc
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
from grpc._cython import cygrpc
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle' from grpc import _compression
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
from grpc._cython import cygrpc
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
raise ImportError(
ImportError: import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
from grpc import _compression
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
from grpc._cython import cygrpc
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 63, in _configure_system
import grpc # noqa: F401
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/init.py", line 22, in
from grpc import _compression
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_compression.py", line 20, in
from grpc._cython import cygrpc
ImportError: dlopen(/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_kCFStreamPropertySocketNativeHandle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/_private/workers/default_worker.py", line 8, in
import ray
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 101, in
_configure_system()
File "/Users/huangying/miniconda3/envs/sf/lib/python3.8/site-packages/ray/init.py", line 65, in _configure_system
raise ImportError(
ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.

@xiushuiguande
Copy link

ImportError: Failed to import grpc on Apple Silicon. On Apple Silicon machines, try pip uninstall grpcio; conda install grpcio. Check out https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support for more details.
@hy928302776 根据报错提示，在ray 的installation.html尝试解决一下

@hy928302776
Copy link
Author

可以了，谢谢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ian-huu ian-huu

@zhouaihui zhouaihui

@6fj 6fj

@anakinxc anakinxc

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ian-huu @hy928302776 @xiushuiguande @zhouaihui @6fj @anakinxc