Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NCCL error windows #202

Open
Vubni opened this issue Jan 4, 2024 · 1 comment
Open

NCCL error windows #202

Vubni opened this issue Jan 4, 2024 · 1 comment

Comments

@Vubni
Copy link

Vubni commented Jan 4, 2024

I usage: python train.py -c configs\ljs_base.json -m ljs_base
output:

DEBUG:numba.core.byteflow:bytecode dump:
>          0    NOP(arg=None, lineno=1051)
           2    LOAD_FAST(arg=0, lineno=1054)
           4    LOAD_CONST(arg=1, lineno=1054)
           6    BINARY_SUBSCR(arg=None, lineno=1054)
           8    LOAD_FAST(arg=0, lineno=1054)
          10    LOAD_CONST(arg=2, lineno=1054)
          12    BINARY_SUBSCR(arg=None, lineno=1054)
          14    COMPARE_OP(arg=4, lineno=1054)
          16    LOAD_FAST(arg=0, lineno=1054)
          18    LOAD_CONST(arg=1, lineno=1054)
          20    BINARY_SUBSCR(arg=None, lineno=1054)
          22    LOAD_FAST(arg=0, lineno=1054)
          24    LOAD_CONST(arg=3, lineno=1054)
          26    BINARY_SUBSCR(arg=None, lineno=1054)
          28    COMPARE_OP(arg=5, lineno=1054)
          30    BINARY_AND(arg=None, lineno=1054)
          32    RETURN_VALUE(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:pending: deque([State(pc_initial=0 nstack_initial=0)])
DEBUG:numba.core.byteflow:stack: []
DEBUG:numba.core.byteflow:state.pc_initial: State(pc_initial=0 nstack_initial=0)
DEBUG:numba.core.byteflow:dispatch pc=0, inst=NOP(arg=None, lineno=1051)
DEBUG:numba.core.byteflow:stack []
DEBUG:numba.core.byteflow:dispatch pc=2, inst=LOAD_FAST(arg=0, lineno=1054)
DEBUG:numba.core.byteflow:stack []
DEBUG:numba.core.byteflow:dispatch pc=4, inst=LOAD_CONST(arg=1, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$x2.0']
DEBUG:numba.core.byteflow:dispatch pc=6, inst=BINARY_SUBSCR(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$x2.0', '$const4.1']
DEBUG:numba.core.byteflow:dispatch pc=8, inst=LOAD_FAST(arg=0, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2']
DEBUG:numba.core.byteflow:dispatch pc=10, inst=LOAD_CONST(arg=2, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$x8.3']
DEBUG:numba.core.byteflow:dispatch pc=12, inst=BINARY_SUBSCR(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$x8.3', '$const10.4']
DEBUG:numba.core.byteflow:dispatch pc=14, inst=COMPARE_OP(arg=4, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$12binary_subscr.5']
DEBUG:numba.core.byteflow:dispatch pc=16, inst=LOAD_FAST(arg=0, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6']
DEBUG:numba.core.byteflow:dispatch pc=18, inst=LOAD_CONST(arg=1, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$x16.7']
DEBUG:numba.core.byteflow:dispatch pc=20, inst=BINARY_SUBSCR(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$x16.7', '$const18.8']
DEBUG:numba.core.byteflow:dispatch pc=22, inst=LOAD_FAST(arg=0, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9']
DEBUG:numba.core.byteflow:dispatch pc=24, inst=LOAD_CONST(arg=3, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$x22.10']
DEBUG:numba.core.byteflow:dispatch pc=26, inst=BINARY_SUBSCR(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$x22.10', '$const24.11']
DEBUG:numba.core.byteflow:dispatch pc=28, inst=COMPARE_OP(arg=5, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$26binary_subscr.12']
DEBUG:numba.core.byteflow:dispatch pc=30, inst=BINARY_AND(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$28compare_op.13']
DEBUG:numba.core.byteflow:dispatch pc=32, inst=RETURN_VALUE(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$30binary_and.14']
DEBUG:numba.core.byteflow:end state. edges=[]
DEBUG:numba.core.byteflow:-------------------------Prune PHIs-------------------------
DEBUG:numba.core.byteflow:Used_phis: defaultdict(<class 'set'>, {State(pc_initial=0 nstack_initial=0): set()})
DEBUG:numba.core.byteflow:defmap: {}
DEBUG:numba.core.byteflow:phismap: defaultdict(<class 'set'>, {})
DEBUG:numba.core.byteflow:changing phismap: defaultdict(<class 'set'>, {})
DEBUG:numba.core.byteflow:keep phismap: {}
DEBUG:numba.core.byteflow:new_out: defaultdict(<class 'dict'>, {})
DEBUG:numba.core.byteflow:----------------------DONE Prune PHIs-----------------------
DEBUG:numba.core.byteflow:block_infos State(pc_initial=0 nstack_initial=0):
AdaptBlockInfo(insts=((0, {}), (2, {'res': '$x2.0'}), (4, {'res': '$const4.1'}), (6, {'index': '$const4.1', 'target': '$x2.0', 'res': '$6binary_subscr.2'}), (8, {'res': '$x8.3'}), (10, {'res': '$const10.4'}), (12, {'index': '$const10.4', 'target': '$x8.3', 'res': '$12binary_subscr.5'}), (14, {'lhs': '$6binary_subscr.2', 'rhs': '$12binary_subscr.5', 'res': '$14compare_op.6'}), (16, {'res': '$x16.7'}), (18, {'res': '$const18.8'}), (20, {'index': '$const18.8', 'target': '$x16.7', 'res': '$20binary_subscr.9'}), (22, {'res': '$x22.10'}), (24, {'res': '$const24.11'}), (26, {'index': '$const24.11', 'target': '$x22.10', 'res': '$26binary_subscr.12'}), (28, {'lhs': '$20binary_subscr.9', 'rhs': '$26binary_subscr.12', 'res': '$28compare_op.13'}), (30, {'lhs': '$14compare_op.6', 'rhs': '$28compare_op.13', 'res': '$30binary_and.14'}), (32, {'retval': '$30binary_and.14', 'castval': '$32return_value.15'})), outgoing_phis={}, blockstack=(), active_try_block=None, outgoing_edgepushed={})
DEBUG:numba.core.interpreter:label 0:
    x = arg(0, name=x)                       ['x']
    $const4.1 = const(int, 0)                ['$const4.1']
    $6binary_subscr.2 = getitem(value=x, index=$const4.1, fn=<built-in function getitem>) ['$6binary_subscr.2', '$const4.1', 'x']
    $const10.4 = const(int, -1)              ['$const10.4']
    $12binary_subscr.5 = getitem(value=x, index=$const10.4, fn=<built-in function getitem>) ['$12binary_subscr.5', '$const10.4', 'x']
    $14compare_op.6 = $6binary_subscr.2 > $12binary_subscr.5 ['$12binary_subscr.5', '$14compare_op.6', '$6binary_subscr.2']
    $const18.8 = const(int, 0)               ['$const18.8']
    $20binary_subscr.9 = getitem(value=x, index=$const18.8, fn=<built-in function getitem>) ['$20binary_subscr.9', '$const18.8', 'x']
    $const24.11 = const(int, 1)              ['$const24.11']
    $26binary_subscr.12 = getitem(value=x, index=$const24.11, fn=<built-in function getitem>) ['$26binary_subscr.12', '$const24.11', 'x']
    $28compare_op.13 = $20binary_subscr.9 >= $26binary_subscr.12 ['$20binary_subscr.9', '$26binary_subscr.12', '$28compare_op.13']
    $30binary_and.14 = $14compare_op.6 & $28compare_op.13 ['$14compare_op.6', '$28compare_op.13', '$30binary_and.14']
    $32return_value.15 = cast(value=$30binary_and.14) ['$30binary_and.14', '$32return_value.15']
    return $32return_value.15                ['$32return_value.15']

DEBUG:numba.core.byteflow:bytecode dump:
>          0    NOP(arg=None, lineno=1057)
           2    LOAD_FAST(arg=0, lineno=1060)
           4    LOAD_CONST(arg=1, lineno=1060)
           6    BINARY_SUBSCR(arg=None, lineno=1060)
           8    LOAD_FAST(arg=0, lineno=1060)
          10    LOAD_CONST(arg=2, lineno=1060)
          12    BINARY_SUBSCR(arg=None, lineno=1060)
          14    COMPARE_OP(arg=0, lineno=1060)
          16    LOAD_FAST(arg=0, lineno=1060)
          18    LOAD_CONST(arg=1, lineno=1060)
          20    BINARY_SUBSCR(arg=None, lineno=1060)
          22    LOAD_FAST(arg=0, lineno=1060)
          24    LOAD_CONST(arg=3, lineno=1060)
          26    BINARY_SUBSCR(arg=None, lineno=1060)
          28    COMPARE_OP(arg=1, lineno=1060)
          30    BINARY_AND(arg=None, lineno=1060)
          32    RETURN_VALUE(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:pending: deque([State(pc_initial=0 nstack_initial=0)])
DEBUG:numba.core.byteflow:stack: []
DEBUG:numba.core.byteflow:state.pc_initial: State(pc_initial=0 nstack_initial=0)
DEBUG:numba.core.byteflow:dispatch pc=0, inst=NOP(arg=None, lineno=1057)
DEBUG:numba.core.byteflow:stack []
DEBUG:numba.core.byteflow:dispatch pc=2, inst=LOAD_FAST(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack []
DEBUG:numba.core.byteflow:dispatch pc=4, inst=LOAD_CONST(arg=1, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$x2.0']
DEBUG:numba.core.byteflow:dispatch pc=6, inst=BINARY_SUBSCR(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$x2.0', '$const4.1']
DEBUG:numba.core.byteflow:dispatch pc=8, inst=LOAD_FAST(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2']
DEBUG:numba.core.byteflow:dispatch pc=10, inst=LOAD_CONST(arg=2, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$x8.3']
DEBUG:numba.core.byteflow:dispatch pc=12, inst=BINARY_SUBSCR(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$x8.3', '$const10.4']
DEBUG:numba.core.byteflow:dispatch pc=14, inst=COMPARE_OP(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$12binary_subscr.5']
DEBUG:numba.core.byteflow:dispatch pc=16, inst=LOAD_FAST(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6']
DEBUG:numba.core.byteflow:dispatch pc=18, inst=LOAD_CONST(arg=1, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$x16.7']
DEBUG:numba.core.byteflow:dispatch pc=20, inst=BINARY_SUBSCR(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$x16.7', '$const18.8']
DEBUG:numba.core.byteflow:dispatch pc=22, inst=LOAD_FAST(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9']
DEBUG:numba.core.byteflow:dispatch pc=24, inst=LOAD_CONST(arg=3, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$x22.10']
DEBUG:numba.core.byteflow:dispatch pc=26, inst=BINARY_SUBSCR(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$x22.10', '$const24.11']
DEBUG:numba.core.byteflow:dispatch pc=28, inst=COMPARE_OP(arg=1, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$26binary_subscr.12']
DEBUG:numba.core.byteflow:dispatch pc=30, inst=BINARY_AND(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$28compare_op.13']
DEBUG:numba.core.byteflow:dispatch pc=32, inst=RETURN_VALUE(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$30binary_and.14']
DEBUG:numba.core.byteflow:end state. edges=[]
DEBUG:numba.core.byteflow:-------------------------Prune PHIs-------------------------
DEBUG:numba.core.byteflow:Used_phis: defaultdict(<class 'set'>, {State(pc_initial=0 nstack_initial=0): set()})
DEBUG:numba.core.byteflow:defmap: {}
DEBUG:numba.core.byteflow:phismap: defaultdict(<class 'set'>, {})
DEBUG:numba.core.byteflow:changing phismap: defaultdict(<class 'set'>, {})
DEBUG:numba.core.byteflow:keep phismap: {}
DEBUG:numba.core.byteflow:new_out: defaultdict(<class 'dict'>, {})
DEBUG:numba.core.byteflow:----------------------DONE Prune PHIs-----------------------
DEBUG:numba.core.byteflow:block_infos State(pc_initial=0 nstack_initial=0):
AdaptBlockInfo(insts=((0, {}), (2, {'res': '$x2.0'}), (4, {'res': '$const4.1'}), (6, {'index': '$const4.1', 'target': '$x2.0', 'res': '$6binary_subscr.2'}), (8, {'res': '$x8.3'}), (10, {'res': '$const10.4'}), (12, {'index': '$const10.4', 'target': '$x8.3', 'res': '$12binary_subscr.5'}), (14, {'lhs': '$6binary_subscr.2', 'rhs': '$12binary_subscr.5', 'res': '$14compare_op.6'}), (16, {'res': '$x16.7'}), (18, {'res': '$const18.8'}), (20, {'index': '$const18.8', 'target': '$x16.7', 'res': '$20binary_subscr.9'}), (22, {'res': '$x22.10'}), (24, {'res': '$const24.11'}), (26, {'index': '$const24.11', 'target': '$x22.10', 'res': '$26binary_subscr.12'}), (28, {'lhs': '$20binary_subscr.9', 'rhs': '$26binary_subscr.12', 'res': '$28compare_op.13'}), (30, {'lhs': '$14compare_op.6', 'rhs': '$28compare_op.13', 'res': '$30binary_and.14'}), (32, {'retval': '$30binary_and.14', 'castval': '$32return_value.15'})), outgoing_phis={}, blockstack=(), active_try_block=None, outgoing_edgepushed={})
DEBUG:numba.core.interpreter:label 0:
    x = arg(0, name=x)                       ['x']
    $const4.1 = const(int, 0)                ['$const4.1']
    $6binary_subscr.2 = getitem(value=x, index=$const4.1, fn=<built-in function getitem>) ['$6binary_subscr.2', '$const4.1', 'x']
    $const10.4 = const(int, -1)              ['$const10.4']
    $12binary_subscr.5 = getitem(value=x, index=$const10.4, fn=<built-in function getitem>) ['$12binary_subscr.5', '$const10.4', 'x']
    $14compare_op.6 = $6binary_subscr.2 < $12binary_subscr.5 ['$12binary_subscr.5', '$14compare_op.6', '$6binary_subscr.2']
    $const18.8 = const(int, 0)               ['$const18.8']
    $20binary_subscr.9 = getitem(value=x, index=$const18.8, fn=<built-in function getitem>) ['$20binary_subscr.9', '$const18.8', 'x']
    $const24.11 = const(int, 1)              ['$const24.11']
    $26binary_subscr.12 = getitem(value=x, index=$const24.11, fn=<built-in function getitem>) ['$26binary_subscr.12', '$const24.11', 'x']
    $28compare_op.13 = $20binary_subscr.9 <= $26binary_subscr.12 ['$20binary_subscr.9', '$26binary_subscr.12', '$28compare_op.13']
    $30binary_and.14 = $14compare_op.6 & $28compare_op.13 ['$14compare_op.6', '$28compare_op.13', '$30binary_and.14']
    $32return_value.15 = cast(value=$30binary_and.14) ['$30binary_and.14', '$32return_value.15']
    return $32return_value.15                ['$32return_value.15']

DEBUG:numba.core.byteflow:bytecode dump:
>          0    NOP(arg=None, lineno=1051)
           2    LOAD_FAST(arg=0, lineno=1054)
           4    LOAD_CONST(arg=1, lineno=1054)
           6    BINARY_SUBSCR(arg=None, lineno=1054)
           8    LOAD_FAST(arg=0, lineno=1054)
          10    LOAD_CONST(arg=2, lineno=1054)
          12    BINARY_SUBSCR(arg=None, lineno=1054)
          14    COMPARE_OP(arg=4, lineno=1054)
          16    LOAD_FAST(arg=0, lineno=1054)
          18    LOAD_CONST(arg=1, lineno=1054)
          20    BINARY_SUBSCR(arg=None, lineno=1054)
          22    LOAD_FAST(arg=0, lineno=1054)
          24    LOAD_CONST(arg=3, lineno=1054)
          26    BINARY_SUBSCR(arg=None, lineno=1054)
          28    COMPARE_OP(arg=5, lineno=1054)
          30    BINARY_AND(arg=None, lineno=1054)
          32    RETURN_VALUE(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:pending: deque([State(pc_initial=0 nstack_initial=0)])
DEBUG:numba.core.byteflow:stack: []
DEBUG:numba.core.byteflow:state.pc_initial: State(pc_initial=0 nstack_initial=0)
DEBUG:numba.core.byteflow:dispatch pc=0, inst=NOP(arg=None, lineno=1051)
DEBUG:numba.core.byteflow:stack []
DEBUG:numba.core.byteflow:dispatch pc=2, inst=LOAD_FAST(arg=0, lineno=1054)
DEBUG:numba.core.byteflow:stack []
DEBUG:numba.core.byteflow:dispatch pc=4, inst=LOAD_CONST(arg=1, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$x2.0']
DEBUG:numba.core.byteflow:dispatch pc=6, inst=BINARY_SUBSCR(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$x2.0', '$const4.1']
DEBUG:numba.core.byteflow:dispatch pc=8, inst=LOAD_FAST(arg=0, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2']
DEBUG:numba.core.byteflow:dispatch pc=10, inst=LOAD_CONST(arg=2, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$x8.3']
DEBUG:numba.core.byteflow:dispatch pc=12, inst=BINARY_SUBSCR(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$x8.3', '$const10.4']
DEBUG:numba.core.byteflow:dispatch pc=14, inst=COMPARE_OP(arg=4, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$12binary_subscr.5']
DEBUG:numba.core.byteflow:dispatch pc=16, inst=LOAD_FAST(arg=0, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6']
DEBUG:numba.core.byteflow:dispatch pc=18, inst=LOAD_CONST(arg=1, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$x16.7']
DEBUG:numba.core.byteflow:dispatch pc=20, inst=BINARY_SUBSCR(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$x16.7', '$const18.8']
DEBUG:numba.core.byteflow:dispatch pc=22, inst=LOAD_FAST(arg=0, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9']
DEBUG:numba.core.byteflow:dispatch pc=24, inst=LOAD_CONST(arg=3, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$x22.10']
DEBUG:numba.core.byteflow:dispatch pc=26, inst=BINARY_SUBSCR(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$x22.10', '$const24.11']
DEBUG:numba.core.byteflow:dispatch pc=28, inst=COMPARE_OP(arg=5, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$26binary_subscr.12']
DEBUG:numba.core.byteflow:dispatch pc=30, inst=BINARY_AND(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$28compare_op.13']
DEBUG:numba.core.byteflow:dispatch pc=32, inst=RETURN_VALUE(arg=None, lineno=1054)
DEBUG:numba.core.byteflow:stack ['$30binary_and.14']
DEBUG:numba.core.byteflow:end state. edges=[]
DEBUG:numba.core.byteflow:-------------------------Prune PHIs-------------------------
DEBUG:numba.core.byteflow:Used_phis: defaultdict(<class 'set'>, {State(pc_initial=0 nstack_initial=0): set()})
DEBUG:numba.core.byteflow:defmap: {}
DEBUG:numba.core.byteflow:phismap: defaultdict(<class 'set'>, {})
DEBUG:numba.core.byteflow:changing phismap: defaultdict(<class 'set'>, {})
DEBUG:numba.core.byteflow:keep phismap: {}
DEBUG:numba.core.byteflow:new_out: defaultdict(<class 'dict'>, {})
DEBUG:numba.core.byteflow:----------------------DONE Prune PHIs-----------------------
DEBUG:numba.core.byteflow:block_infos State(pc_initial=0 nstack_initial=0):
AdaptBlockInfo(insts=((0, {}), (2, {'res': '$x2.0'}), (4, {'res': '$const4.1'}), (6, {'index': '$const4.1', 'target': '$x2.0', 'res': '$6binary_subscr.2'}), (8, {'res': '$x8.3'}), (10, {'res': '$const10.4'}), (12, {'index': '$const10.4', 'target': '$x8.3', 'res': '$12binary_subscr.5'}), (14, {'lhs': '$6binary_subscr.2', 'rhs': '$12binary_subscr.5', 'res': '$14compare_op.6'}), (16, {'res': '$x16.7'}), (18, {'res': '$const18.8'}), (20, {'index': '$const18.8', 'target': '$x16.7', 'res': '$20binary_subscr.9'}), (22, {'res': '$x22.10'}), (24, {'res': '$const24.11'}), (26, {'index': '$const24.11', 'target': '$x22.10', 'res': '$26binary_subscr.12'}), (28, {'lhs': '$20binary_subscr.9', 'rhs': '$26binary_subscr.12', 'res': '$28compare_op.13'}), (30, {'lhs': '$14compare_op.6', 'rhs': '$28compare_op.13', 'res': '$30binary_and.14'}), (32, {'retval': '$30binary_and.14', 'castval': '$32return_value.15'})), outgoing_phis={}, blockstack=(), active_try_block=None, outgoing_edgepushed={})
DEBUG:numba.core.interpreter:label 0:
    x = arg(0, name=x)                       ['x']
    $const4.1 = const(int, 0)                ['$const4.1']
    $6binary_subscr.2 = getitem(value=x, index=$const4.1, fn=<built-in function getitem>) ['$6binary_subscr.2', '$const4.1', 'x']
    $const10.4 = const(int, -1)              ['$const10.4']
    $12binary_subscr.5 = getitem(value=x, index=$const10.4, fn=<built-in function getitem>) ['$12binary_subscr.5', '$const10.4', 'x']
    $14compare_op.6 = $6binary_subscr.2 > $12binary_subscr.5 ['$12binary_subscr.5', '$14compare_op.6', '$6binary_subscr.2']
    $const18.8 = const(int, 0)               ['$const18.8']
    $20binary_subscr.9 = getitem(value=x, index=$const18.8, fn=<built-in function getitem>) ['$20binary_subscr.9', '$const18.8', 'x']
    $const24.11 = const(int, 1)              ['$const24.11']
    $26binary_subscr.12 = getitem(value=x, index=$const24.11, fn=<built-in function getitem>) ['$26binary_subscr.12', '$const24.11', 'x']
    $28compare_op.13 = $20binary_subscr.9 >= $26binary_subscr.12 ['$20binary_subscr.9', '$26binary_subscr.12', '$28compare_op.13']
    $30binary_and.14 = $14compare_op.6 & $28compare_op.13 ['$14compare_op.6', '$28compare_op.13', '$30binary_and.14']
    $32return_value.15 = cast(value=$30binary_and.14) ['$30binary_and.14', '$32return_value.15']
    return $32return_value.15                ['$32return_value.15']

DEBUG:numba.core.byteflow:bytecode dump:
>          0    NOP(arg=None, lineno=1057)
           2    LOAD_FAST(arg=0, lineno=1060)
           4    LOAD_CONST(arg=1, lineno=1060)
           6    BINARY_SUBSCR(arg=None, lineno=1060)
           8    LOAD_FAST(arg=0, lineno=1060)
          10    LOAD_CONST(arg=2, lineno=1060)
          12    BINARY_SUBSCR(arg=None, lineno=1060)
          14    COMPARE_OP(arg=0, lineno=1060)
          16    LOAD_FAST(arg=0, lineno=1060)
          18    LOAD_CONST(arg=1, lineno=1060)
          20    BINARY_SUBSCR(arg=None, lineno=1060)
          22    LOAD_FAST(arg=0, lineno=1060)
          24    LOAD_CONST(arg=3, lineno=1060)
          26    BINARY_SUBSCR(arg=None, lineno=1060)
          28    COMPARE_OP(arg=1, lineno=1060)
          30    BINARY_AND(arg=None, lineno=1060)
          32    RETURN_VALUE(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:pending: deque([State(pc_initial=0 nstack_initial=0)])
DEBUG:numba.core.byteflow:stack: []
DEBUG:numba.core.byteflow:state.pc_initial: State(pc_initial=0 nstack_initial=0)
DEBUG:numba.core.byteflow:dispatch pc=0, inst=NOP(arg=None, lineno=1057)
DEBUG:numba.core.byteflow:stack []
DEBUG:numba.core.byteflow:dispatch pc=2, inst=LOAD_FAST(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack []
DEBUG:numba.core.byteflow:dispatch pc=4, inst=LOAD_CONST(arg=1, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$x2.0']
DEBUG:numba.core.byteflow:dispatch pc=6, inst=BINARY_SUBSCR(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$x2.0', '$const4.1']
DEBUG:numba.core.byteflow:dispatch pc=8, inst=LOAD_FAST(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2']
DEBUG:numba.core.byteflow:dispatch pc=10, inst=LOAD_CONST(arg=2, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$x8.3']
DEBUG:numba.core.byteflow:dispatch pc=12, inst=BINARY_SUBSCR(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$x8.3', '$const10.4']
DEBUG:numba.core.byteflow:dispatch pc=14, inst=COMPARE_OP(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$6binary_subscr.2', '$12binary_subscr.5']
DEBUG:numba.core.byteflow:dispatch pc=16, inst=LOAD_FAST(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6']
DEBUG:numba.core.byteflow:dispatch pc=18, inst=LOAD_CONST(arg=1, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$x16.7']
DEBUG:numba.core.byteflow:dispatch pc=20, inst=BINARY_SUBSCR(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$x16.7', '$const18.8']
DEBUG:numba.core.byteflow:dispatch pc=22, inst=LOAD_FAST(arg=0, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9']
DEBUG:numba.core.byteflow:dispatch pc=24, inst=LOAD_CONST(arg=3, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$x22.10']
DEBUG:numba.core.byteflow:dispatch pc=26, inst=BINARY_SUBSCR(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$x22.10', '$const24.11']
DEBUG:numba.core.byteflow:dispatch pc=28, inst=COMPARE_OP(arg=1, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$20binary_subscr.9', '$26binary_subscr.12']
DEBUG:numba.core.byteflow:dispatch pc=30, inst=BINARY_AND(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$14compare_op.6', '$28compare_op.13']
DEBUG:numba.core.byteflow:dispatch pc=32, inst=RETURN_VALUE(arg=None, lineno=1060)
DEBUG:numba.core.byteflow:stack ['$30binary_and.14']
DEBUG:numba.core.byteflow:end state. edges=[]
DEBUG:numba.core.byteflow:-------------------------Prune PHIs-------------------------
DEBUG:numba.core.byteflow:Used_phis: defaultdict(<class 'set'>, {State(pc_initial=0 nstack_initial=0): set()})
DEBUG:numba.core.byteflow:defmap: {}
DEBUG:numba.core.byteflow:phismap: defaultdict(<class 'set'>, {})
DEBUG:numba.core.byteflow:changing phismap: defaultdict(<class 'set'>, {})
DEBUG:numba.core.byteflow:keep phismap: {}
DEBUG:numba.core.byteflow:new_out: defaultdict(<class 'dict'>, {})
DEBUG:numba.core.byteflow:----------------------DONE Prune PHIs-----------------------
DEBUG:numba.core.byteflow:block_infos State(pc_initial=0 nstack_initial=0):
AdaptBlockInfo(insts=((0, {}), (2, {'res': '$x2.0'}), (4, {'res': '$const4.1'}), (6, {'index': '$const4.1', 'target': '$x2.0', 'res': '$6binary_subscr.2'}), (8, {'res': '$x8.3'}), (10, {'res': '$const10.4'}), (12, {'index': '$const10.4', 'target': '$x8.3', 'res': '$12binary_subscr.5'}), (14, {'lhs': '$6binary_subscr.2', 'rhs': '$12binary_subscr.5', 'res': '$14compare_op.6'}), (16, {'res': '$x16.7'}), (18, {'res': '$const18.8'}), (20, {'index': '$const18.8', 'target': '$x16.7', 'res': '$20binary_subscr.9'}), (22, {'res': '$x22.10'}), (24, {'res': '$const24.11'}), (26, {'index': '$const24.11', 'target': '$x22.10', 'res': '$26binary_subscr.12'}), (28, {'lhs': '$20binary_subscr.9', 'rhs': '$26binary_subscr.12', 'res': '$28compare_op.13'}), (30, {'lhs': '$14compare_op.6', 'rhs': '$28compare_op.13', 'res': '$30binary_and.14'}), (32, {'retval': '$30binary_and.14', 'castval': '$32return_value.15'})), outgoing_phis={}, blockstack=(), active_try_block=None, outgoing_edgepushed={})
DEBUG:numba.core.interpreter:label 0:
    x = arg(0, name=x)                       ['x']
    $const4.1 = const(int, 0)                ['$const4.1']
    $6binary_subscr.2 = getitem(value=x, index=$const4.1, fn=<built-in function getitem>) ['$6binary_subscr.2', '$const4.1', 'x']
    $const10.4 = const(int, -1)              ['$const10.4']
    $12binary_subscr.5 = getitem(value=x, index=$const10.4, fn=<built-in function getitem>) ['$12binary_subscr.5', '$const10.4', 'x']
    $14compare_op.6 = $6binary_subscr.2 < $12binary_subscr.5 ['$12binary_subscr.5', '$14compare_op.6', '$6binary_subscr.2']
    $const18.8 = const(int, 0)               ['$const18.8']
    $20binary_subscr.9 = getitem(value=x, index=$const18.8, fn=<built-in function getitem>) ['$20binary_subscr.9', '$const18.8', 'x']
    $const24.11 = const(int, 1)              ['$const24.11']
    $26binary_subscr.12 = getitem(value=x, index=$const24.11, fn=<built-in function getitem>) ['$26binary_subscr.12', '$const24.11', 'x']
    $28compare_op.13 = $20binary_subscr.9 <= $26binary_subscr.12 ['$20binary_subscr.9', '$26binary_subscr.12', '$28compare_op.13']
    $30binary_and.14 = $14compare_op.6 & $28compare_op.13 ['$14compare_op.6', '$28compare_op.13', '$30binary_and.14']
    $32return_value.15 = cast(value=$30binary_and.14) ['$30binary_and.14', '$32return_value.15']
    return $32return_value.15                ['$32return_value.15']

INFO:ljs_base:{'train': {'log_interval': 200, 'eval_interval': 1000, 'seed': 1234, 'epochs': 20000, 'learning_rate': 0.0002, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 64, 'fp16_run': True, 'lr_decay': 0.999875, 'segment_size': 8192, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'training_files': 'filelists/ljs_audio_text_train_filelist.txt.cleaned', 'validation_files': 'filelists/ljs_audio_text_val_filelist.txt.cleaned', 'text_cleaners': ['english_cleaners2'], 'max_wav_value': 32768.0, 'sampling_rate': 22050, 'filter_length': 1024, 'hop_length': 256, 'win_length': 1024, 'n_mel_channels': 80, 'mel_fmin': 0.0, 'mel_fmax': None, 'add_blank': True, 'n_speakers': 0, 'cleaned_text': True}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'n_layers_q': 3, 'use_spectral_norm': False}, 'model_dir': './logs\\ljs_base'}
WARNING:ljs_base:E:\g++\синтез is not a git repository, therefore hash value comparison will be ignored.
Traceback (most recent call last):
  File "train.py", line 290, in <module>
    main()
  File "train.py", line 50, in main
    mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,))
  File "C:\Users\egora\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\multiprocessing\spawn.py", line 246, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "C:\Users\egora\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\multiprocessing\spawn.py", line 202, in start_processes
    while not context.join():
  File "C:\Users\egora\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\multiprocessing\spawn.py", line 163, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "C:\Users\egora\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\multiprocessing\spawn.py", line 74, in _wrap
    fn(i, *args)
  File "E:\g++\синтез\train.py", line 62, in run
    dist.init_process_group(backend='nccl', init_method='env://', world_size=n_gpus, rank=rank)
  File "C:\Users\egora\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\distributed\c10d_logger.py", line 74, in wrapper
    func_return = func(*args, **kwargs)
  File "C:\Users\egora\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\distributed\distributed_c10d.py", line 1148, in init_process_group
    default_pg, _ = _new_process_group_helper(
  File "C:\Users\egora\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\distributed\distributed_c10d.py", line 1268, in _new_process_group_helper
    raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in
@Brian951
Copy link

NCCL is not supported on windows, you can change the backend for init_process_group from "nccl" to "gloo" to run this on windows; or run your training under a WSL Ubuntu environment (which supports NCCL) instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Brian951 @Vubni