Connection is haulted when connection id is retired

[Karthikdasari0423]

Detailed Description of the Problem

When the existing connection id is retired, server should accept new connection id and should continue the connection but after 3 seconds connection is getting haulted and server not accepting new connection id

Expected Behavior

Server should accept new connection id and should continue

Steps to Reproduce the Behavior

Maybe you cant repro in your setup locally
You need middle box, i can help to repro with your debug images

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

global
      log /dev/log    local0
      log /dev/log    local1 notice
      stats socket /run/haproxy/admin.sock mode 660 level admin
      stats timeout 30s
      user root
      group root
      daemon

      # Default SSL material locations
      ca-base /etc/ssl/certs
      crt-base /etc/ssl/private

      # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
      ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
      ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
      ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000

frontend www
   mode http
   bind :8090
   bind :4443  ssl crt /etc/ssl/certs/ssl-cert-snakeoil.pem alpn h2

   http-request set-header X-Forwarded-Proto https if { ssl_fc }
   # Redirects to HTTPS
   http-request redirect scheme https unless { ssl_fc }

   # enables HTTP/3 over QUIC
   bind quic4@:4443 ssl crt /etc/ssl/certs/ssl-cert-snakeoil.pem alpn h3
   bind quic6@:4443 ssl crt /etc/ssl/certs/ssl-cert-snakeoil.pem alpn h3

   # 'Alt-Svc' header invites client to switch to the QUIC protocol
   # Max age (ma) is set to 1 minute (60 seconds), but
   # can be increased once verified working as expected
   http-response set-header alt-svc "h3=\":4443\";ma=86400;"

   http-after-response add-header alt-svc "h3=\":4443\";ma=86400;"
   default_backend ehoneah-backend

backend ehoneah-backend
   mode http
   balance roundrobin
   server web-01 localhost:443 check

Output of haproxy -vv

HAProxy version 2.9.0 2023/12/05 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: http://www.haproxy.org/bugs/bugs-2.9.0.html
Running on: Linux 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT +PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 1.1.1t+quic  7 Feb 2023
Running on OpenSSL version : OpenSSL 1.1.1t+quic  7 Feb 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

No response

Additional Information

No response

[vkssv]

Hi Karthikdasari0423 !

Thanks for reporting this !

A quick question concerning your configuration, just to precise: for backend connections you don't have the intention to use SSL ? The standard SSL port number 443 (server web-01 localhost:443 check), on which your server is listening at localhost is used just for testing (I mean this server on the localhost is configured to receive clear HTTP traffic, in spite of listening on 443 ) ?

Could you, please, attach haproxy logs and the server's logs, corresponded to this behavior, and tcpdumps at the middle box side (where haproxy is installed) and at the server side will be very helpful.

Kind regards,

[a-denoyelle]

I'm not sure but by connection ID, I think you are referring to QUIC CID ?

[a-denoyelle]

If so, there is known limitation on our side as CID switching is linked to connection migration which is currently not supported by haproxy.

[Karthikdasari0423]

I'm not sure but by connection ID, I think you are referring to QUIC CID ?

yes, i am referring to QUIC CID

[Karthikdasari0423]

If so, there is known limitation on our side as CID switching is linked to connection migration which is currently not supported by haproxy.

okay, but i didnt found anything related to this in Known bugs page[https://www.haproxy.org/bugs/bugs-2.9.html]

[Karthikdasari0423]

Hi Karthikdasari0423 !

Thanks for reporting this !

A quick question concerning your configuration, just to precise: for backend connections you don't have the intention to use SSL ? The standard SSL port number 443 (server web-01 localhost:443 check), on which your server is listening at localhost is used just for testing (I mean this server on the localhost is configured to receive clear HTTP traffic, in spite of listening on 443 ) ?

Could you, please, attach haproxy logs and the server's logs, corresponded to this behavior, and tcpdumps at the middle box side (where haproxy is installed) and at the server side will be very helpful.

Kind regards,

yes, i dont have any intention to use SSL on my backend server as this is my LAB environment and it is used just for testing purposes.

i can attach servers logs and tcpdumps but not sure about haproxy logs cause it seems to me a bit tricky to collect HAProxy logs

[vkssv]

i can attach servers logs and tcpdumps but not sure about haproxy logs cause it seems to me a bit tricky to collect HAProxy logs

I think now it seems to be resolved, so we no longer need any further details. Thanks a lot !
We will update information about known limitations ASAP.

[Karthikdasari0423]

i can attach servers logs and tcpdumps but not sure about haproxy logs cause it seems to me a bit tricky to collect HAProxy logs

I think now it seems to be resolved, so we no longer need any further details. Thanks a lot ! We will update information about known limitations ASAP.

Thanks @vkssv @a-denoyelle