Inconsistent behaviors with SO_REUSEPORT on Linux and *BSD

[panjf2000]

SO_REUSEPORT enables duplicate address and port bindings across various
Unix-like OSs, whereas there is platform-specific inconsistency:
Linux implemented SO_REUSEPORT with load balancing for incoming connections
while *BSD implemented it for only binding to the same address and port, which
makes it pointless to enable SO_REUSEPORT on *BSD and Darwin for gnet with
multiple event-loops because only the first event-loop will be constantly woken
up to accept incoming connections while the rest of event-loops remain idle.
Thus, we disable SO_REUSEPORT on *BSD and Darwin by default.

Note that FreeBSD 12 introduced a new socket option named SO_REUSEPORT_LB
with the capability of load balancing, it's the equivalent of Linux's SO_REUSEPORT.

References

• The SO_REUSEPORT socket option
• SO_REUSEPORT_LB on FreeBSD
• How do SO_REUSEADDR and SO_REUSEPORT differ?

[kolinfluence]

@panjf2000 are these related?
#565
remarked here.

not sure if u have seen this repo
https://github.com/luyu6056/tls
https://github.com/luyu6056/gnet

even the http2 is working but i cant seem to get so-reuseport to work. no idea why even when i set it to true.

honestly, if i can get luyu6056's tls to work with so reuseport, i'll be contented with the older version use.

@panjf2000 should we wait for the tls on gnet? been waiting for this feature for years. but i'm not rushing u. i understand it's insane amount of work.

p.s. : i got it "working" with tls here but not with the soreuseport option... need soreuseport. can see if i can use it now on the older version?

tlsserver.go (cannot run it twice even with soreuseport option enabled)

package main

import (
        "log"
        "time"

        //"github.com/panjf2000/gnet"
        "github.com/kolinfluence/gnet"
        "github.com/luyu6056/tls"

)

type echoServer struct {
        *gnet.EventServer
}

func (es *echoServer) OnInitComplete(s gnet.Server) (action gnet.Action) {
        log.Printf("Echo server started on %s (loops: %d)", s.Addr, s.NumEventLoop)
        return
}

func (es *echoServer) React(frame []byte, c gnet.Conn) (action gnet.Action) {
        // Echo the incoming data back to the client
        c.AsyncWrite(frame)
        return
}

func main() {

        // Create TLS configuration using certificates
        tlsConfig := &tls.Config{
                Certificates:             make([]tls.Certificate, 1),
                PreferServerCipherSuites: true,
                MinVersion:               tls.VersionTLS12,
        }

        // Load server certificate and private key
        var err error
        tlsConfig.Certificates[0], err = tls.LoadX509KeyPair("./cert.pem", "./key.pem")
        if err != nil {
                log.Fatalf("Failed to load key pair: %v", err)
        }

        // Configure server options
        options := []gnet.Option{
                gnet.WithTCPKeepAlive(time.Minute * 5),
                gnet.WithTls(tlsConfig),
                gnet.WithReusePort(true),
                //gnet.WithMulticore(true),
        }

        // Start the server
        log.Fatal(gnet.Serve(new(echoServer), "tcp://:8443", options...))
}

tlsclient.go

package main

import (
    "bufio"
    "crypto/tls"
    "fmt"
    "log"
    "time"
)

func main() {
    // Connect to the server with TLS
    conf := &tls.Config{
        InsecureSkipVerify: true, // Use this for testing with self-signed certs
    }

    conn, err := tls.Dial("tcp", "localhost:8443", conf)
    if err != nil {
        log.Fatalf("Failed to connect to server: %v", err)
    }
    defer conn.Close()

    fmt.Println("Connected to server.")

    // Create a reader to read responses from the server
    reader := bufio.NewReader(conn)

    // Variables for counting requests and responses
    var requestsSent int
    var responsesReceived int

    // Start a ticker to display req/s every second
    ticker := time.NewTicker(1 * time.Second)
    go func() {
        for range ticker.C {
            fmt.Printf("Requests sent: %d, Responses received: %d\n", requestsSent, responsesReceived)
            // Reset counters
            requestsSent = 0
            responsesReceived = 0
        }
    }()

    // Loop indefinitely
    for {
        // Send "hello" to the server
        _, err := conn.Write([]byte("hello\n"))
        if err != nil {
            log.Fatalf("Failed to write to server: %v", err)
        }
        requestsSent++

        // Read the server's response
        data, err := reader.ReadString('\n')
        if err != nil {
            log.Fatalf("Failed to read from server: %v", err)
        }
        responsesReceived++
        fmt.Printf("data = %s\n",data)
    }
}

[kolinfluence]

when tlsserver is running...

GOMAXPROCS=1 ./tlsclient 
Connected to server.
Requests sent: 40142, Responses received: 40141
Requests sent: 57849, Responses received: 57849
Requests sent: 61713, Responses received: 61713
Requests sent: 61986, Responses received: 61986
Requests sent: 60308, Responses received: 60308
Requests sent: 56197, Responses received: 56197
Requests sent: 63092, Responses received: 63092
Requests sent: 60763, Responses received: 60763
Requests sent: 59409, Responses received: 59409
Requests sent: 59875, Responses received: 59875
Requests sent: 61441, Responses received: 61441

showing req/s only. very good in fact. just need soreuseport working ...

package main

import (
    "bufio"
    "crypto/tls"
    "fmt"
    "log"
    "time"
)

func main() {
    // Connect to the server with TLS
    conf := &tls.Config{
        InsecureSkipVerify: true, // Use this for testing with self-signed certs
    }

    conn, err := tls.Dial("tcp", "localhost:8443", conf)
    if err != nil {
        log.Fatalf("Failed to connect to server: %v", err)
    }
    defer conn.Close()

    fmt.Println("Connected to server.")

    // Create a reader to read responses from the server
    reader := bufio.NewReader(conn)

    // Variables for counting requests and responses
    var requestsSent int
    var responsesReceived int

    // Start a ticker to display req/s every second
    ticker := time.NewTicker(1 * time.Second)
    go func() {
        for range ticker.C {
            fmt.Printf("Requests sent: %d, Responses received: %d\n", requestsSent, responsesReceived)
            // Reset counters
            requestsSent = 0
            responsesReceived = 0
        }
    }()

    // Loop indefinitely
    for {
        // Send "hello" to the server
        _, err := conn.Write([]byte("hello\n"))
        if err != nil {
            log.Fatalf("Failed to write to server: %v", err)
        }
        requestsSent++

        // Read the server's response
        _, err = reader.ReadString('\n')
        if err != nil {
            log.Fatalf("Failed to read from server: %v", err)
        }
        responsesReceived++
        //fmt.Printf("data = %s\n",data)
    }
}

[panjf2000]

Did it work on Linux? If so, then it's irrelevant.

[kolinfluence]

@panjf2000 ubuntu 22.04. amd ryzen 5
anyway... i definitely need so_reuseport. if u cant get the tls working in 5 days, possible to take a quick look at the repos and see why it cant run more than 1 instance? it's strange coz i've already set so_reuseport to true and in the package it is also enabled... not sure why.

[kolinfluence]

this tls on "old version" is working so fine now in fact (other than so reuseport) that i can really live with it if so reuseport is enabled.

i sound so desperate for this feature coz i've stuck with even worst results from this:
https://github.com/0-haha/gnet-tls-go1-20/

i really wont be asking for a lot. just tls working with so reuseport... if u know what i mean from testing gnet for so long and needing this tls + so reuseport so badly.

[kolinfluence]

@panjf2000 also, do u know if you made my version work with so_reuseport, "officially" this is a very good working version of tls for gnet (older version but still gnet)

... and then i will try to figure out how to make it work for https and then http2 coz it's already working with the luyu6056 version, just need to distill down to the actual usage guide

[panjf2000]

I'd suggest you look for other alternatives if you're so urgent.