Gnet cannot support large value(>10K) redis benchmark test

[slow-zhang]

Describe the bug
I try to implement a redis-server by gnet, but I cannot test by redis-benchmark

cmd: redis-benchmark -h <host> -p<port> -t set,get -n 1000000 -d 1000000 -r 1000000

how I implement it in React:

1. read cmd in frame
   // async
2. using immemeory map set/get
3. return value
   

React: React(frame []byte, c gnet.Conn) (out []byte, action gnet.Action)
To Reproduce
just run bench

Expected behavior
it can test and show the result

Screenshots
the Screenshot shows only when -d (data size) is small it can run

System Info (please complete the following information):

• OS (e.g. Ubuntu 16.04 & osx1.15):
• Go version (e.g. Go 1.16):
• gnet version (e.g. v1.6.1):

Additional context
the code is simple, if you need, I can upload later

[slow-zhang]

how i start gnet: log.Fatal(gnet.Serve(echo, dir, gnet.WithMulticore(true), gnet.WithReadBufferCap(8*1024*1024)))

[panjf2000]

I don't think you have a full understanding of gnet, I'd recommend you to use gnet.ICodec to implement your redis server, parse and process the redis packets one by one, instead of returning gnet.Close when there is an incomplete packet.

[slow-zhang]

can i pass the connection reader to the goredis.NewRespReader?

[slow-zhang]

i find that the example of http is like redis. but I am confused because line 61 returns directly if there are some incomplete requests, will this cause the next error
for example, take ABC as a complete req
user: send ABC
Decode: A: invalid
Decode: BC: invalid

[slow-zhang]

another question:
i want an inmemory redis-server using map, it will have a lock on the map. what will I do the map operator?
in React? but it don't know how the decode str to react
in Decode?
could you provide some examples for it?

[panjf2000]

When you find there is an incomplete packet, you should return it without any error, in that case, gnet will buffer these data and continue to read data from socket, if you return gnet.Close like you did in your example, the connection will be closed, so take more time to understand gnet before you develop anything on top of it.

[panjf2000]

If you are trying to develop a redis server with gnet, take a reference from https://github.com/IceFireDB/redhub.

[slow-zhang]

how gnet know it needs to handle a req instead of read it the next time?
the interface decode/encode is hard to use? for decode read data and write to out, it seems all work will be done in encode
.
form your reply I assume if out is empty the encode will be ignore and data will be send to buffer again?

[slow-zhang]

i find that the reference you provided is not used gnet.ICodec. Do you know why?

[panjf2000]

how gnet know it needs to handle a req instead of read it the next time?

If the Decode() returns a nil []byte, gnet will continue to read data from socket.

[panjf2000]

i find that the reference you provided is not used gnet.ICodec. Do you know why?

gnet.ICode is just a recommended way to do it, you can also put all your code in React(), as long as you ensure you're doing it correctly.

[slow-zhang]

If the Decode() returns a nil []byte, gnet will continue to read data from socket.

but since we read the data and reset the buf before, will some data lost?

[slow-zhang]

will this possible:
take ABC as a complete req
user: send ABC

1. Decode: A: invalid, return nil
2. Decode: BC: invalid

will the 1. write A back to socket to make the 2. be a complete request ?

[slow-zhang]

this http example will discard incomplete req for it reset buffers

I find if we discard only when we handled data it will be fine

[slow-zhang]

but it still cannot pass the large value benchmark

the cmd /usr/local/opt/redis@3.2/bin/redis-benchmark -t set,get -n 3 -d 100000 -r 100000 -c 1 -p <port> will be slow.
by debug i found there are always some incomplete req to the decode so it hangs for a long time

[slow-zhang]

create other issue about this #297