Unexpected and random content-length in HTTP/1.0 responses with filter bwlim-out and 3.0-dev7-50d8c1-78 #2536
Comments
rnsanchez
added
status: needs-triage
|
Doing some rollbacks:
|
|
The issue here is about the bwlim filter mixed with |
capflam
added
status: reviewed
I'm doing this in order to stay on HTTP/1.0 + |
|
HTTP/1.0 without explicit |
|
When interfacing with Lighttpd, I have to manually remove, in order to get closer to our frequent use-case (end users in HTTP/1.0 mode, with unknown content-length). The case in this ticket is a major simplification of what we might see in production (where "might happen" usually turns out to be "millions of times" in our experience, so I try to not poke Murphy too much). What really caught my attention is that this usually worked without issues and, after recently going do 3.0dev, I found myself chasing ghosts until I had to check everything from the ground up---that's when I first caught the weird content-length. Rolling back to latest 2.8, working as usual. Right now, I have 2.8 interfacing an |
|
The behavior about updates of Here, the scenario you are describing seems reasonable but with no special processing about these during headers modifications, there are many way to produce invalid messages. For instance, changing the I will try to review all this part to mitigate this kind of issue, but I doubt it will be possible to catch all cases where these headers may be modified without changing the internal HTTP message state. And at the end, it seems safest to simply forbid these changes. What is exactly the issue by keeping this header in HTTP/1.0 in close mode ? EDIT: I asked because if there is a true use-case, may be we could add an option to workaround it. And I know you are not doing this kind of thing for arbitrary reason :) |
|
In my testing, I actually may have a If I could suggest anything with no regard to how much work this could cause, having a "http 1.0 mode" could be useful. Or at least some way of instructing HAProxy to trust its buffers while disregarding (removing) a |
|
Well it is not so easy because a regex may be used to remove a header or it may be removed from lua for instance. So in the end, it means name of every header removed must be tested. It is possible but a bit annoying. Everyone will pay for a marginal case. In this case, I prefer to add an option for this specific use-case. And in the same time, I should also improve the config parsing to trigger errors on this unexpected actions. So, if I understand well, you remove the header to reproduce an issue on your production when there is no |
I was afraid it turned out to be more complex (and indeed it did).
The former, yes: to reproduce locally what I cannot get even close as to what we see in production. It will be that odd-ball request/response that will shake things unexpectedly, and having it captured, I can pretend I have the same setup that unknown users A and B used in their exchange, while still being proxied by HAProxy. |
|
Just to let you know that we will talk about this issue with Willy on the next Monday to decide how we want to handle modifications of |
|
Please test with the latest lighttpd release, lighttpd 1.4.76. Among other things, HAProxy can use HTTP/2 to connect to lighttpd (which first added HTTP/2 support in lighttpd 1.4.56, but HTTP/2 is not enabled by default in lighttpd until lighttpd 1.4.59, and some HTTP/2 bugs in lighttpd have been fixed in the intervening 3 1/2 years since lighttpd 1.4.58) |
|
@rnsanchez, We discussed with Willy about this issue last week. We probably may improve the situation by allowing rewrites of the |
capflam
added
dev
|
Got it. On our end, we also face challenges when we have to adjust either of those (a lot more challenging for us when deadling with TE, though). |
…ages During the 2.9 dev cycle, to be able to support zero-copy data forwarding, a change on the H1 mux was performed to ignore the headers modifications about payload representation (Content-Length and Transfer-Encoding headers). It appears there are some use-cases where it could be handy to change values of these headers or just remove them. For instance, we can imagine to remove these headers on a server response to force the old HTTP/1.0 close mode behavior. So thaks to this patch, the rules are relaxed. It is now possible to remove these headers. When this happens, the following rules are applied: * If "Content-Length" header is removed but a "Transfer-Encoding: chunked" header is found, no special processing is performed. The message remains chunked. However the close mode is not forced. * If "Transfer-Encoding" header is removed but a "Content-Length" header is found, no special processing is performed. The payload length must comply to the specified content length. * If one of them is removed and the other one is not found, a response is switch the close mode and a "Content-Length: 0" header is forced on a request. With these rules, we fit the best to the user expectations. This patch depends on the following commit: * MINOR: mux-h1: Add a flag to ignore the request payload This patch should fix the issue #2536. It should be backported it to 2.9 with the commit above.
|
Just to let you know, I pushed a patch for this issue. It is planned to backport it to 2.9. You should now be able to remove It only affect the message representation on the outgoing side. On incoming side, the producer must still comply to the announced headers. |
|
As a side note, this only concerns the H1 multiplexer. I must still check other HTTP muxes (for instance FCGI), to be sure there is no issue for them. In addition, for the 3.1, I will try to invest some time to support, as far as possible, changes about these headers (For instance, by reducing the content-length value). But, this part remains a big foggy for now. |
capflam
added
status: feedback required
|
Fix confirmed on Thank you for your great work, as usual! :-) |
|
Thanks for the feedback, as usual ! :-) |
capflam
removed
dev
Detailed Description of the Problem
I was preparing to use HAProxy's bandwidth throttling for a testcase, and noticed an incorrect and unexpected behavior:
Notice
content-length: 0:Repeating; notice how close the content-length is to an internal buffer size:
Now simply disabling bwlimit filter, as suggested by @felipewd (the throttling is from lighttpd; I forgot to disable it, and I use it at 500KB/s), works fine:
HAProxy's output, upon repeating the requests above:
Now here's what strace witnessed (cropped; last line is particularly important):
Expected Behavior
I'd expect 1) a rate-limited response but more importantly 2) a sane response (no
content-length,connection: close).Steps to Reproduce the Behavior
Do you have any idea what may have caused this?
@felipewd guessed bwlim-out. I must agree, though I have not used it up until now.
Do you have an idea how to solve the issue?
No response
What is your configuration?
Output of
haproxy -vvLast Outputs and Backtraces
No response
Additional Information
No response
The text was updated successfully, but these errors were encountered: