DNS Server Error : 3

[mayankmurari]

I am intermittently getting 503 response from Kong with below error while calling upstream.

failure to get a peer from the ring-balancer

I have captured Kong logs at Debug level where I can see DNS Server Errors. Kong version being used is 3.4.2. Can someone please help why DNS resolution fails intermittently and how to fix this?

2024/04/19 13:10:34 [debug] 1262#0: *1034212 [lua] targets.lua:285: executing requery for: x.x.x.x.x.com
2024/04/19 13:10:34 [debug] 1262#0: *1034212 [lua] targets.lua:493: queryDns(): querying dns for x.x.x.x.x.com
2024/04/19 13:10:34 [warn] 1262#0: *1034212 [lua] targets.lua:504: queryDns(): querying dns for x.x.x.x.x.com failed: dns server error: 3 name error. Tried ["(short)x.x.x.x.x.com:(na) - cache-miss","x.x.x.x.x.com.ns.svc.cluster.local:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com.svc.cluster.local:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com.cluster.local:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com.x.com:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com.1234.x.com:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com:5 - cache-miss/scheduled/querying/1:ns2.p78.dns.x.net removed/1:ns3.p78.dns.x.net removed/1:ns4.p78.dns.x.net removed/1:ns1.p78.dns.x.net removed/dereferencing CNAME","(short)x-x-x.x.x-x-1.x.x.com:(na) - cache-miss","x-x-x.x.x-x-1.x.x.com:5 - cache-miss/scheduled/querying/1:ns2.p78.dns.x.net removed/1:ns1.x.x.x.net removed/1:ns3.x.x.x.net removed/1:ns4.x.x.x.net removed/dereferencing CNAME","(short)x-x.x.x-x-1.x.x.com:(na) - cache-miss","x-x.x.x-x-1.x.x.com:5 - cache-miss/scheduled/querying/1:ns4.x.x.x.net removed/1:ns3.x.x.x.net removed/1:ns1.x.x.x.net removed/1:ns2.x.x.x.net removed/dereferencing CNAME","(short)x-x-x.x.x-x-1.x.x.net:(na) - cache-miss","x-x-x.x.x-x-1.x.x.net:5 - cache-miss/scheduled/querying/dns client error: 101 empty record received","x-x-x.x.x-x-1.x.x.net.ns.svc.cluster.local:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.svc.cluster.local:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.cluster.local:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.x.com:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.0998.x.com:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net:33 - cache-hit/stale/scheduled/dns client error: 101 empty record received","x-x-x.x.x-x-1.x.x.net.ns.svc.cluster.local:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.x.svc.cluster.local:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.cluster.local:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.x.com:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.x.x.com:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net:1 - cache-hit/stale/scheduled/dns
2024/04/19 13:10:34 [debug] 1262#0: *1034212 [lua] targets.lua:400: f(): blank dns record for x.x.x.x.x.com, assuming A-record
2024/04/19 13:10:34 [debug] 1262#0: *1034212 [lua] targets.lua:408: f(): dns record type changed for x.x.x.x.x.com, nil -> 1
2024/04/19 13:10:34 [debug] 1262#0: *1034212 [lua] targets.lua:474: f(): updating balancer based on dns changes for x.x.x.x.x.com
2024/04/19 13:10:34 [debug] 1262#0: *1034212 [lua] targets.lua:484: f(): querying dns and updating for x.x.x.x.x.com completed
2024/04/19 13:11:02 [debug] 1261#0: *1030310 [lua] targets.lua:285: executing requery for: x.x.x.x.x.com
2024/04/19 13:11:02 [debug] 1261#0: *1030310 [lua] targets.lua:493: queryDns(): querying dns for x.x.x.x.x.com
2024/04/19 13:11:02 [debug] 1261#0: *1030310 [lua] targets.lua:408: f(): dns record type changed for x.x.x.x.x.com, nil -> 1
2024/04/19 13:11:02 [debug] 1261#0: *1030310 [lua] targets.lua:474: f(): updating balancer based on dns changes for x.x.x.x.x.com
2024/04/19 13:11:02 [debug] 1261#0: *1030310 [lua] targets.lua:484: f(): querying dns and updating for x.x.x.x.x.com completed
2024/04/19 13:11:03 [debug] 1261#0: *1033330 [lua] targets.lua:285: executing requery for: x.x.x.x.x.com
2024/04/19 13:11:03 [debug] 1261#0: *1033330 [lua] targets.lua:493: queryDns(): querying dns for x.x.x.x.x.com
2024/04/19 13:11:03 [debug] 1261#0: *1033330 [lua] targets.lua:445: f(): unchanged dns record entry for x.x.x.x.x.com: x.x.x.x:nil
2024/04/19 13:11:03 [debug] 1261#0: *1033330 [lua] targets.lua:484: f(): querying dns and updating for x.x.x.x.x.com completed

Adding more details from a failed transaction in Kong logs. The IP for the failed transaction with 503 seems to be getting removed.

"tries":[{"balancer_latency":0,"ip":"x.x.x.x","port":443,"balancer_latency_ns":46080,"balancer_start":1713770818488,"balancer_start_ns":1.7137708184887e+18}]
2024/04/22 07:30:39 [debug] 1280#0: *94581 [lua] targets.lua:458: f(): removed dns record entry for x-x-x-x.x.x.x.com: x.x.x.x:nil

[Water-Melon]

@mayankmurari Thank you for your feedback.

@chobits Could you take a look at this issue? Thanks.

[chobits]

2024/04/19 13:10:34 [warn] 1262#0: *1034212 [lua] targets.lua:504: queryDns(): querying dns for x.x.x.x.x.com failed: dns server error: 3 name error. Tried ["(short)x.x.x.x.x.com:(na) - cache-miss","x.x.x.x.x.com.ns.svc.cluster.local:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com.svc.cluster.local:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com.cluster.local:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com.x.com:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com.1234.x.com:5 - cache-miss/scheduled/querying/dns server error: 3 name error","x.x.x.x.x.com:5 - cache-miss/scheduled/querying/1:ns2.p78.dns.x.net removed/1:ns3.p78.dns.x.net removed/1:ns4.p78.dns.x.net removed/1:ns1.p78.dns.x.net removed/dereferencing CNAME","(short)x-x-x.x.x-x-1.x.x.com:(na) - cache-miss","x-x-x.x.x-x-1.x.x.com:5 - cache-miss/scheduled/querying/1:ns2.p78.dns.x.net removed/1:ns1.x.x.x.net removed/1:ns3.x.x.x.net removed/1:ns4.x.x.x.net removed/dereferencing CNAME","(short)x-x.x.x-x-1.x.x.com:(na) - cache-miss","x-x.x.x-x-1.x.x.com:5 - cache-miss/scheduled/querying/1:ns4.x.x.x.net removed/1:ns3.x.x.x.net removed/1:ns1.x.x.x.net removed/1:ns2.x.x.x.net removed/dereferencing CNAME","(short)x-x-x.x.x-x-1.x.x.net:(na) - cache-miss","x-x-x.x.x-x-1.x.x.net:5 - cache-miss/scheduled/querying/dns client error: 101 empty record received","x-x-x.x.x-x-1.x.x.net.ns.svc.cluster.local:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.svc.cluster.local:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.cluster.local:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.x.com:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.0998.x.com:5 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net:33 - cache-hit/stale/scheduled/dns client error: 101 empty record received","x-x-x.x.x-x-1.x.x.net.ns.svc.cluster.local:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.x.svc.cluster.local:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.cluster.local:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.x.com:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net.x.x.com:33 - cache-hit/stale/scheduled/dns server error: 3 name error","x-x-x.x.x-x-1.x.x.net:1 - cache-hit/stale/scheduled/dns

See this warn level log entry, it contains two important parts: the final result and Kong's attempt.

2024/04/19 13:10:34 [warn] 1262#0: *1034212 [lua] targets.lua:504: queryDns(): querying dns for x.x.x.x.x.com failed: dns server error: 3 name error ........

This part of the warn log entry indicates that the Kong DNS resolver made multiple attempts but ultimately received an NXDOMAIN response from the DNS server.

Tried [".....

From here, it logs all the attempts that kong made to resolve.

You could follow the attempts that kong made to detect if you could resolve the finnal address manually. (Or it might be an intermittent NXDOMAIN from the DNS server.)

Kong will traverse all the domain generated by search/domain option in resolv.conf and dns_order

[chobits]

BTW, for users who might not be familiar with the log details, I'll provide a simple explanation here. For example: "x-x-x.x.x-x-1.x.x.net.x.com:33 - cache-hit/stale/scheduled/dns server error: 3 name error"

• x-x-x.x.x-x-1.x.x.net.x.com means queried domain generated by original domain and search/domain options in resolv.conf;
• :33 means SRV;
• 3 name error means the DNS server replied NXDOMAIN;
• cache-hit/stale/scheduled/ shows us the internal logic for attempting domain:qtype query.

[mayankmurari]

@chobits Thanks for looking into it. We had some further observations from troubleshooting this error.

1. We tried nslookup multiple times from Kubernetes nodes where Kong is deployed and did not observe any >NXDOMAIN.
2. We tried dig from our corporate nameservers in a loop and did not observe any errors.
3. Below is the sequence from the logs where there are intermittent DNS Server 3 errors after querying DNS resulting in completion. Also the upstream IP which was present in Kong 503 logs has a removed dns entry message in log. This keeps on happening continuously while few requests results in 503 and then it start responding with 200 again.

Querying DNS for x.x.x.x : completed
Querying DNS for x.x.x.x : completed
unchanged DMS entry for x.x.x.x : x.x.x.x nil
Querying DNS for x.x.x.x failed : dns server error
new dns record entry for x-x-x-x.x.x.x.com: x.x.x.x:nil
removed dns record entry for x-x-x-x.x.x.x.com: x.x.x.x:nil

4. Does Kong DNS_Order specifically SRV,A has any role to play here?

[chobits]

1.We tried nslookup multiple times from Kubernetes nodes where Kong is deployed and did not observe any >NXDOMAIN.
2.We tried dig from our corporate nameservers in a loop and did not observe any errors.
3.Below is the sequence from the logs where there are intermittent DNS Server 3 errors after querying DNS resulting in completion. Also the upstream IP which was present in Kong 503 logs has a removed dns entry message in log. This keeps on happening continuously while few requests results in 503 and then it start responding with 200 again.
Querying DNS for x.x.x.x : completed
Querying DNS for x.x.x.x : completed
unchanged DMS entry for x.x.x.x : x.x.x.x nil
Querying DNS for x.x.x.x failed : dns server error
new dns record entry for x-x-x-x.x.x.x.com: x.x.x.x:nil
removed dns record entry for x-x-x-x.x.x.x.com: x.x.x.x:nil

If the issue persists consistently, I believe it's easy to reproduce using a client tool like nslookup or dig. If you still cannot reproduce it, you can simply capture the network packets using tcpdump for debugging purposes, like: sudo tcpdump -i any port 53.

And due to multiple attempts that Kong made, some of them might not yield the final result (IP address). Therefore, in the warn log, you just need to verify whether any one that you think should be resolved did not work as expected. For example, if your expected RRs with A type should be replied by the DNS, you dont need to care of the failed SRV attempts in the warn, just checking if some A type queries got NXDOMAIN, then you can use some dns client tool to check if that domain and A type could be resolved, or you can use tcpdump to capture the package to verify it.

Does Kong DNS_Order specifically SRV,A has any role to play here?

Kong will try to query the DNS server with the type specified by dns_order

For example, given dns_order=SRV,A,CNAME in kong.conf and search one.com two.com in resolv.conf, Kong will resolve the following queried type and domain for the example host:

example SRV
example.one.com SRV
example.two.com SRV
example A
example.one.com A
example.two.com A
example CNAME
example.one.com CNAME
example.two.com CNAME

(BTW, pls do not remove CNAME from dns_order, if your local DNS server replied RRs contain CNAME entries.