DoH queries failing with CloudFlare and Google DNS

[nerdseeder]

Hi everyone,

I am using PiHole that forward DNS queries to DoH proxy. I am seeing regular DNS resolution failures and DNS query take time to resolve. I tried both cloudflared dns proxy and DNScrypt. I am guessing these providers throttle DNS queries for users. PiHole and cloudflared are running on same instance (Rasberry Pi 4 4G model). I am also observing too many PTR queries coming from ntopng. I will try disabling IP resolution query in ntopng.

cat /etc/default/cloudflared
CLOUDFLARED_OPTS=--metrics 0.0.0.0:8888 --address 0.0.0.0 --port 5555  --upstream https://dns.google/dns-query --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query --bootstrap https://162.159.36.1/dns-query --bootstrap https://162.159.46.1/dns-query

tail cloudflared.log
[2020-09-21T13:04:41.452147937Z]: failed to connect to an HTTPS backend "https://dns.google/dns-query": failed to perform an HTTPS request: Post "https://dns.google/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2020-09-21T13:04:46.453392112Z]: failed to connect to an HTTPS backend "https://1.1.1.1/dns-query": failed to perform an HTTPS request: Post "https://1.1.1.1/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2020-09-21T13:04:51.453996111Z]: failed to connect to an HTTPS backend "https://1.0.0.1/dns-query": failed to perform an HTTPS request: Post "https://1.0.0.1/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2020-09-21T13:06:03.605122099Z]: failed to connect to an HTTPS backend "https://dns.google/dns-query": failed to perform an HTTPS request: Post "https://dns.google/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2020-09-21T13:06:03.686881543Z]: failed to connect to an HTTPS backend "https://dns.google/dns-query": failed to perform an HTTPS request: Post "https://dns.google/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2020-09-21T13:06:16.224977328Z]: failed to connect to an HTTPS backend "https://dns.google/dns-query": failed to perform an HTTPS request: Post "https://dns.google/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2020-09-21T13:06:16.404906181Z]: failed to connect to an HTTPS backend "https://dns.google/dns-query": failed to perform an HTTPS request: Post "https://dns.google/dns-query": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2020-09-21T13:06:25.707826222Z]: failed to connect to an HTTPS backend "https://dns.google/dns-query": failed to perform an HTTPS request: Post "https://dns.google/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2020-09-21T13:06:26.450271402Z]: failed to connect to an HTTPS backend "https://dns.google/dns-query": failed to perform an HTTPS request: Post "https://dns.google/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2020-09-21T13:06:26.511473732Z]: failed to connect to an HTTPS backend "https://dns.google/dns-query": failed to perform an HTTPS request: Post "https://dns.google/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

root@rspi-4:/var/log# grep SERVFAIL pihole.log | tail
Sep 21 18:54:07 dnsmasq[15057]: reply error is SERVFAIL
Sep 21 19:38:36 dnsmasq[1187]: reply error is SERVFAIL
Sep 21 19:48:24 dnsmasq[1187]: reply error is SERVFAIL
Sep 21 19:50:45 dnsmasq[1187]: reply error is SERVFAIL
Sep 21 19:51:38 dnsmasq[1187]: reply error is SERVFAIL
Sep 21 19:51:42 dnsmasq[1187]: reply error is SERVFAIL
Sep 21 19:53:15 dnsmasq[1187]: reply error is SERVFAIL
Sep 21 19:53:16 dnsmasq[1187]: reply error is SERVFAIL
Sep 21 19:53:21 dnsmasq[1187]: reply error is SERVFAIL
Sep 21 19:54:08 dnsmasq[1187]: reply error is SERVFAIL

root@rspi-4:/var/log# grep -c SERVFAIL pihole.log
3672

root@rspi-4:/var/log# grep -c 'query\[PTR\]' pihole.log
55773

I also tried using unbound but seeing issue with it while resolving deep CNAME records.

 

[JB701]

Does 1.1.1.1 as an ip load on your browser?

 

[nerdseeder]

yeah works fine.

curl -H 'accept: application/dns-json' 'https://1.1.1.1/dns-query?name=google.com&type=A'                                                                                                nischay@Nischays-MacBook-Pro
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":false,"CD":false,"Question":[{"name":"google.com","type":1}],"Answer":[{"name":"google.com","type":1,"TTL":286,"data":"216.58.197.78"}]}

 

[dkdhanda8]

I had cloudflared crashing for my rpi multiple times and have read multiple times about its instability , been using DNS crypt and having 0 issues with cloudflare doh.

 

[nerdseeder]

Yeah, I tried DNScrypt. It is also failing to resolve queries. I stopped PTR queries from ntopng and looks like it improved DNS resolution. As I mentioned I was observing 55k PTR queries per day.

 

[JB701]

You should disable Ntopng when not diagnosing issues. It takes up a lot of CPU and RAM

 

---

[royalroy]

Hey, I think cloudfare is facing issues in India. Can anyone traceroute and post their results? Mine is going to Singapore instead of the usual Mumbai extreme-ix -> Cloudfare. Latency is thus, very high, like 80 ms instead of the usual 30-40 ms.

 

[Adithya]

They've been going through singapore for some time now. There's multiple issues raised on their own forums which gets ignored.

 

[nerdseeder]

@royalroy

From BSNL Fiber - Bengaluru
#################################
root@ubuntu-server-1:~# mtr -r -c 5 1.1.1.1
Start: 2020-09-22T18:38:30+0530
HOST: ubuntu-server-1             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 59.x.x.x                0.0%     5    2.7   2.8   2.7   3.1   0.2
  2.|-- 218.248.160.1              0.0%     5    3.1   3.0   2.7   3.4   0.2
  3.|-- 218.248.124.38             0.0%     5    3.6   3.8   3.6   3.9   0.1
  4.|-- 117.216.207.216            0.0%     5    3.4   3.4   3.1   3.5   0.1
  5.|-- 117.216.207.217            0.0%     5    3.5   3.5   3.4   3.6   0.1
  6.|-- 125.17.82.125              0.0%     5   52.4  53.5  50.9  60.9   4.1
  7.|-- 182.79.208.202             0.0%     5   51.5  55.1  51.4  63.0   4.7
  8.|-- 182.79.161.171             0.0%     5   93.0  63.2  54.1  93.0  16.7
  9.|-- one.one.one.one            0.0%     5   52.7  52.9  52.5  53.6   0.4


From MetroNet Bengaluru (Bell Tele Services)
###################################
root@rspi4-8gb:~# mtr -r -c 5 1.1.1.1
Start: 2020-09-22T13:10:04+0000
HOST: rspi4-8gb                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 172.x.x.x              0.0%     5    0.6   0.6   0.6   0.7   0.1
  2.|-- 172.x.x.x               0.0%     5    5.9  12.7   4.8  34.5  12.7
  3.|-- 43.254.160.41.static.bell  0.0%     5    1.4   1.3   1.2   1.4   0.1
  4.|-- static-225.8.98.14-tataid  0.0%     5    2.7   3.4   2.7   4.0   0.6
  5.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
  6.|-- 10.117.135.249             0.0%     5   31.4  30.6  26.2  34.8   3.7
  7.|-- 121.240.238.37.static-ban  0.0%     5    3.7   3.9   3.7   4.1   0.1
  8.|-- 172.31.167.46             80.0%     5   11.7  11.7  11.7  11.7   0.0
  9.|-- 115.114.85.222            80.0%     5   77.1  77.1  77.1  77.1   0.0
10.|-- 115.114.85.241            20.0%     5  150.0 149.8 149.5 150.0   0.2
11.|-- if-ae-34-2.tcore1.svq-sin 20.0%     5  176.6 178.2 176.6 180.3   1.8
12.|-- 120.29.215.101            80.0%     5  146.2 146.2 146.2 146.2   0.0
13.|-- one.one.one.one           20.0%     5   66.7  67.8  66.7  69.2   1.2

 

[Deleted member 77424]

I am also using cloudflared for DoH on my pihole. Since last week cloudflare was unreachable or having dns query times in 1000s of ms.

Was getting this error on checking cloudflared process status

failed to connect to an HTTPS backend "https://1.1.1.1/dns-query": failed to perform an HTTPS request: Post "https://1.1.1.1/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Changed dns from https://1.1.1.1/dns-query to https://dns.google/dns-query in cloudflared. Everything ok so far.
Cloudflare used to ping mumbai or chennai (within 25-35ms, now it's going to singapore with average of 90ms)
Something must have happened to Cloudflare Indian servers.

 

[nerdseeder]

You should disable Ntopng when not diagnosing issues. It takes up a lot of CPU and RAM

Yeah, that is true.

 

[Adithya]

Weird how there is no official explaination or anything from CF yet considering they have been transparent with most of their issues.

 

[nerdseeder]

I am also using cloudflared for DoH on my pihole. Since last week cloudflare was unreachable or having dns query times in 1000s of ms.

Was getting this error on checking cloudflared process status

failed to connect to an HTTPS backend "https://1.1.1.1/dns-query": failed to perform an HTTPS request: Post "https://1.1.1.1/dns-query": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Changed dns from https://1.1.1.1/dns-query to https://dns.google/dns-query in cloudflared. Everything ok so far.
Cloudflare used to ping mumbai or chennai (within 25-35ms, now it's going to singapore with average of 90ms)
Something must have happened to Cloudflare Indian servers.
View attachment 4570

I would recommend using DNSCrypt proxy. It is more reliable and gives you more fine tune control. I observed many issues with Cloudflared DNS proxy.

Installation

dnscrypt-proxy 2 - A flexible DNS proxy, with support for encrypted DNS protocols. - DNSCrypt/dnscrypt-proxy

github.com

 

[JB701]

I think Airtel users get <5ms pings to 1.1.1.1 and Jio users get 40ms pings in Delhi. Maybe it's something with TATA route?

 

[royalroy]

Well, on my Mom's Jio 4G, Cloudfare traffic still goes to Chennai. However, with Jio fiber, many are reporting that it goes to Singapore. @JB701 Does Jio have some pact with Tata then?