I integrated an unbound instance as the DNS resolver of my homegrown CGNAT gateway yesterday. Hooked it up to the main (userland) routing process with a pair of veth devices so that all of the DNS-related networking went out the same path as the rest of the customer IP traffic and never touched the host network stack.
This worked great, except… only with some authoritative servers! The root servers worked fine, as did .com TLD, example.com resolved great… but things like google.com didn’t.
…this was a ”fun” day of debugging.
My thoughts immediately went to NAT bugs (the unbound process is behind NAT44/NAT66), MTU issues, routing problems at our network core, even rate limiting after my extended testing. But no, it was nothing like that.
Consistently, root servers worked, some other ones too, but most domain names would not resolve no matter what.
Turns out all of the outgoing unbound UDP packets had an invalid checksum.
When Linux sends packets out the veth, it doesn’t calculate proper checksums for them. They are, after all, useless on the local system and if the packets ever leave, the NIC hardware will fill them in.
But here I was, reading the packets directly in a userland process on the same system, not checking CHECKSUM_PARTIAL.
My NAT recalculates checksums incrementally, assuming they’re correct to begin with. Boom.
So what did we learn? That many authoritative DNS servers ignore UDP checksums!
@karppinen I also learned that there’s such a a thing as incremental checksum recalculation!
@pajp yeah, The Elders had this all figured out!
@pcdog that's what I thought at first yeah!
