If you block all bots using robots.txt and do not link it anywhere on the main domain, can anyone know a subdomain exists on your website?

@pratik @smokey not unless the domain servers allows AXFR — otherwise yeah brute force is the only way to guess at it which would be a DNS attack

@prologic @smokey Thanks. Guess unless I run for office, I don’t have to be worried about discovery. As long as it’s not discovered through normal search engines, I’m good. I didn’t know if there are any tools that let you see all the sub domains if you enter the main domain.

@pratik The only subdomain I was able to find via search was "jot" (which I know has been linked to from elsewhere).

@joejenett Oh well. Coz I have linked it on here. But how did you find that?

@pratik Using your main domain name in quotation marks (on Google). I've linked to Jottings a few times but those posts didn't show up in the search results.

@joejenett Interesting. I just searched too. My mb profile shows up too. Also, a site called Serendeputy that somehow indexed my jot blog

@pratik Maybe. Some scripts like Wordpress might be set to ping search engines every time you make a new post. That would be a giveaway that a site exists on a subdomain. Likewise WP may be set to ping Wordpress.com for the WP Reader, another giveaway.

As a rule, Google treats subdomains as stand alone websites as if they had their own domain. But, as a domain registrar, Google knows when a domain is registered and will attempt to check. It has no way of knowing about new subdomians unless it finds an inbound link or a ping of some sort. But give is one inbound link and it will find you. Still it should respect robots.txt.

@bradenslen Interesting. I'm not on WP but on Blot and I confirmed that it doesn't send trackbacks even to the sites I link to. Although I have set up robots.txt to block all bots, Google still manages to index the URLs but not the description. That kinda renders the blocking moot.

@pratik Check this: www.searchenginejournal.com/google-ro...

Google is no longer supporting noindex in robots.txt but is supporting disallow.

More info from Google: support.google.com/webmaster...

@bradenslen Ah! No wonder. Thanks for the pointer. I just added the meta tag for noindex to my sites. BTW Google no longers allows access to the Webmasters Tools dashboard without a G Suites account.

@pratik DNS is not a secure protocol, so any time you visit your obscure subdomain the DNS provider doing that lookup for you will know about it and will cache the result for a while. Also, anyone on that same network could “observe” the DNS conversation and see the subdomain.

@pratik So it depends upon how paranoid you want to be, really. No reputable robots-respecting search engine would index your private subdomain this way, but unprotected WiFi sometimes attract snoops gathering data for who knows what purpose. Why VPNs are recommended.

@fgtech unless you uSe DNSSec of DNS over HTTPS

@fgtech True. Well, I'm not trying to be ultra-protective but mostly the the casual surfer who may be sniffing around. I'm aware that on the Internet, you cannot really hide completely.

@pratik I never use G Webmaster tools. Bing neither. I let them do whatever they are going to do.

@bradenslen Haha! True. It's pointless. I went through a SEO phase a decade ago. After that I swung the other way and was focused on wiping off my presence. Now, I just try not to put anything out there that I wouldn't want found.

@prologic True! Secure DNS is not very widespread yet but if you have a trustworthy provider and know how to set it up that can be a handy tool. VPN will cover more bases provided you have a trustworthy option for that.

@pratik 👍 Just checking on the paranoia level since I hadn’t yet seen the “DNS is leaky” argument in the conversation.

@fgtech it ought to be! I run my own local DNS service at home and I use secure forwarders 🤷‍♂️

@prologic Nice! Do you have your configuration documented somewhere? I use VPN for now, as is probably obvious from my comments, but experimenting with secure DNS is on my todo list for next year sometime.

@fgtech I simply run an instance of AdGuardHome

@prologic Nice. I’ll check it out!

@prologic Good to know, thanks.