.Google.com's John Mueller responded to an inquiry concerning why Google indexes webpages that are actually refused coming from creeping through robots.txt as well as why the it is actually risk-free to neglect the related Explore Console reports regarding those crawls.Robot Visitor Traffic To Query Criterion URLs.The person talking to the question documented that bots were producing hyperlinks to non-existent concern criterion URLs (? q= xyz) to pages along with noindex meta tags that are likewise shut out in robots.txt. What motivated the inquiry is actually that Google.com is actually crawling the web links to those web pages, obtaining blocked through robots.txt (without envisioning a noindex robots meta tag) at that point acquiring reported in Google Explore Console as "Indexed, though obstructed by robots.txt.".The individual inquired the following concern:." But listed here is actually the major inquiry: why will Google index web pages when they can not even observe the information? What's the conveniences because?".Google's John Mueller verified that if they can not creep the page they can't see the noindex meta tag. He likewise produces an appealing reference of the website: hunt driver, suggesting to overlook the outcomes given that the "ordinary" customers will not observe those results.He wrote:." Yes, you are actually correct: if our experts can not crawl the webpage, our company can't find the noindex. That claimed, if our experts can't creep the pages, then there's not a whole lot for our company to mark. Thus while you may view a few of those web pages with a targeted web site:- concern, the typical consumer will not see all of them, so I definitely would not bother it. Noindex is actually likewise fine (without robots.txt disallow), it just implies the Links will end up being actually crept (as well as find yourself in the Search Console record for crawled/not listed-- neither of these statuses induce problems to the rest of the site). The integral part is that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's solution validates the constraints in operation the Website: search advanced search driver for diagnostic main reasons. Some of those factors is given that it's certainly not linked to the regular hunt index, it is actually a separate thing entirely.Google.com's John Mueller discussed the website hunt driver in 2021:." The short answer is that a website: inquiry is actually not implied to be total, nor used for diagnostics reasons.A web site concern is actually a details sort of hunt that restricts the outcomes to a particular web site. It is actually primarily only the word site, a bowel, and then the site's domain.This question confines the end results to a specific web site. It is actually not meant to become a detailed selection of all the pages from that website.".2. Noindex tag without using a robots.txt is fine for these kinds of scenarios where a bot is connecting to non-existent pages that are actually getting found out by Googlebot.3. URLs with the noindex tag will generate a "crawled/not catalogued" item in Browse Console and that those will not possess an adverse effect on the remainder of the website.Review the concern and also address on LinkedIn:.Why would Google mark web pages when they can not also view the material?Included Image through Shutterstock/Krakenimages. com.