What stops a page from being indexed
| Signal | Where it lives | Effect |
|---|---|---|
| HTTP status | Server response | Only 200 pages are indexed. 404/410 drop out; 5xx errors, if they persist, do too. |
| robots.txt Disallow | /robots.txt | Blocks crawling. Google can't read the page (or its noindex tag). |
| Meta robots noindex | <meta name="robots"> | Removes the page from the index once crawled. |
| X-Robots-Tag noindex | HTTP header | Same as meta noindex — common on PDFs and staging sites. |
| Canonical elsewhere | <link rel="canonical"> | A strong hint to index a different URL instead. |
| Redirect | 3xx response | The target URL is indexed, not this one. |
robots.txt is not a noindex
Blocking a URL in robots.txt stops Google crawling it, but the URL can still be indexed (without a description) if other sites link to it. To keep a page out of results, allow crawling and add
noindex.Why link builders should check indexability
A link only passes value if search engines can crawl and index the page it sits on. Before you pitch a guest post, accept a placement or pay for a link, run the target page through this checker. A noindexed, robots-blocked or canonicalized page is a link that probably won't count — read more in our guide to getting backlinks indexed.
Indexable but still not indexed?
If every check passes and the page still isn't in Google, the cause is usually quality or discovery, not a technical block: thin or duplicate content, few internal links pointing to it, or a site Google doesn't crawl often. Strengthen internal links (see internal linking), make the page genuinely useful, and request indexing in Search Console.