Recently, several website owners have reached out to me asking for help with indexing issues. In most cases, I find that Google has classified its pages as “crawled – not currently indexed” in the page indexing report in Google Search Console.
In almost In every case I’ve examined, these sites have quality issues. This is typically “commodity content” – essentially repeating what many others have already written on a topic without offering anything new or more helpful than what is currently available online.
In this article I will tell how I look at the crawled, currently unindexed report in GSC. I provide you with a tool to help you find the pages in this report that you should analyze further. And I’ll give you some improvement tips to help you possibly recover. However, I must give a fair warning. For most websites, recovery is difficult if you have a large number of pages that you want to index but they are stuck in the crawled state and are not currently indexed.
What Google said about “Crawled-currently not indexed” at the Google Search Central event in Toronto.
I attended the Google Search Central event in April 2026. Organizers asked us not to directly attribute quotes to any Google employee, but they gave us permission to share what was said.
A moderator reported on how the search works. He said when Google crawls a page, it essentially means they are downloading it. Then, “If we think it is useful we could put it in a database,” or in other words, in the Google index.
He then talked about what things Google wants to include in the index. He said that AI has lowered the inhibition threshold for creating things. If someone can create content about anything, then the type of content Google wants to add to its index is content that offers two things: personal experience, And Knowledge that no one else has.
He said if Google crawled your page and decided not to index it, it could be for two reasons:
1. There could be a technical problem.
I have found this to be rare. However, just last week I reviewed a website that was undergoing a migration where all pages were stuck in crawl mode and are not currently indexed. Note: This is not the same as Discovered – not currently indexed, which means Google knows about the pages but hasn’t crawled them yet.
My first step was to analyze whether Google could see the content on pages. I used the Page Inspection tool in GSC by clicking on the magnifying glass next to the URL in the crawled, currently unindexed list and clicking “Test Live URL”. Surprisingly, when I looked at the live tested page, there was only a headline, a few standard words and no content at all.
In this case, the website owner actually had a technical problem. Your robots.txt contained this line: Disallow: /*?*. The idea was to block URL crawling with parameters like ?replytocom or ?utm_source. However, their new theme relied on these parameters for their CSS files and Javascript, so they effectively blocked Google and all other search engines from seeing the majority of their content.
We have now removed this lock and pages are slowly being displayed in the index again.
If your live test shows that Google can actually see the content of your pages, it is very unlikely that it is a technical issue causing problems with crawling the currently unindexed page.
I should also mention that some sites should be included in your crawled, currently unindexed list if it is not the canonical version. If you see /feed/ pages or pagination or pages with URL parameters, that’s normal.
2. Quality
The Google employee in Toronto went on to explain another reason why Google crawls a page and doesn’t index it. He said it could be because: “We looked at it and decided it wasn’t good.“That’s what he said If thousands have covered the exact same topic, they might conclude that your site is probably not useful for searches. There may be other options that are more popular or of better quality.
He also said that Google sometimes experiments by allowing your page to be indexed for a while to see if users like it: “We’re experimenting to see which one produces happier users.“That’s a pretty wild statement!”
I bet that if you have pages that you want to index but Google has them in the crawled and not currently indexed section, then your main problem is related to the default content.
Raw material content is the most likely cause
Google talked a lot about default content at this event.
Commodity content is content that almost anyone could write on a topic. This is generally a repetition of what already exists on other sites online. Non-commercial content provides a unique perspective or contains content that others lack or cannot be easily reproduced. It typically demonstrates first-hand knowledge or experience.
Take this article you are currently reading. Anyone could use AI to write a helpful article that defines crawled but currently unindexed pages. However, my article is about my experience as a professional paid to give my opinion on this topic. I’ve shared the real-world technical example above, I’ve shared first-hand information I learned from attending a Google event, and I’m about to share my observations on pages that were deemed not worthy of indexing.
My observations of pages stuck in crawling that are not currently indexed
These sites are usually not garbage. They’re good, decent articles – just as good as the sites Google ranks. And that’s exactly the point. The sites are nothing special or more valuable than what currently exists.
Here is the process I use to analyze these pages.
To find the list, click “Pages” under “Indexing” in GSC. Then click Crawled – not currently indexed:
Below you will see a list of URLs to examine. (Below I’ll tell you more about a tool I created to help you Filter this list to see the URLs that really matter.)
I will find a URL in this list that I really want to index.
First, I look for some search queries that you think the site would rank for. There is usually an AI answer on the SERP, which is very helpful. A user often finds the answer to his question there. If this is the case, then why would they click on your website to read the exact same thing?
I expand the AI overview and then open Gemini in the Chrome sidebar. Then I hold down CTRL/Command and click on the top websites linked to in the AIO. If you do this while Gemini is open in the Chrome sidebar, these tabs will be added to your Gemini conversation.
Then I type “/” which opens the skills I saved to chrome://skills/ and select my non-commodity check. (If you are a Member of my paid communityYou can find this complete ability Here.)
This skill is a very long prompt that goes into some of the things Google tells us in the documentation that its algorithms are designed to reward create helpful contentincluding but not limited to:
- Offers the content Original information, reporting, research or analysis?
- Offers the content insightful analysis or interesting information This is beyond the obvious?
- If the content draws on other sources, is simply copying or rewriting those sources avoided and provided instead? significant added value and originality?
- Offers the content significant value compared to other pages in search results?
And Gemini gives me a few reasons why the linked pages offer the reader added value. Note: Sometimes sites are ranked not because of their non-merchandise value, but because they are an authoritative source. If you’re a known authority, you can get away with a little more “commodity.”
Now we must acknowledge that Gemini does not have insider insight into Google’s ranking systems. That is not the case knowledge why certain pages rank. What we’re trying to figure out here is what things might make a page worth showing to searchers.
Then I open my client’s page and say: “Now analyze this page using the same criteria. This site does not rank well. It is our customer. Please let us know where you think there is a defect. There is no need to suggest improvements at this time.”
Here is the result for a crawled, currently unindexed page where I used this prompt.
John Mueller and Martin Splitt recently discussed “crawled – not currently indexed” on a podcast.
Just as I was about to publish this, Google released a “Search Off the Record” podcast on “How to read the indexing report“There’s a lot in here, so I bolded the parts I thought were important.
This discussion begins at 8:32 p.m. in the video
Chapter 9: Discovered vs. Crawled, Not Indexed: Is it a technical issue or a site quality issue?
“And also as you add or change your site, or if your site is very new, you can actually also use this report to see a little bit of how your site goes through the different stages, because at some point you’re going to see pages in Discovered that aren’t currently indexed. That tells you that we know they exist, but we haven’t actually visited them. And if we haven’t visited them, we can’t put them in the index.” Crawled – not currently indexed, meaning we have visited and not indexed. And that can have very different reasons. Would you say this is often or only sometimes a sign of a quality problem?
So In any case, if our systems have serious concerns about the quality of a website, they will reduce the number of pages they index. Because if we are very concerned about overall quality, it makes little sense for our systems to spend a lot of time on the site.
So we’ll probably crawl a lot less, we’ll index a lot less and then You’ll see things like “crawled,” “not indexed,” or “discovered” and “not indexed,” which from our perspective is essentially our system saying, “We know about it, we’ve looked at it, and if we’re happy, we’ll take another look to see if we can index it.” It’s not so much that I would say that you should take these situations and try to fix them. From a technical perspective, you don’t need to fix this technical issue as Google is currently not indexing this page, but rather You almost have to if you see a larger pattern like this, that Google isn’t indexing many of your pages and there’s no technical reason for it. You almost have to take a step back and think about the overall quality.
And thinking about quality is really challenging because often it’s about your website and your baby. And of course it’s the best baby ever. But take a step back and try to look at it through the eyes of someone who isn’t directly involved with your website. Sometimes this opens up some ideas for areas where you can improve. If the majority of your website is AI-generated and it’s been working for a while, people may look at that AI-generated website and think, “Well, I can tell this is AI-generated.” For me there is nothing unique or valuable here. That’s not to say that all AI-generated content is bad, but sometimes you just come across websites that make you think: Anyone could have written that. That doesn’t mean anything to me. Yes, that’s right. And I think what makes that difficult is not just the fact that the way you wrote it is obviously what you thought was best, and so of course you think it’s high quality. It’s really, really hard to break out of your own perspective. But sometimes there are so many other things that are just as good. So why should we add it to the index?
And then that can tell you that maybe this content isn’t as valuable as I thought because other people report the same thing. And then what value does it have if this version of it is included in the index? Yes, that’s right. I think we could do a whole podcast about quality. I think maybe As far as quality is concerned, it should also be mentioned that it’s not just the text that matters. So often People will say, “Well, my writing is unique or my articles are good and they’re packed on a page that’s terrible to access. When everyone tries to load it, their computer’s fan spins up and they think, “Oh my God, I have to run away to make sure my computer doesn’t explode.” Maybe this is an extreme case, but You’ve all seen those pages where basically the text is there, but it’s almost hidden, hidden behind ads, hidden behind interstitials, hidden behind other things that move and come and go, maybe hidden under a lot of filler content that we sometimes see, for example, with recipes that have this really long story on top that maybe most people don’t really care about. And then comes the recipe. These are all things where the overall quality is much more than just the text you say, that’s my main content. This is what Google should count for my website. And from our perspective, we need to consider almost the entire experience on a page because that’s what users see. It’s not about users going to a webpage and activating a magic mode that just pulls out the text, but about them enjoying the full experience of that website with all the 3D and 4D animations and everything. I very much agree. Agree, oh my god.”
How can you fix this problem?
Oh man, this is the hard part of this article because in many cases I feel like it is extremely difficult to get pages out of crawling that are not currently indexed. I mean, if excessive advertising and filler is to blame, there are obviously things there that need to be improved. If there’s a technical issue, fix it and request a re-indexing through GSC – or just be patient and wait for Google to try crawling your pages again.
However, if it’s a quality issue, you’ll probably need to put significant effort into improving these pages.
For many sites I analyze, their superpower in the past has been the ability to cover a topic thoroughly. Lately there is a trend to not just cover one topic, but to anticipate all requests and deal with them too. This was mentioned a few times in the Google Search Central event. If you create large amounts of content using this method, you risk being subject to a tiered content penalty. I can’t prove this yet, but I suspect that the June 2026 spam update impacted a number of sites that were creating mass content at scale. If this is true, no manual action will appear in GSC. All you will see is a drop in organic traffic with no explanation.
I fear for many SEO agencies because for many, content creation is the most important tool in their toolbox. AI has made it much easier to cover content on any topic. I’m not against using AI to help with content creation. But, If your SEO company can use AI to create content about your topics, then it is unlikely to be original, insightful, and significantly more helpful than what currently exists. There are exceptions. I know a few agencies that use clever AI pipelines to survey a company, extract their relevant experiences and turn them into good, original content.
While I don’t recommend using AI to write your content for you without human input, I think you can brainstorm with AI to help improve it. The problem, however, is that the solutions require effort. The word “effort” is used 120 times Google Quality Rating Guidelines. You want to find ways to use your experiences to create content that expands the body of knowledge that currently exists on your topics.
Try this simple prompt. Share your content with an LLM or open Gemini in the sidebar and ask, “Is this content likely to be considered standard content?”
I just opened Gemini in Google Docs and asked for this exact article you are reading:
Try this next for some ideas.
“Give me 20 ideas that will help me draw from my first-hand experiences to make this article even more helpful and much better than anything else on the subject on the Internet.”
Damn, there are some good ideas in here.
Some tools to help you evaluate your crawled but not currently indexed pages
I created a few tools using Google’s Antigravity. You can find them at tools.mariehaynes.com.
There are two new tools:
1. Filter your crawled URLs that are not currently indexed. Export your crawled, currently unindexed URLs from GSC. If you are exporting as CSV, open the ZIP file and locate the table.csv file. You can upload it to this tool and it will remove /feed/ pages and others so you can see and click on the URLs you want to examine.
2. GSC index checker. You must log in to your Google account to use this tool. However, please note that I don’t see any of your data. A list of URLs is checked to determine their indexing status. You can choose from the most recent pages in your sitemap, manually insert a list of URLs, or have the tool retrieve your most visited pages from GSC.
What you’re looking for here is whether these pages that are important to you are actually indexed, or whether they are stuck in the crawl space and are not currently indexed.
I hope this article helps! Google seems to be stricter about indexing these days.
Mary
Or come to us Search bar for the latest news on search and AI.