There is nothing more frustrating than coming across articles that contain YouTube videos that have been removed for some reason. This is extremely frustrating for users and can be easy for website owners to miss (especially for large websites). And from an SEO perspective, fixing unavailable videos can avoid UX barriers and unhappy users. So join me as I search for pages of YouTube videos that never play.
When reporting on major algorithm updates, and especially major core updates, I have repeatedly explained the importance of meeting or exceeding user expectations. I also explained that Google has an uncanny ability to understand when users are dissatisfied, which can lead to the impact of major algorithm updates (e.g. major core updates) over time… And now that we’re talking about Navboost Know and know how Google uses user interaction signals To influence rankings, we understand how Google can determine whether content meets user expectations.
So when I review websites, I look for situations where users may become frustrated, not getting what they need due to queries, etc. For example, there is nothing worse than visiting a site to learn about a topic and attempting to watch a video that supports the content and the video has been removed or marked private. This is a big UX barrier in my opinion and something I’ve often found during audits when helping companies impacted by major algorithm updates (and especially major core updates). If the video was extremely important to the site, how could this content meet or exceed user expectations? And what impression do you leave when there are broken media elements on your pages?
Maybe the site owner or author once had a great video to share that supported the content, but the YouTube video was removed for some reason. For example, the person who posted the video may have violated a YouTube policy or marked it private. We’ve all experienced the dreaded “This video is not available” message when we try to watch an embedded YouTube video, and that’s never a good feeling…
Hunting Broken YouTube Videos:
Okay, we know it’s a big user experience problem when people come across embedded YouTube videos that have been removed for some reason. But how do we find these videos? Well, that’s the problem. In the past, there wasn’t a great way to get these videos out to the public. And this is especially the case with large websites with tens, hundreds of thousands, or even millions of URLs. You really can’t go through URL by URL trying to find these problems…
Again, there has never been a good solution to bring the removed YouTube videos back to the surface. Until now. 🙂 🙂
Send a “frog signal” for help!
As I was reviewing another major website with this problem, I decided to bring in the big guns. I wrote an email to Dan Sharp out of Screaming frog. It was like sending a bat signal, but green and with a frog. 🙂 It didn’t take long for Dan to get back to me and explain that he was looking into the matter. And then, not long after, boom, he sent an email explaining how to handle the situation.
Dan explained that you could crawl a website by adding YouTube as a CDN (so it’s handled internally), use a custom search and look for error messages that appear when a video has been removed, and ensure that the JavaScript playback is enabled and that .js files can be crawled, and then you could uncover the removed videos.
I was excited to try it out…so I fired up Screaming Frog, used the configuration Dan explained, and browsed a small website with removed YouTube videos.
And I saw this:
Great! The video problem plaguing Gotham would soon be solved! Okay, enough with the Batman references.
This was great, but the approach required crawling an entire website. What if I wanted to search through a series of URLs to look for this issue? This would be incredible as you could export top URLs from GSC and then crawl them (because if you read any of my recent posts you know that highly visible AND lower quality is an extremely dangerous SEO combination). Maybe you just want to crawl the most visible URLs that were deleted during a recent algorithm update
I emailed Dan about this and he once again came up with a great solution. The updated configuration requires you to add the URLs you want to crawl to the Include tab, including youtube.com and .js files.
Then you need to put Screaming Frog into list mode and crawl away. After the crawl completes, you can use the Custom Search tab to view the returned URLs that match the criteria. For example, YouTube videos that have either been marked private or are not available.
Let’s look at a quick example of checking a specific set of URLs. Perhaps these were the top URLs that were deleted during a major algorithm update (hint hint).
Step-by-step: How to view YouTube videos that are unavailable or marked as private based on a specific URL list
1. Launch Screaming Frog and switch to list mode.
2. Add YouTube as a CDN.
Click the configuration menu and then click CDNs. Add https://www.youtube.com/embed/ to the list. These URLs are now considered internal URLs when crawling.
3. Make sure JavaScript rendering is enabled.
Click the Configuration menu, Spider, and then Rendering. Use the dropdown menu and select JavaScript.
4. Select Googlebot Smartphone as user agent.
Click the Configuration menu and then click User Agent. Select the Googlebot smartphone as the user agent.
5. Click the Configuration menu and then click Include. We want to add the URLs to be checked. Maybe just enter ten of these URLs for a test. Tip: You can mix in URLs that don’t have video issues to make sure everything works fine.
6. Make sure youtube.com and \.js are in the include list! We want to crawl YouTube.com as if it were a CDN (internal), and then we want to crawl JavaScript files.
7. Add your custom search filters.
Click Configuration, Custom, and then Custom Search. Click the +Add button at the bottom right. Enter a label for the first filter (e.g. “Video not available”). And then “Contains”, “Text”, type “Video not available” and select “Page Text” from the last drop-down list. Complete this for “The video is private” as well. At the end you will have two custom search filters available. Click OK to finish.
8. Upload your URL list.
This is the same list that you added to the Include tab. You not You need to include youtube.com and \.js in this list. It should just be the list of URLs you want to check from the website you are checking. Click Upload in the top menu, then choose how you want to enter the URLs (paste, manually, via file, etc.).
9. Crawl Away!
Click “Start” and run your crawl. Grab your drink of choice while you wait for Screaming Frog to do the heavy lifting.
10. Review the custom search results.
Click the Custom Search tab, then use the drop-down menu to select one of the reasons why the videos didn’t resolve. If you click on any of the YouTube videos in the list and check the Inlinks tab, you can see the URL where it was embedded.
11. Problem exporting to YouTube.
You can also export all displayed URLs by clicking the Bulk Export menu, Custom Search, and then Custom Search Filter.
That’s it! Now you have a solution to uncover the dreaded YouTube embeds that are unavailable or marked as private.
How Google could help: Providing better video indexing reports in GSC.
Some of you may be wondering why GSC doesn’t show the URLs with removed videos. There is a category of errors called “Video not found in host service”. This would be a solid report to uncover all the videos that won’t resolve… but that’s not the case. If Google could simply list all videos that are not available on the source service (even outside of YouTube), that would be incredible. I’ll ping Google about this to see if something like this is possible.
Final tips and recommendations for website owners who want to view YouTube videos that are unavailable or marked as private:
- You can crawl an entire website and search for removed YouTube videos. However, for larger websites, crawling a range of URLs may be a better solution. For example, exporting URLs via the GSC API based on some criteria and then running them through the process I documented in this post.
- Don’t forget to add youtube.com and \.js to the Include tab. This is necessary for this to work properly.
- Make sure JavaScript rendering is enabled and make sure you are using a mobile user agent. I recommend the Googlebot smartphone.
- If you want, you can add more YouTube errors to be detected using Custom Search. If you notice additional messages on your site beyond those listed in this post, simply add another custom search filter.
- Watch the number of URLs you crawl per second. I would start lower and watch for any problems. Then adjust as needed.
- Remember that you can export the results Custom search filterr, which can be extremely convenient for sending to customers or other members of your team.
Improve user experience by finding YouTube videos that never play:
I hope you can use this solution to help your business or customers view YouTube videos that are unavailable or marked as private. Again, there’s nothing more annoying or frustrating than coming across articles with embedded YouTube videos that won’t play. And since we know that Google uses user engagement signals for rankings, I would always make sure you can meet or exceed user expectations based on the search query. Finding these videos is definitely a step in the right direction.
Now go search for unavailable YouTube videos. 🙂 🙂
GG