Are My Relevant Web Pages Being Crawled and Indexed?
Do you want more traffic to your site?
Of course you do, and it starts with understanding two key actions taken by Google (and other search engines but we’ll focus on Google): crawling and indexing.
Your pages are being crawled, and if you want them to be, they’re also being indexed. How frequently they’re being crawled and indexed is the major question.
Let’s start by defining these two key terms.
What is crawling?
Google’s Spider crawler constantly searches for new web pages and finds them because of links to a page. Once on a particular page, Google will follow the path your page leads them down through your links to other pages on your website. When the crawl is done, they’re indexed.
It’s that simple… almost. We’ll get to what can influence the frequency of crawling.
But first,
What is indexing?
Simply put, indexing is gathering and processing the information acquired during the crawl and adding web pages into Google’s search engine.
Being indexed is not the same thing as being listed in Google’s SERPs. When a page has been indexed it has been downloaded to the server of the search engine; a site doesn’t have to be indexed to be listed.
And in some cases, you may not want your page to be indexed, such as when it has thin or no-value content, for example, thank you pages, login pages, and internal search results pages. In that case, you want to insert a no-index tag, which means that although your page is still being crawled, it’s not added to Google’s search index.
The take-away: Index only the most important parts of your website.
Note: Every WordPress post and page is indexed automatically but to improve your search results, you need Google to re-index your site frequently (use Google Search Console to find out how often your site is being crawled).
Your site is probably already being indexed but if it’s new, there’s any easy way to check:
Enter site:yourdomain.com into Google’s search bar. If you’re indexed, the SERP should look like this:
If not, it will return a message that your search did not match any documents.
If that’s the case, it doesn’t mean you have to sit back and wait for Google to notice you. It’s kind of like seeing that perfect potential mate from across the room. If you sit there and wait for him or her to approach, you may be waiting a very long time. Alternatively, if you take action and make it happen, chances are you’ll have a date. Google is a little less risky though, they won’t reject you if you follow the right steps.
But, like dating, indexing is not a one-time deal that gets you where you want to be. You need Google to keep re-indexing your site because they don’t update automatically.
How can you get your site crawled and indexed more frequently?
Google offers us the following tips to improve our page indexing:
- Create short, meaningful page titles.
- Use page headings that convey the subject of the page.
- Use text rather than images to convey content. (Google can understand some image and video, but not as well as it can understand text. At minimum, annotate your video and images with alt text and other attributes as appropriate.)
(Source).
What other major factors affect indexing?
Regular new quality content that is relevant and doesn’t break any rules (see below) is one of the primary ways. Add new, relevant, and user friendly, high-quality content regularly and update existing content by creating a content marketing strategy. Have a look here at how that’s different from a simple content strategy:
Include the primary keyword in your domain name.
Use high quality backlinks that are reputable and trustworthy.
Internal links are a great practice for SEO and for keeping people engaged on your websites and answering their questions the second they come up. When linking to a section in the same page, use the same anchor text to encourage deep crawling.
An XML sitemap is a roadmap to help Google look deeply into your website because it lists every URL. A variety of plug-ins are available that you can use to generate a sitemap. For WordPress, use Yoast SEO.
WordPress Ping Services automatically notifies search engines that your site has been updated with new content, which keeps those bots crawling and (hopefully) indexing.
What prevents Google from indexing a website?
Playing by Google’s rules will keep you in their good books––their index, so avoid the following:
- Duplicate content
- Keyword and meta-tag stuffing
In some cases, you may want to prevent Google from indexing your site. See their support guide for some simple guidelines.
