Posts

Are My Relevant Web Pages Being Crawled and Indexed?

Do you want more traffic to your site?

Of course you do, and it starts with understanding two key actions taken by Google (and other search engines but we’ll focus on Google): crawling and indexing.

Your pages are being crawled, and if you want them to be, they’re also being indexed. How frequently they’re being crawled and indexed is the major question.

Let’s start by defining these two key terms.

What is crawling?

Google’s Spider crawler constantly searches for new web pages and finds them because of links to a page. Once on a particular page, Google will follow the path your page leads them down through your links to other pages on your website. When the crawl is done, they’re indexed.

It’s that simple… almost. We’ll get to what can influence the frequency of crawling.

But first,

What is indexing?

Simply put, indexing is gathering and processing the information acquired during the crawl and adding web pages into Google’s search engine.

Being indexed is not the same thing as being listed in Google’s SERPs. When a page has been indexed it has been downloaded to the server of the search engine; a site doesn’t have to be indexed to be listed.

And in some cases, you may not want your page to be indexed, such as when it has thin or no-value content, for example, thank you pages, login pages, and internal search results pages. In that case, you want to insert a no-index tag, which means that although your page is still being crawled, it’s not added to Google’s search index.

The take-away: Index only the most important parts of your website.

Note: Every WordPress post and page is indexed automatically but to improve your search results, you need Google to re-index your site frequently (use Google Search Console to find out how often your site is being crawled).

Your site is probably already being indexed but if it’s new, there’s any easy way to check:

Enter site:yourdomain.com into Google’s search bar. If you’re indexed, the SERP should look like this:

If not, it will return a message that your search did not match any documents.

If that’s the case, it doesn’t mean you have to sit back and wait for Google to notice you. It’s kind of like seeing that perfect potential mate from across the room. If you sit there and wait for him or her to approach, you may be waiting a very long time. Alternatively, if you take action and make it happen, chances are you’ll have a date. Google is a little less risky though, they won’t reject you if you follow the right steps.

But, like dating, indexing is not a one-time deal that gets you where you want to be. You need Google to keep re-indexing your site because they don’t update automatically.

How can you get your site crawled and indexed more frequently?

Google offers us the following tips to improve our page indexing:

  • Create short, meaningful page titles.
  • Use page headings that convey the subject of the page.
  • Use text rather than images to convey content. (Google can understand some image and video, but not as well as it can understand text. At minimum, annotate your video and images with alt text and other attributes as appropriate.)

(Source).

What other major factors affect indexing?

Regular new quality content that is relevant and doesn’t break any rules (see below) is one of the primary ways. Add new, relevant, and user friendly, high-quality content regularly and update existing content by creating a content marketing strategy. Have a look here at how that’s different from a simple content strategy:

Include the primary keyword in your domain name.

Use high quality backlinks that are reputable and trustworthy.

Internal links are a great practice for SEO and for keeping people engaged on your websites and answering their questions the second they come up. When linking to a section in the same page, use the same anchor text to encourage deep crawling.

An XML sitemap is a roadmap to help Google look deeply into your website because it lists every URL. A variety of plug-ins are available that you can use to generate a sitemap. For WordPress, use Yoast SEO.

WordPress Ping Services automatically notifies search engines that your site has been updated with new content, which keeps those bots crawling and (hopefully) indexing.

What prevents Google from indexing a website?

Playing by Google’s rules will keep you in their good books––their index, so avoid the following:

  • Duplicate content
  • Keyword and meta-tag stuffing

In some cases, you may want to prevent Google from indexing your site. See their support guide for some simple guidelines.

Have I implemented Google Analytics and Google Search Console?

Have you seen the movie Tag? It’s about five friends who’ve played a 23-year-long game of Tag.

Search Engine Optimization (SEO) is kind of like that: a continuous game of Tag–but inverted. You want to be IT.  

In the game of Tag, being IT means you’ve let your guard down and left yourself wide open to the competition.

In SEO, being IT means you’ve done the opposite and you’re ranking #1 because you’ve kicked some butt.

But in SEO, maintaining IT status requires continuous effort.

Luckily, there are tools that can help you, so you don’t have to run all over the place trying to stay ahead.

Google Analytics and Google Search Console are two of these tools, and they’re some of the best tools for SEO.

Why?

They give you detailed metrics about your site, which you can use to get more traffic.

More traffic = more conversions.

Sounds simple, right? It is.

And the more we know, the better equipped we are to solve the issues that are preventing us from ranking in the top spot of Google’s search results and converting users to buyers.

Contrary to popular belief (not knowledge), ignorance is not bliss–especially in the world of digital marketing!

What’s one of the most loved things about these tools?

They’re free! And there is no catch. Simply embed the proper code on your website.

Granted, not all free stuff is good. I don’t like free advice I haven’t asked for and I don’t like free hotdogs.

In fact, there is a lot of research out there that supports the notion that the more you pay for something, the more you value it.

But tools like Google Search Console and Google Analytics are the exception.

The fact that they’re free is not the best thing about either tool though.

Keep reading to find out why they’re the scaffolding of a successful SEO strategy and how they can help you discover what’s working on your site and what’s not (you might be surprised).

Before I get into what each one is, I feel compelled first to point out that both tools are extremely useful for both beginners and experts.

Also, installation and setup for both tools are fairly intuitive, so I won’t offer a step-by-step guide here. Instead, I’ll get to the meat of how these tools are a necessary part of any SEO strategy.

So, what is Google Search Console (GSC) and Google Analytics (GA)? Are they the same thing?

They appear similar, but the information they provide is different.

Google Analytics tells you about your visitors: who they are, how they’re getting to your site, where they are geographically, and how much time they’re spending on your site.

Google Search Console, on the other hand, is a suite of tools that provide information about external links and keywords.

But read it from the makers themselves:

“Google Search Console is a free service offered by Google that helps you monitor and maintain your site’s presence in Google Search results. You don’t have to sign up for Search Console for your site to be included in Google’s search results, but doing so can help you understand how Google views your site and optimize its performance in search results.”

With GSC, you can understand how you’re ranking and why, so you can focus your efforts and continue to improve. You can’t be successful in internet marketing if you’re a passive learner. Well, maybe you can be by sheer luck, but the odds are against you.

But that doesn’t mean it takes a lot of hard work either. GSC and GA are some of the best tools available because they’re easy to use, they’re comprehensive, and they’re free.

Let’s start with GSC.

How can Google Search Console help your SEO efforts?

Here’s a quick look at what you can do with this suite of tools:

  • Quickly find the overall performance of your website.
  • Manage what content gets crawled and what you don’t want appearing in search results.
  • Maintain your site with minimal disruption on the front end.
  • Resolve spam issues and malware.
  • Discover which keyword queries made you appear in search results.
  • Determine what sites are linking to your pages.
  • Manage mobile site performance.

Google Analytics provides filters to help you access valuable data about your site. The options are Clicks, Impressions, CTR (click-through rate), and Position.

Clicks counts the number of clicks that brought users to your site from the SERP.

Impressions displays how many links to your site appeared in the SERP, even if they don’t display in their field of vision (if the user didn’t scroll down to the bottom of the page). However, this only applies to the first page, so if your link appeared on the second page, it doesn’t count.

CTR is the sum of the clicks divided by the impression count (because an impression is required for a click).

Position measures the average position of the topmost result from your site.

Pretty cool, right?

Keep in mind that these metrics will change depending on whether you aggregate results by site or page.

These metrics are a good starting point from which to refine your results further. From there you can look at several more options.

Queries – lists all the keywords that brought users to your site.

Page – shows you which of your site’s pages appeared in the results.

Country – displays where the searches originated.

Device – calculates which devices and how many of them were used to search.

Search – indicates the type of search used: video, images, web, etc.

Date – allows you to choose the timeframe.

What can you do with these data?

Start with impressions. If you’re showing up in search results but not getting clicks, then you need to optimize for those specific keywords. You can do this by selecting those first four filters (clicks, impressions, CTR, and position), as well as queries. This will return a list of keywords. Go through each one, applying those first four filters again, to see how each keyword is performing in better detail.

Then grab your finest SEO tool–your brain–and go into manual mode. Search each keyword in Google to see the SERP. If you’re on page, you need to look very closely and objectively at why you’re not getting a lot of clicks.

What’s the one thing that draws you into an article, a book, or a film more than anything else?

You guessed it–the title!

The title tag and meta description are the most significant but often overlooked parts of SEO. Even little tweaks, such as changing one word, can boost your CTR. We won’t get into the nitty-gritty of them because that could (will) be an entirely separate blog. For now, ensure your title and meta tags are optimized, compelling, and specific. Look at what is also showing in the results page. How does yours compare?

Mobile keyword research is an essential part of serious SEO, and you might be surprised to learn that mobile keywords are quite different from desktop keywords. How much do you spend on your phone? For some, it’s the only device they use.

GSC can return results on which keywords are being used specifically in mobile searches, which you can then use to enhance your mobile SEO strategy.

Another impressive feature of GSC is that it gives you the ability to compare dates. A report generated by GSC only covers the previous four weeks so you can look at older dates (up to 90 days back) as well as compare dates within a specified range. This feature will assist you in examining and analyzing any changes in your position in the SERP.

How can Google Analytics increase your conversion rate?

This tool gives hundreds of different metrics, but there are a few on which you want to focus.

Discover what people are looking for in your site by finding out what they’re typing into the search field. When you know exactly what people are searching for you can provide targeted information, right? Right!

Check on-site behaviour with the behaviour flow report and see how people are navigating around your site. This gives insight into where your users are dropping off so that you can create more internal links or CTAs, and improve those pages with better content.

Using the goals function, track your conversions according to each of your marketing activities, so you know which ones are returning the best ROI.

Analyze your social media posts to determine which channels are generating the most traffic.

Determine what devices are generating more conversions. If your mobile conversation rate is low, this is an opportunity to optimize your pages for mobile.

Discover what pages are generating the most conversions so you can guide traffic to other areas of your site.

Wrapping up…

This blog is just a sample of what GA and GSC can do.

Remember, these tools are made for beginners and more experienced SEO geeks.

After an easy installation, play around with each tool and learn how to take advantage of their features to improve your user’s experience and ultimately, increase your conversions.

A final tip – like any of your SEO strategies, GSC and GA are not one-time deals. If you want to drive thousands of people to your site, you have to use these tools consistently to stay on top of your success. Like Tag, it’s a never-ending game. (We didn’t get to mention Tag Manager, even with all the puns at our fingertips)

Oh, and we love to help so,

Logging out,

Logical Mix