In the last few years, WordPress SEO has evolved into a small little industry of its own with an army of consultants, tools, and techniques.
The list of websites using WordPress keeps growing and with it does the demand for optimizing the sites for search engines.
Ranking your site in search results is a three-step process:
1. Get your pages and posts read (crawled) by search engines
2. Get your pages and posts understood (indexed) by search engines
3. Help the search engines understand the quality of your site and pages through backlinks
In this post, I will show you a step-by-step process to make sure search engines are able to find and understand the pages on your WordPress site.
But before I jump into explaining the nuts and bolts of optimizing a WordPress site, let’s peek inside the WordPress platform.
WordPress started as a blogging tool in 2003. Since then, it has evolved into the most popular Content Management System in the world.
Did you know that 26% of websites on the net run on WordPress?
Several popular websites like Mashable and TechCrunch use WordPress. Since WordPress is open source, it is free to use, adding to its appeal.
The Anatomy of A WordPress Site:
WordPress has three main elements:
1. The core WordPress platform – this is constantly updated by Automattic (company that built WordPress) and individuals who contribute to the code base.
2. Plugins – These add-on modules you can install and manage from inside the WordPress admin panel. Plugins are written by individuals and businesses who want to add new features to WordPress. This is one of the key reasons for WordPress’ success as you can add pretty much any feature through a plugin. A majority of plugins is free but there are many paid or premium plugins also. If you can’t find a plugin for a specific need, you can always write it yourself or hire a programmer to develop the plugin.
3. Themes – they control the look and feel of a WordPress site. A theme includes layout, styles and elements for home page and other commonly used pages like ‘About Us’ and ‘Pricing’ page. There are thousands of free and paid themes available for any specific need. By using an appropriate theme, you can give your site a unique look.
Themes and Plugins together make WordPress highly customizable, a big reason for its popularity.
The content on a WordPress site is organized into following sections or folders
– WP-Admin – stores the core WordPress files. You rarely access these files unless you need to customize something. You should not worry about optimizing any of these files for search. Ideally, you don’t Google to list them in search results. You should block access to some of the files with sensitive information, as we will see later.
– WP-Content – stores files that came with Plugins and Themes. All files uploaded by users such as images that you add to your pages are also stored here. Search engines may need these files to render pages on the site.
– WP-Includes – Certificates, Fonts etc. needed to manage the site and render pages effectively.
While code, fonts and uploaded files are stored in directories, the content created by users such as pages and posts are stored in a MySQL database.
When a visitor to your site tries to access a page, the request is sent to the site. The WordPress system gets the page or post information from the database and returns it to the user’s browser along with files (code) needed to display the page properly.
The same process is followed when Google or Bing search bots visit your site. The bot needs all supporting files to properly format and read index its content.
If you have permission, you can access the code and uploaded files like images or pdf files directly by going to the appropriate directory on the site. When a search bot comes to your site, it is going to access all files in these directories and index them, unless you specifically ask it not to do so.
Challenges in Getting WordPress Pages Indexed Correctly
While WordPress is extremely user and SEO friendly, it does pose some challenges.
The search bots (and users) can access a WordPress post via multiple paths. The most obvious one is using the URL of the page or post. But the same page can also be accessed through alternate paths that include post category, author, tags etc. Each path has its own URL. For example, a post in product category can be accessed using the following paths:
When a search bot visits the site and access the same page using different URLs like above, it sees two pages (one for each URL) but with the same content. This can trigger duplicate content penalty by Google.
Another issue, easily addressed through the settings, is the URL structure of pages and posts, called Permalinks in WordPress. A Permalink is the URL of the page or post.
WordPress gives you several alternative for creating the URLs through the Permalinks. The default is page ID that is not SEO friendly.
The third challenge is deciding which files and folders should be accessible to Bots and which ones should not be. This depends on the information in the file and on how users & search engines will use it.
Creating SEO friendly URLs for Pages and Posts
WordPress gives you complete control over the URL of your pages and posts. You can define a site wide format for URLs from the dashboard.
Select Settings > Permalinks from the admin dashboard.
On the Permalinks settings page, you can select the type of URL structure you want for your site. The best option is to select the Post name structure. When you use this option, WordPress will create URL for the page that matches the page or post name. If you use keywords in the post or page name, the URL will automatically include it and become keyword optimized.
No matter which Permalinks setting you select, always edit the URL while creating the page to make sure it starts with the keyword and does not include any stop words such as and, in, to etc.
SEO friendly (keyword optimized) URLs are a major relevance signal to search engines and help them index your pages for the right keyword.
How to Get Your Site Crawled by Search Engines
When you set up a new site, nobody knows about it, not even Google. If Google does not know about the page, it cannot include the page in search results. So the first step in ranking your pages is to get them into Google’s database.
Google can find out about your site and pages two ways:
1. If you tell it about your site
2. If another site links to your site Google search bots find this link
If your site is new, it likely has no backlinks from other sites. So the only option for you is to tell Google about your site. But even for an existing site, whenever you add a new page, it is a good idea to tell Google about the new page.
Once Google knows about your page, the search bot will come knocking to find out more about your site, a process called crawling. The search bot will go through the pages and dump information about them in Google’s database:
· Page Title
· Page URL
· Page Content, including meta tags for media files (images, videos and audio)
Once the page’s content is in its system, Google will analyse the content of the page and Meta tags to understand content on the page. Finally, Google will use the index information along with the quality / authority signals from backlinks to decide search ranking for the page.
Let’s see how you can get your pages crawled by the search bots so they get into Google’s system.
The first step is to allow the search engines to index the site. You ca control this from the Settings > Reading in the dashboard:
When the site is under development, you should select the checkbox Discourage search engines from indexing this site. When the box is selected, it prevents search engine bots from crawling the site.
Once the site is live, uncheck this box to allow the bots to crawl the site.
But we still need to find a way to tell the search engines about the cool pages and posts on our site. We use sitemaps to do that.
Creating A Sitemap
A sitemap is a directory of pages on your website. It is similar to store directory you see in a mall that helps you find the shop you are looking for.
A sitemap is an XML file. It includes page information such as URL, last update date for the page, change frequency and relative priority compared to other pages on the site.
Search bots use sitemap as a guide for crawling the site. All pages included in the sitemap would normally be crawled by the search bot. The <lastmod> tag tells search bot whether the page has changes since the last time it was crawled. If the search bot sees an update date after the last crawl date, it will crawl the page again to capture the new content.
The best way to create a sitemap for WordPress web site is to use a plugin. Several plugins are available for generating sitemap for WordPress. The most popular ones are Yoast SEO, All-in-One SEO pack and WP-Sitemap.
While any of these will do the job, my personal favorite is Yoast SEO because it can do a lot more than generate sitemap.
Follow these steps to install Yoast SEO:
- - Select Plugins > Add New from your WordPress dashboard
- On the Add Plugins page, search for Yoast SEO
- Click Install Now button to install
- Once the plugin is installed, click Activate button to activate the plugin
Yoast SEO can automatically generate sitemap and update it as you add new pages or remove them from the site. You will need to configure it correctly to generate a sitemap that helps avoid duplicate content issue.
Remember, if you include a page in sitemap, it will be crawled and indexed by bots. While you can control indexing using page level settings and Robots.txt file (explained later in this post), it is better to avoid including pages in sitemap that you don’t want indexed.
Yoast SEO is easy to configure but if you need help, read this post on WPbeginner for detailed instructions:
To enable sitemap generation, select SEO > XML Sitemap from the admin dashboard:
On the User sitemap tab, disable Author / user sitemap to prevent the plugin from including author / user archive pages in the sitemap.
On the Post Types tab, include Pages and Posts in Sitemap but exclude images unless you really want them indexed. If your theme uses custom post types, they will show up here. Include them if your business demands it. For example, if you are selling course online and your theme uses custom post type Course, you should include it in the sitemap.
On the taxonomy tab, exclude Categories and Tags from sitemap unless there is a reason for you to have it in the sitemap. Categories can both help in SEO and be a problem. Categories and tags may be used as a weak relevance signal for your posts and help Google better understand the topic of the post. But if the posts are also indexed using categories, it can create duplicate content issue.
After you save changes, you can go to yourdomain.com/sitemap_index.xml to see and verify that sitemap generated by Yoast SEO.
So far so good. You have the right URL structure; the search bots can crawl the site and have the sitemap to find their way to different pages on the site. But there is still some work to do before we can actually invite the search engines to crawl and index the site.
Managing Crawl and Duplicate Issues Your WordPress Site
The sitemap is the directory of pages for search bots to crawl. But you will likely have links going from pages included in the sitemap to other pages that are not included in the sitemap. These include CSS pages, fonts, images and archive pages for users, categories and other taxonomies. You can use a combination of Noindex, Nofollow and Canonical tags and Robots.txt file to control how Google indexes your pages.
The first step in controlling indexing is to set the right Meta tags for pages on the site. Once again, Yoast SEO saves the day for us.
Select SEO > Titles and Meta.
On Post Types tab, for Posts and Pages, select:
- Meta Robots – Index
- Date in snippet preview – Show
- Yoast SEO Meta Box – Show
For media files like images, you should normally select Noindex unless you want search engines to index and include them in search results.
If the theme is using custom post types, you would most likely select Index option for them.
When you select Index, Yoast will generate a Meta tag that tells search bots to index the pages in the database. These pages will be included in relevant search results by Google.
The Noindex option, on the other hand, tells search engines to exclude the page from its index (database). These pages will not be included in the search results.
On the Taxonomies tab, select Noindex option for everything. Do the same on the Archives tab. Setting Noindex on these tabs ensures that the duplicate path to posts will not be indexed avoiding the duplicate content problem.
For example, when you set the Noindex option for Categories, search engines will not index the URL to the post via category page.
With Yoast SEO, you also set Indexing and Follow options at page / post level. This will come on handy to control indexing of pages like Cart page or Checkout page.
If you don’t want a page to be indexed but want the SEO benefit to pass to a page linked from this page, you should set it as Noindex, Follow in Yoast. This will instruct the search bots to not index the page itself but follow any outgoing link from the page.
If you have duplicate pages that you cannot avoid for any reason, you should designate one of them as original. On the other page, set the Canonical tag to the URL of the first page. Doing so will tell search bots that the first page is original and all SEO benefits coming to the second page will pass to the first page. Google loves Canonical tag and it will not penalize your site for duplicate content if you use it.
The Optimal Robots.txt file for WordPress
Every website has a Robots.txt file that tells a search bot which files and folders it can access. WordPress generates a default Robots.txt file that you should modify to suit your needs.
Since each site is different and has different pages or sections that serve specific needs of the site, you want to control how search engines index your site
You can access the Robots.txt file at yourdomain.com/robots.txt.
A Robots.txt file has three important instructions
- User Agent – specifies the bots to which the instructions apply. Default is * (all Bots)
- Disallow – the specified Bot(s) should not access files / folders included in the Disallow instruction
- Allow - the specified Bot(s) may access files / folders included in the Allow instruction
The ideal Robots.txt file for WordPress should not block anything other than specific files / uploads you want to protect
- Cart pages / images / protected files
- Category pages / archives etc. should not be blocked. Instead, these should use Noindex, follow settings from the Title and Meta settings in Yoast.
Submitting Sitemap to Google and Bing
Now you are ready to submit the sitemap to search engines for crawling. To do that, you need to create your Webmaster tools account (also called search console).
Log into your webmaster tools account and navigate to Crawl > Sitemaps.
Click ADD / TEST SITEMAP Button.
Enter the sitemap URL and click Submit.
Congratulations! You have just informed Google about existence of your site and its pages.
This should get Google search bot to come and crawl your site.
Getting Your WordPress Site Properly Indexed by Search Engines
Once you submit the sitemap, the search engine bots will crawl you site and get the pages indexed. But how do you make sure the search engines understand the content in your pages?
Search engines analyze Meta tags and page content to understand the content and figure out which relevance for keywords. You should use Yoast SEO to set the correct Meta tags.
Setting SEO Title and Meta Description for WordPress Pages
The ideal SEO title should start with the target keyword. The Meta description does not have a direct impact on SEO but a well-written one gets more people to click on your page in the search results.
To set these Meta tags, navigate to the page or post from the dashboard.
Scroll down to Yoast SEO box.
Click the Edit Snippet button. Enter a keyword optimized SEO Title and Description. Close the Snippet Editor.
Enter your target keyword in Focus keyword field. Yoast SEO will tell you if your page is optimized for the keyword. If it is not optimized, Yoast will also tell you how to fix the issues.
You now have a site with pages that are optimized for the right keywords and are indexed by Google.
Start building links to your pages and see your rankings improve.
Got something to share?
What did you think about this WordPress SEO guide?
Maybe you have a question.
Either way, Share your thought in the comments below right now.