Speed up reviews with autocomplete snippets

When reviewing themes for the WordPress.org theme directory, you will find that there are mistakes that are more common than others. This means that we often use the same type of messages in different tickets and reviews.

We can use this fact to make our reviews faster. I have done so many reviews that many of the texts are in my muscle memory, but I still save some of my reviews with the purpose of being able to copy paste them into other reviews later.

But we can improve on this further by using the code editor of our choice to add the most common texts for us.

I use VS code, and I followed these instructions to create my own review specific snippets:

In VS Code, go to File, Preferences, and select user Snippets

Now you can choose to create a snippet for a specific language, or a global snippet file. I write my reviews in a simple .txt file so I created a global file.

Documentation for how to create a new snippet is included in the file header of the new file, which made it very easy.

Place your global snippets here.
Each snippet is defined under a snippet name and has a scope, prefix, body and description. 
Add comma separated ids of the languages where the snippet is applicable in the scope field. 
If scope is left empty or omitted, the snippet gets applied to all languages. The prefix is what is used to trigger the snippet and the body will be expanded and inserted. 
Possible variables are: 
$1, $2 for tab stops, $0 for the final cursor position, 
and ${1:label}, ${2:another} for placeholders. 
Placeholders with the same ids are connected.

Example:
"Print to console": {
	"scope": "javascript,typescript",
 	"prefix": "log",
 	"body": [
 		"console.log('$1');",
 		"$2"
 	],
 	"description": "Log output to console"
 }

I decided not to use a scope, here is an example of one of my custom snippets:

"Missing license": {
	"prefix": "missing license",
	"body": [
	"Themes are required to be 100% GPL compatible.",
	"Reviewers can not confirm that a theme is GPL compatible unless all the license information is included in the theme.",
	"Missing license information for:",
	"(Placeholder: include additional information like a script name or image used in screenshot)",
        "$0",
	],
	"description": "Missing license information for included assets."
},

So when I write my review, all I need to do to add the information about the missing license is to start typing “license” or “missing” and press enter.

Here is the first iteration of the file: Download.

My Theme Review Process 2018

Updated december 2018

The Theme Review Team on WordPress.org encourages reviewers to find their own flow and tools that work for them. There is no right or wrong way to review a theme, as long as the requirements are checked and the author receives useful feedback.

My personal review process require a lot of manual effort.

Tools:

Since 2017, I have gone from VVV, via Local by flywheel which I used for a long time, to Docker.

While I liked the GUI of flywheel, it was just way too slow and sometimes buggy. I also found that it was not suitable for me when I wanted to do other projects than WordPress. Which happens about 3 times per year 🙂

Docker was easy to get started with, but there was a bit of a learning curve for me to get things exactly the way I wanted them. I now use two very basic docker compose files (with a persistent theme and plugin directory).

Sometime during the year, there was a big update to Sublime Text which broke all my settings. I could not get my extras like the PHP Code Sniffer to work again and truth be told, I gave up. I switched to VS Code, and I love it.

I often switch between different PHPCS standards. I use the WordPress-Coding-Standards and the WPThemeReview standard.

I still use Poedit to check translation files, and the tiny program called GifCam to create gifs of broken themes.

I use the theme unit test data, and we have also created a new version of the Theme Unit Test which includes blocks.

Plugins:

The Theme Sniffer and Themecheck

Preparations:

I always import the Theme unit test and I keep the requirements page and the WordPress developer reference open.

Process:

Normally it takes me 5-10 minutes to see if a regular sized (or underscore based) theme needs to be closed because there are 5 distinct issues, or if the review can continue.

But summarizing the findings and adding references, explanations and images to help the author fix the issues takes a little longer.

I try to consider whom I am writing the review for:

  • -Do I need to add references?
  • -Do I need to help them find a solution, or is it enough if I mention the issue?

Not all steps are performed on all themes, it depends how the theme is built.

I also split large themes over several reviews, because I don’t always have enough time to check several hundred files in one go.

If I quickly find 5 distinct issues I may close the theme as not approved without doing a full review.

First I check if the screenshot in the ticket is reasonable and not a logo or mockup.

Then I check the Theme and author URI before I download the zip file.

I look through the theme files before I install the theme.

If I do not recognise the theme author, I normally start by checking the license information in style.css and the readme.txt file, and check the theme and author URI.  This is to make sure that I am not spending time on reviewing a theme if the theme is not GPL compatible.

Then I continue by opening functions.php, header.php and footer.php:

  • Are there hard coded scripts?
  • Is there text that is not translation ready?
  • Is everything escaped?
  • If there is a footer credit link does it match theme or author URI?
  • Are the functions in functions.php prefixed and are scripts and styles enqueued correctly?
  • Are menus, widget areas and theme support added correctly?

Then I check if there are folders named js, css, dist, assets or similar, and if there are minified files. I also check if there are any plugins, demo files or similar included in the theme folders.

  • If there are minified files, is the non-minified version included as well?

This is when I start making notes and adding items to the review.

I already have texts written for common errors that I can add and adapt for the specific theme.

Run the plugins.

  • Are there any errors reported?
  • Check any warnings manually.

Then, depending on the theme, I use my code editor to search the entire theme folder for the following:

  • get_theme_mod -This helps me find theme mods that are not escaped.
  • remove -This helps me see if the author has removed any customizer settings or non-presentational hooks.
  • function, -This gives me a handy list to help me check prefixes and whether custom functionality is used in place of WordPress functionality.
  • script -This helps me find hard coded scripts.
  • alt=, title=, placeholder= -This will help me find translated texts in attributes and make sure the texts are escaped.
  • add_theme_support – This helps me check if the author has added support for title (required), custom logo (instead of using their own) etc.
  • If there are multiple post formats and templates I might also search for wp_link_pages, query and reset.

Then I install and activate the theme to make sure that:

  • There are no php or js errors.
  • There is no redirect on activation.
  • Notices are not global and can be dismissed.
  • Plugins are not required, only recommended.

Testing the theme layout and custom options.

  • Is the correct number of posts displayed?
  • Are comments displayed correctly?
  • Are there any php or js errors?
  • Perform a search.
  • Test the 404 page.
  • Test the navigation.
  • Set up a menu, is it working? Test the menu in a smaller window, is it working? Make sure that it is not overlapping or in other ways disturbing the admin bar.
  • Test custom widgets and make sure there are no php notices, warnings or errors.
  • Make sure that there is only minor content creation.

When I go to the customizer and test the options, I also open the customizer.php or related files in my editor and make sure the options are sanitized and set up correctly.

  • Check to see if there is demo content, content creation or plugin territory functionality.
  • Is the upsell reasonable and is it added with the customizer API?

Go to the post editor and double check if any meta fields has been added, and that they are for presentation, not content.

  • Test the meta options and repeat for pages.
  • Test page- and post templates.
  • If there is starter content, reset the install to fresh and test it.
  • Does the theme match the tags and description in style.css?

Whether I find issues or not, I still open each file in the theme folder manually and review them quickly, including minified files. This is still the fastest way for me to check for untranslated texts and anything that stands out.

When the review is finished, I save it as a .txt file and then cut and paste the text in the Trac ticket.

Getting ready to do accessibility reviews

The scope of this post is reviewing accessible themes submitted to WordPress.org. Updated December 6 2017.

The accessibility-ready tag has been available in the WordPress.org theme directory for over 2 years. Themes that include the tag are not set live until the theme has had a full review plus an additional accessibility review.

While most agree that accessibility is important, the number of theme reviewers who are comfortable reviewing it is still very low; no more than 4 people at any one time.

Some of the reviewers are members of the WordPress.org accessibility team and their main focus is on improving the accessibility of WordPress as a whole, and they only have limited time to help out with theme reviews.

This means that:

  • The waiting time for accessible themes are even longer than for regular themes.
  • Author’s remove the tag because the waiting time is too long.
  • Authors submit accessible themes without adding the tag, so it becomes more difficult for users to find accessible themes.

I personally believe that the low number of reviewers is because of lack of confidence: Reviewers does not want to promise that a theme is accessible if they are not 100% sure.

So how can we encourage and make it easier for active reviewers to start reviewing accessibility (and usability)?

First of all I would like to assure you that:

  • It is not more difficult to do an accessibility review than to do a regular theme review.
  • It does not take longer than a regular review.
  • You do not need to test the theme with a screen reader.
  • The members of the theme review or accessibility team can help answer your questions.
  • The world will not end if you missed something during the review*.
  • The theme author is responsible for making the theme accessible, not the reviewer.

The requirements for using the accessibility-ready tag each have a section named “How to test” and suggestions of tools to use, so I could even argue that the accessibility requirements are easier to test than some of our other requirements.

You can literally open the accessibility requirements page, read the first requirement and start testing it simply by tabbing through the selected theme. -If you can’t see the focus, then the theme needs to be updated.

But it is important to remember that as with any part of the theme review, it becomes easier and much more valuable for you as a developer if you also have a basic understanding of why something is required. 

The accessibility requirements are based on a guideline from W3C called WCAG 2.0, or Web Content Accessibility Guidelines 2.0 (I am sure that you have heard of this guideline before).

WCAG has 3 levels: A, AA, and AAA. The theme accessibility requirements mainly refer to level AA.

Even when we have a list of requirements, everything cannot be covered, and we need to use both our own judgement and our collective knowledge as a team. This is true for all kinds of reviews that we do. But in cases where the accessibility requirements does not cover everything, we don’t need to rely on trends or vague and varying “best practices”, instead, we can go back to read the WCAG.

I understand that trying to read and make sense of the official documents can feel overwhelming, and luckily for us, there are several other people who have recognized this and has written some helpful guides.

Web Content Accessibility Guidelines—for People Who Haven’t Read Them

Mozilla: Understanding the Web Content Accessibility Guidelines

How to do an assessment -From Duke University

The first thing to be aware of is that using an accessible WordPress theme does not automatically mean that a website is accessible, because the content must also be accessible. That a theme passes the review does not mean that the website is compliant with WCAG 2.0, or with the U.S section 508, which is sometimes believed.

Steps to get ready

I recommend the following steps to get ready to review accessible themes. If you need to skip a step, you can always save the resource and refer to it later.

  1. Read the WordPress accessibility coding standards.
  2. Read the Web Content Accessibility Guidelines. https://www.w3.org/TR/WCAG20/ 
  3. Visit and subscribe to the blog of the WordPress.org Accessibility team: https://make.wordpress.org/accessibility/
  4. Join the accessibility team channel on Slack (Please be respectful and remember that it is not a support channel).
  5. Read about the requirements, recommendations and review process for accessible themes.
  6. Take the WordPress accessibility course by Joe Dolson on Lynda.com. Joe is a member of the accessibility team, a plugin and theme developer and the one who has done most of the accessibility theme reviews.
  7. Sign up for A11yweekly, a newsletter by David A Kennedy. It includes a section called “New to A11y” and has a lot of good tips and links to resources.  You can also find the previous issues here. David has helped us with accessibility reviews in the past and is one of the maintainers of the default WordPress themes.
  8. Read and learn from other reviews:
    1. All open tickets with the accessibility-ready tag are in a separate queue on Trac: https://themes.trac.wordpress.org/report/9
    2. You can read old reviews by listing accessible themes in the directory and using the Trac Tickets link on the themes page.
  9. Don’t forget to test and improve your own themes! Submit a theme with the accessibility-ready tag to go through the process yourself.

I also recommend the following videos on WordPress.tv:

Adrian Roselli: Selfish Accessibility

Rian Rietveld: The Accessibility-Ready Tag for Your Theme – Why and How

Rachel Cherry : Understanding and Supporting Web Accessibility

Graham Armfield: Designing for Accessibility

 

 

*-The process in case you happen to miss something during an accessibility review is the same as for any other requirement: If we find out that a live theme does not pass a requirement, the author will be notified in ticket and we will ask them to submit an update. If the problem is not solved, we can ask the author to remove the tag, and ultimately in a worst case scenario, the theme can be suspended from the directory.

How to get your theme live faster

When authors submit themes to the WordPress.org theme directory they are sometimes surprised that they might need to wait up to 8 weeks for a review.
One of the resons to why it can take this long is that many authors submit themes that are incomplete.

By checking these four things you can greatly reduce the time it takes to review your theme:

  • License
  • Code Errors
  • Security
  • Translation

Back to basics

The list of requirements is long and we as reviewers understand if authors miss some of them. In fact very few themes are set live in the theme directory without changes.
However, I feel strongly that theme authors, –especially those who sell themes -at least should get the basics right.

License

WordPress themes need to be 100% compatible with GPLv2 or later.
The theme directory cannot redistribute your theme if you use items that have a license that limits redistribution or commercial use.
I expect authors to understand the difference between copyright and license. I expect them to include information about both in the theme.
If a theme has a commercial version,  that version also needs to be 100% compatible with GPL.

How to check:
Open style.css. Make sure that your stylesheets file header has a license statement.

Examples:
License: GNU General Public License v2 or later
License URI: http://www.gnu.org/licenses/gpl-2.0.html
or:
License: GNU General Public License v2 or later
License URI: LICENSE
Where license.txt is included and contains a copy of the license.

Open the readme file.  Make sure that you have included a copyright notice for the theme.

Example:
Twenty Seventeen WordPress Theme, Copyright 2016 WordPress.org

Make sure that you have listed the names, source, copyright and license information for all resources like third party scripts, styles and images, including images used in the screenshot, since the screenshot is part of the theme.

Code

Do not include code that you don’t understand. This might seem obvious,  but many authors copy and paste code from others without understanding what the code does. Keep your code simple. Comment and document your custom code. The reviewer might need to ask questions about your code, and more advanced code takes longer to review.

Do not include code, files or folders that are not used in the theme. Asking us to review code that is not even used will delay your review further. It’s a waste of our time.

Test your theme on different PHP versions. Test all your custom page templates, options, and custom widgets. Check for JS errors and missing files. Do not rely on the reviewer to do this for you. The reviewer is not your tester, we expect you to check this before you submit the theme.

Security

We understand that security can be the hardest part. But there are some simple rules that you can use when developing and checking your theme.

  • Don’t trust any data.
  • Escaping and sanitizing:  You can’t have one, you can’t have none, you can’t have one without the other.
  • Don’t include external files. All scripts, styles and images should be included in the theme. The exception is google fonts.

Untrusted data needs to be validated and/or sanitized before saving, and escaped as late as possible before output. Theme options are not considered safe.

By far the most common problem that we see in themes is missing escaping, or using the wrong function.
Untrusted links should be escaped using esc_url().
esc_attr()  -Attr stands for attribute. This should only be used inside actual html attributes like title, alt, width, height, placeholder and similar. It should not be used between html tags.
esc_html() This should only be used between html tags. It should not be used inside html attributes.

The customizer has a range of different control types, and authors can also add custom controls.  We often find that authors use sanitize_text_field() for all their control types, even controls that are not text fields.
Use a sanitize_callback that is suitable for your control type.

How to check:
Search your theme files for: sanitize_callback, esc_attr, esc_html and echo get_theme_mod.

Make sure that all text strings are translatable

There is no shortcut for this. The best way to make sure that all your texts are translation ready is to add the translation functions when you build the theme, instead of trying to add them afterwards. Finding text strings that are missing translation functions is a very time consuming job, where both the reviewer and the theme author needs to manually open all files -including JS files, to find these text strings.
You can understand why this slows the review process down for everyone, and why this increases the waiting time.

 

 

Help the reviewers and your fellow theme authors to reduce the queue time for everyone by only submitting complete, tested themes.
By checking these four things before you submit your theme, you have also reduced the risk of having your theme closed as not approved.

The Theme Review Team is open to everyone, and you can help out by reviewing submitted themes for these issues. With the help of more reviewers, the waiting time will be shortened.

Become a Reviewer

My theme review process

Updated September 2017.

The Theme Review Team on WordPress.org encourages reviewers to find their own flow and tools that work for them. There is no right or wrong way to review a theme, as long as the requirements are checked and the author receives useful feedback.

My personal review process require a lot of manual effort. I know that some of the things that I do manually can be replaced by the new Theme Sniffer plugin, and I will eventually adjust to that.

Tools:

I use VVV (on Windows 10) but also have a live install available in case I need to show the author something.

I use Sublime Text code editor with the PHP Code Sniffer installed.
I like to use Poedit to check translation files, but Sublime Text can also read the pot files.
To create GIFS (for example to illustrate a broken menu), I use GifCam.
The theme unit test data: https://codex.wordpress.org/Theme_Unit_Test
A .txt document with my  standard review text.

Plugins:

The new developer version of the Theme Sniffer.

Preparations:

I always import the Theme unit test and I keep the requirements page and the WordPress developer reference open.

Process:

Normally it takes me 5-10 minutes to see if a regular sized (or underscore based) theme needs to be closed because there are 5 distinct issues, or if the review can continue.
But summarizing the findings and adding references, explanations and images to help the author fix the issues takes longer, from 20 mins and up.

I try to consider whom I am writing the review for:
-Do I need to add references?
-Do I need to help them find a solution, or is it enough if I mention the issue?

Not all steps are performed on all themes, it depends how the theme is built.
I also split large themes over several reviews, because I don’t always have enough time to check several hundred files in one go.
If I quickly find 5 distinct issues I may close the theme as not approved without doing a full review.

First I check if the screenshot in the ticket is reasonable and not a logo or mockup.
Then I check the Theme and author URI before I download the zip file.

I look through the theme files before I install the theme.
I normally start by opening functions.php, header.php and footer.php:

  • Are there hard coded scripts?
  • Is there text that is not translation ready?
  • Is everything escaped?
  • If there is a footer credit link does it match theme or author URI?
  • Are the functions in functions.php prefixed and are scripts and styles enqueued correctly?
  • Are menus, widget areas and theme support added correctly?

Then I check if there are folders named js, css, dist, assets or similar, and if there are minified files. I also check if there are any plugins, demo files or similar included in the theme folders.

  • If there are minified files, is the non-minified version included as well?

Then I open style.css, readme and license files and make sure that the license and copyright information for the theme and the assets have been added.

This is when I start making notes and adding items to the review.
I already have texts written for common errors that I can add and adapt for the specific theme.

Run Theme Check

  • Are there any errors reported?
  • Check any warnings manually.

Then, depending on the theme, I use my code editor to search the entire theme folder for the following:

  • get_theme_mod -This helps me find theme mods that are not escaped.
  • remove -This helps me see if the author has removed any customizer settings or non-presentational hooks.
  • function, -This gives me a handy list to help me check prefixes and whether custom functionality is used in place of WordPress functionality.
  • script -This helps me find hard coded scripts.
  • alt=, title=, placeholder= -This will help me find translated texts in attributes and make sure the texts are escaped.
  • add_theme_support – This helps me check if the author has added support for title (required), custom logo (instead of using their own) etc.
  • If there are multiple post formats and templates I might also search for wp_link_pages, query and reset.

Then I install and activate the theme to make sure that:

  • There are no php or js errors.
  • There is no redirect on activation.
  • Notices are not global and can be dismissed.
  • Plugins are not required, only recommended.

I don’t spend more than a few minutes on testing the theme layout, more effort is put into testing any custom options.

  • Is the correct number of posts displayed?
  • Are comments displayed correctly?
  • Are there any php or js errors?
  • Perform a search.
  • Test the 404 page.
  • Test the navigation.
  • Set up a menu, is it working? Test the menu in a smaller window, is it working? Make sure that it is not overlapping or in other ways disturbing the admin bar.
  • Test custom widgets and make sure there are no php notices, warnings or errors.
  • Make sure that there is only minor content creation.

When I go to the customizer and test the options, I also open the customizer.php or related files in my editor and make sure the options are sanitized and set up correctly.

  • Check to see if there is demo content, content creation or plugin territory functionality.
  • Is the upsell reasonable and is it added with the customizer API?

Go to the post editor and double check if any meta fields has been added, and that they are for presentation, not content.
Test the meta options.
Repeat for pages.
Test page- and post templates.

If there is starter content, reset the install to fresh and test it.

Is there any functionality or special items that should be granted an exception from the requirements?
Does the theme match the tags and description in style.css?

Whether I find issues or not, I still open each file in the theme folder manually and review them quickly, including minified files.
This is still the fastest way for me to check for untranslated texts and anything that stands out.

When the review is finished, I save it as a .txt file and then cut and paste the text in the Trac ticket.

By saving the reviews, I have built up a library of texts that I can re-use for other themes with similar problems.

Tips

The more themes you review, the easier it will be for you to find things that stand out, and your reviews will be faster.
Read the WordPress code, read more themes and also read other peoples reviews.
Read the theme developer handbook and get familiar with the developer code references.
Learn Underscores.
Share your knowledge with other reviewers, and accept that you can’t know everything, -ask questions when you are not sure about something.