Some sites seem to do really well in search engines just by ripping off other sites’ content and republishing it, possibly with an additional image or a changed headline, either with visible attribution or without it, using a correct rel=canonical or not.

Other sites that do this even minimally seem to suffer for it, whether by an algorithmic factor like Panda or a manual action or just by not being able to rank well.

Does Google penalize sites that have a lot of duplicate content?

Magic SEO Ball says: Reply hazy, try again.

One of the differences between professional SEOs and Google search quality engineers is that SEOs tend to think specifically and speak broadly, while Googlers tend to think broadly and speak specifically.

When an SEO – especially one who isn’t very good and doesn’t know enough to use precise language – says that his site was “penalized” or “punished” by Google, there are a few things he might mean:

1. I did something to harm my site (eg, server errors, page speed, blocking crawlers, indexation issues, canonicalization issues), which caused me to lose rankings, which caused me to lose traffic.
2. My competitors improved their sites, which caused them to gain rankings, which caused my site to decline by the same amount, which caused me to lose traffic.
3. Google changed its organic ranking algorithm to favor something that I’m not doing, or not doing well, which caused me to lose rankings, which caused me to lose traffic.
4. Google did something else completely different with search results pages, like knowledge graph or answer cards or seven-result SERPs or rich snippets or … which didn’t cause me to lose rankings at all, but which did cause me to lose traffic.
5. Google released an algorithmic ranking factor (eg, Panda, Penguin) and this apparently suppressed my site in search results, even though nobody at Google will ever be able to confirm this for me, causing me to lose rankings, which caused me to lose traffic.
6. I got caught doing something that violated Google’s webmaster guidelines, or I got caught not preventing someone from using my site to do something that violated Google’s webmaster guidelines, and Google put a manual action on my site, which caused me to lose rankings, which caused me to lose traffic.

When Google says “penalty,” however, they are talking about only one possible thing: a manual action (#6).

For instance: There is No Duplicate Content Penalty in Google. Here we have a representative of Google answering a question in the narrowest possible way to be able to say that there is not a penalty for duplicate content, which is technically true as long as you define “penalty” very narrowly, but which doesn’t even come close to answering the question.

Calling duplicate content a “filter” instead of a penalty is helpful for the five percent of SEOs who understand the difference – and there is a big and meaningful difference – between the two, and likely to be received as completely obfuscatory nonsense from the ninety-five percent of SEOs who just want an answer to this question: Is duplicate content bad?

So we will answer their question. Duplicate content is bad, for several reasons.

There actually is a duplicate content penalty

A later article in the same site linked above states very clearly that, in some specific cases, there can be a domain-wide duplicate content penalty, in the event that a certain site overwhelmingly uses other sites’ content without offering much unique material of any value. This is an actual penalty – a manual action, in Google’s words – that requires first fixing the problem and then submitting a reconsideration request to resolve.

The aforementioned duplicate content filter

As we all have seen, sometimes duplicate content is relegated to some index-below-the-index that isn’t even visible to searchers unless they click a link to view all the results.

Duplicate content opens the door to unnecessary canonicalization issues

There are a lot of ways to handle duplicate content on a site or among sites. One popular and recommended way is using rel=canonical to send the signal that a certain version is the preferred one, and that it should get the link equity of the others. The canonical tag does basically work most of the time, but it is only a good solution to a problem that’s fundamentally avoidable.

There are also a great many cases where the canonical doesn’t work as intended. For example, if the second domain has a vastly higher domain authority than the first, or if Google crawled the second version earlier and saw it there before it saw the original, or if the second gets far more links and shares than the first, the rel=canonical pointing from second to first may be ignored.

rel=canonical also does not send as much link equity as a 301 redirect, which means losing pagerank whenever it is used. It also needs to be engineered and tested and maintained, which can be a challenge for huge sites because it isn’t visible to users. Faulty implementations of rel=canonical, while now rare, are scary enough (imagine being the webmaster of the site that lost 98% of its traffic because every page suddenly had a canonical pointing to the home page) that one needs to act with caution.

Duplicate content should be avoided

Don’t avoid it at all costs, because there are some scenarios where it’s perfectly useful. But try finding a way to engineer elegant solutions that allow your site not to use duplicate content, wherever possible.

I have some manual actions in Webmaster Tools and I submitted a reconsideration request. Will anybody at Google actually read it?

Magic SEO Ball says: better not tell you now.

Magic SEO Ball is actually aware of a recent incident in which a website received two “manual action” penalties, cleaned itself up by removing all the spam, submitted a reconsideration request, received useless boilerplate drivel back from Google, submitted a new reconsideration request, and then still received the same useless boilerplate drivel back from Google.

After this, the website’s SEO director complained on Quora, on Stack Exchange and on Google’s own product forums.

After submitting the question in those different places and sending it to high level SEOs and a few Googlers, both penalties disappeared within twelve hours.

It would be hard not to interpret this as meaning that nobody at Google ever read the two reconsideration requests and that the site was never reviewed until he elevated the issue by publicizing it.

On the other hand, maybe the penalties were just scheduled to drop at a certain date, and that date was coincidentally the same as when he complained.

Should my site’s URLs without trailing slashes redirect to trailing slash versions? Or should my trailing slash URLs redirect to non-trailing slash versions? What if some page types do it one way and other page types do it another way? What about canonical URLs? And what’s up with file extensions at the end of URLs?

Magic SEO Ball says: concentrate and ask again.

Lots of issues here, some of which are referring to SEO best practices and others of which are closer to personal preference and which just require a simple decision and some consistency.

First issue: file extensions.

It made sense in the 1990s, when we manually wrote websites in individual html files, for those files to be uploaded using FTP clients into folders, and for the files in the folders to have URLs like page.html (or page.htm). This doesn’t make sense now. If you’ve built a new site in the past decade, its URLs probably shouldn’t be using file extensions like .html or .htm, .aspx or .asp, .php, &c.

Second issue: consistency.

Your site needs to have a coherent URL structure, and that coherence needs to extend all the way down to whether every URL has a trailing slash or whether not a single URL has one. Admittedly, this is in part a QA issue: you should never land on a page on your own site and not know if the URL is correct or whether you’re looking at a page that shouldn’t exist.

Pick one option or the other, trailing slash or non-trailing slash, and go with that option for all pages, without exceptions. Since you’re going with one or the other, implement redirects at the app level from the one you didn’t chose to the one you did choose: http://domain.tld/directory/page/ should redirect to http://domain.tld/directory/page or vice versa.

Third issue: choosing trailing slashes or not.

I recommend that you choose trailing slashes for two reasons:

1. Another QA issue: it’s easier to see that you’ve landed on a page with a trailing slash and it’s the correct page, and if you’ve landed on a page without a trailing slash know that it’s the incorrect page, than vice versa.
2. WordPress uses trailing slashes. WordPress is the default web publishing tool, so using WordPress-style URLs is a good idea.

Of course, this is largely a personal judgment call.

Fourth issue: canonicals.

You already know that a self-referencing canonical actually needs to reference itself, as opposed to a different URL that doesn’t exist.

In other words, if you’ve chosen to use trailing slashes, but you’ve created the URL incorrectly at http://domain.tld/directory/page instead of http://domain.tld/directory/page/, the canonical needs to be http://domain.tld/directory/page (the page that actually exists), not http://domain.tld/directory/page/ (the page that should exist).

Got that? Canonical URLs actually have to be for the pages as they exist, not as they ought to exist.

Of course, if both http://domain.tld/directory/page and http://domain.tld/directory/page/ resolve, fix the problem by redirecting according to your URL rules and make sure the canonical on the target page is correct.

Bonus: capital letters.

When you’re creating a new site and putting URL rules in place for trailing slash or non-trailing slash, you may as well decide that none of your URLs will use any capital letters, and that any URL with capital letters should be redirected to an all-lowercase version.

Bonus bonus: do the trailing slash redirect and the lowercase letters redirect in the same step so there’s only ever one redirect instead of two jumps.

Magic SEO Ball says: Concentrate and ask again.

Just on its face, the answer is basically yes, because it’s clear that not everybody in the world speaks English, so if you offer your content in multiple languages, people who aren’t searching for it in English will find it and that means improved SEO performance.

Followup: And if it does, what would be better to do: Have a single domain for all lanuages, i.e. de-de.domain.com for a german audience, or have a dedicated domain for each language, i.e. domain.de for the german audience.

This is really much more complex. In the past, Google representatives have stated that the gold standard in internationalization was to use country-specific TLDs, which would mean putting global content on domain.com, German content on domain.de, French content on domain.fr, &c. That addresses countries only, however, and not languages: there may be many people in Germany and France who prefer the English-language content and there may of course be many German and French speakers in countries besides Germany and France, such as in Austria and Canada.

Lately, we have conferred with some very high level expert SEOs who advise duplicating all content on domain.com, in region- and language-specific directories and using the rel=alternate markup to help search engines figure out which versions should appear to searchers in which countries.

We expect that the answer would be different based on different circumstances, such as the type of site (ecommerce, b2b, content, local-specific).

Based on a Quora question.

Magic SEO Ball says: Ask again later.

Magic SEO Ball fundamentally accepts Google’s contention that a relatively high number of nofollow links won’t hurt your site’s ranking. But this acceptance is with a caveat: those nofollow links won’t necessarily hurt your rankings now, but this may be something that is addressed in future algorithm updates, whether by adjusting a current ranking factor like Penguin or by adding a new one. You need to ask yourself whether your backlinks profile is abnormal for your industry or niche; if the answer is yes, is that because of something dodgy that you’ve done or because you’re the only one building links the clean way while your competitors are all dodgy?

Magic SEO Ball says: Don’t count on it.

Magic SEO Ball appears to be skeptical about Bing’s invitation to webmasters to disavow spammy links pointing to their sites. This makes a lot of sense, as our friends at Bing also appear to be uncertain about how Bing treats bad incoming links. As Vanessa Fox writes:

I still don’t understand what a site owner gets in return for the time spent disavowing links with this tool. I can only conclude that Bing may in fact lower a site’s ranking due to spammy incoming links.

If this is true, it’s contrary to what Bing says about how its ranking algorithm works. Perhaps Bing simply would prefer for webmasters to do the dirty work and identify all the bad links to save Bing the trouble.