Spy tools can be useful in an SEO and/or a PPC campaign. However, many of these tools essentially try to extrapolate scraped results which can lead to some fairly inaccurate results. Also, these tools occasionally come up with in-house metrics (of which they really don’t give you much useful info about how they arrived at the data the present from these “proprietary” metrics”) to help try and differentiate their offerings from their competition.

Spy Tool Reviews

There is a much more in-depth review, with examples, up in our members forum. Here, we will do overviews of some of the more popular tools on the market. Specifically, we will be taking a look at:

• review of SEM Rush
• review of SpyFu
• review of Keyword Spy
• review of Compete.com
• review of Ispionage

Value of Spy Tools

The idea that you are missing out on something is a core marketing tactic so even if you are comfortable with one tool chances are you’ve been tempted to go with another. Keep in mind, from a cost standpoint, the ROI you would take by just finding a few decent keywords to target will likely far outweigh any cost associated with these tools. Your business probably won’t collapse if you pick an A minus tool versus an A plus tool and none of these tools are able to make concrete decisions for you. What these tools provide are additional data points for you to consider in your own research.

We hope you’ll find these reviews useful. There are perhaps a few other services we missed given how many of these tools as there are and our primary focus on SEO. If these reviews are well received we could also review everything from Quantcast & Alexa right on through to AdGooroo, but we need to know if you would be interested in those types of reviews. If there are any other cool products or services you would like us to review just let us know.

A few disclaimers: some of these services have given us free review accounts, whereas we have paid for some of the others. And some of these tools offer affiliate programs, but all reviews were done without those 2 factors influencing the editorial. Most these reviews do not have affiliate links in them (I think SEM Rush is the only one which does have an affiliate link right now), and Aaron reviewed SEM Rush before they even had a public affiliate program.

Already an SEO Book subscriber?

• We license SEM Rush’s SEO data for our competitive research tool.
• The second post in this thread has some killer member’s only tips for getting the most out of competitive research tools.
• This thread is also awesome, and these three threads really shows how to use these tools in a way that really kicks arse.

If you can’t make $1,000’s from reading those threads then you certainly are not a professional grade SEO.

Related posts:

1. Keyword Spy Review
2. ISpionage Review
3. SpyFu Review

Compete Site Profiles

Here is how compete gets their data.

Here is a screen shot of their site profile overlay

It’s kind of like a semi-analytics program view of things which includes:

• Unique Visitors
• Page Views
• Average Stay
• Demographic Info
• Link through’s to Referral and Search Analytics (discussed further down)

Data is available in 7 day, 30 day, 3 month, 6 month, 1 year, and 2 year increments.

The audience profile tab is similar to quantcast and is only available to the verified site owner (unless the site has made it’s info public) and the sub-domain tab shows sub-domains associated with the main domain.

Enterprise users, where there is no standard pricing listed…also get access to category profiles and behavioral categories as shown below:

You also get the option to compare up to 5 sites at once in their site profile section

Those are the options in the profiles section. These statistics are far beyond what most traditional spy tools offer and can be very useful when comparing large sites as small sites do not fare very well with these types of data sets (this is not specific to compete, it’s pretty much industry wide).

Analytics Tools

Compete’s second tool set is the Analytics Tools set. Here you can search through Search Analytics (keywords) and Referral Analytics (sites referring traffic to the domain) as well as a variety of Ranked Lists.

Referral Analytics

This is pretty sweet as you can see what search engines the site’s SEO campaign is doing well in, as well as possible advertising opportunities for your site.

It also will show you Destination sites (where users go after landing on the site you are reviewing.)

In addition to messing around with some of the filters you can take a peek at historical data (trends, seasonal, etc) as noted here.

Ranked Lists

Compete offers ranked lists which you can filter in a few easy steps

Compete lets you look at ranked lists via 3 steps (one from each)

• Step 1 – unique visitors, visits, page views, time spent, monthly attention
• Step 2 – site ranking, ranking + unique visitors, ranking + all metrics
• Step 3 – top 200, 1,000, 15,000, 100,000, 500,000 domains

Search Analytics

Compete’s Search Analytics show keywords referring traffic to a site (or two) with some pretty neat metrics:

• Highly Engaging Keywords – Keywords that make up 40% of the total time index and have a referral share greater than 0.01%
• High Traffic Keywords – Keywords that make up the top 40% of the search referral share
• Paid Keywords
• Natural Keywords
• Engaging Long Tail Keywords – Keywords making up the bottom 60% of search referral share, with a total time index of > .10
• Enthusiast Keywords – Keywords that make up the top 40% of Average Time Index and a Search Referral Share greater than 0.01
• Long Tail Keywords
• Total Time Index – scale of 100 with 100 being the term where the searcher came from…that made up the highest total time spent on the site for ALL visits.
• Average Time Index – scale of 100 with 100 being the term which resulted in the most average time per visit spent on the site.

You can also compare 2 sites like so:

In Closing

The high price point of Compete might scare some users away, but consider that their data is not just relying on scraped Google/Yahoo/Bing results then extrapolated by some internal metrics. Compete is probably more useful to those who “compete” in really competitive markets with some sites as competition, although it can be useful to folks who may be involved in less competitive SERPS with smaller sites as competitors because they can use this data to investigate larger sites in their market, which may not be competitors but could yield helpful industry data.

Related posts:

1. ISpionage Review
2. SpyFu Review
3. Keyword Spy Review

• Keyword and Domain Research
• Keyword Monitor
• PPC Campaign Builder

Keyword and Domain Research

They index the top ten results in Google, Yahoo, and Bing (although I only saw G and Y).

They give you breakdowns of common spy tool elements such as:

• Competitors and Overlapping Keywords
• Ad Copy
• Keyword Specific Ads
• CPC
• Average Search Volume
• Average Rank
• And so on..

The one really neat thing they offer is overlapping keywords between Yahoo and Google for a particular domain. I’m not aware of another spy tool that does that.

Their database does not seem to be very deep but they are newer so that’s to be expected.

The do show overlapping keywords, total keyword count, and a monthly budget under their competition tab.

Here is another spot where they compare Google and Yahoo, this time for overlapping keywords between sites.

Keyword Monitor

This lets you search by domain name or keyword to get ideas for keywords to add into your campaign. You can also add your own manually after the keyword research option. Keyword Monitor will show you the following for your campaign + competition:

The impression share is not something I’ve noticed in most other tools and the other 4 metrics can be useful in determining which competitor might be a bit savior in the PPC game. Other metrics they will show you on the keyword level include whether or not the keyword has direct ranking affiliates, the average CPC/search volume, and total advertiser counts in Google, Yahoo, and Bing.

The tool also shows you related keywords you may wish to add to your campaign or just place on your watch list.

PPC Campaign Builder

The campaign builder allows you to search for keywords via a keyword or domain name input. The steps are as follows:

Keyword Clean Up

This is where you can weed out keywords that contain certain words, are duplicates, or have special characters. You can also choose to remove extra spaces if needed.

Keyword Grouping

Here you can set up ad groups and campaigns right from within iSpionage. It also gives you the option to create one ad group per keyword if you want to get that granular

Campaign Builder

Here you can input bid prices for Broad/Phrase/Exact match bids, set up your ads, and input the url. Then you can export for use in Google, Yahoo, or Bing PPC campaigns.

In Closing

iSpionage has some promise and seems to be much more into the PPC market than the SEO market. If that’s the case then they are taking on some pretty big players as many of the spy tools offer both PPC and SEO data sets. They have some unique features and it will be interesting to see how they develop their product going forward.

Related posts:

1. SpyFu Review
2. Keyword Spy Review
3. Compete.com Review

Keyword Spy offers 3 different accounts.

• Research
• Tracking
• Professional

The countries available within a Keyword Spy account are:

No other competitor really comes close to the breadth of their country offerings.

Keyword Spy Research Account

Keyword Spy’s Research account gives you access to the following data

• PPC Ads (ad copy, the keyword, estimated search volume, estimated CPC, the position last seen of an ad and it’s average position, total days seen/days checked. You can also see the ad url and destination url of the ad in addition to other keywords being bid on for that particular ad, as well as an estimated ROI.
• PPC Keywords – showing individual keywords, ROI, search volume, CPC, total profitable ads, affiliate ads on that keyword, days seen, last/first seen
• Organic Keywords – showing individual keywords, position in the SERPS, total search results, estimated CPC, and the URL
• Competitors in PPC and Organic results.
• Sub-domains

The research portion does *not* include organic or PPC overlap coverage, which kind of stinks especially when you consider the price point they charge.

You get access to their Top 1000 sites and keyword reports which can be previewed here.

You can search by keyword as well. A Keyword search will show you:

• PPC Ad Copies with Keyword in them
• Up to 1000 related Keywords
• Misspelled Keywords
• PPC Competitors
• Organic Competitors

You can filter with these metrics but you can only apply 1 filter to the results at a time. Which can be bothersome if you are doing large scale research as they limit the exported data to 50,000 keywords.

Research Account Metrics

• ROI- they compute this as (Days Seen*Percentage Seen/Number of Days Seen since Last Seen). Below is a screen shot of their formula. Again, this is based on the assumption that the PPC advertiser is shrewd and on top of things. I don’t particularly care for this metric. ROI to person A can be much different than ROI to person B for a variety of reasons.



• First/Last Seen- Last seen is the last day KS saw the ad (they scan daily) and First Seen shows the first day KS saw the ad (I believe its back to August of 09 as of this writing).
• Profitable Ads – Ads that are profitable based on their internal metrics (like ROI and such) out of total number of ads.
• Affiliate Ads – Ads that are affiliate ads (based on destination url) out of total ads found.

Screen shot of PPC keyword tab showing the above mentioned metrics:

Keyword Spy’s Tracking Account

The Tracking account option gives you real time tracking in Google, Yahoo, and Bing for your PPC and Organic campaigns. This can be useful in checking out your coverage and competition across all three engines. You can also benchmark your data with the competition’s scraped data.

Of course, the question is do YOU want your campaigns being monitored by a spy tool that makes its money but showing advertisers their competition’s organic and PPC data??

You can read about more of their tracking/alert/coverage type options here, but outside of tracking and coverage you get:

• Landing Page Intelligence – shows current landing page, ad copy, and destination URL for a particular landing page.



• Organic and PPC overlap data (only between 2 sites) and quite frankly, this is much more research than tracking and should really be included in the research account IMO.
• Benchmarking in PPC/Organic Listings (below is a screen shot of the organic one, they are fairly similar)

So the tracking account is really more for tracking your campaigns across the 3 big PPC engines with some nifty benchmarking and gap analysis features but I don’t see it as being overly useful for smaller PPC advertisers, although the coverage options might be a good fit for those in competitive markets across Google, Yahoo, and Bing. In general, Spy Tools aren’t all that great at looking into smaller sites and markets simply because the resources required to be accurate with somewhat sparse data would be overkill and far to costly. This is why I do not really feel the tracking option is going to be a good fit outside of pretty big PPC advertisers.

Professional Account

The Pro account combines the Research and Tracking account features (up’s the overall trackable keywords, export limit, and query limit) plus gives you access to a couple new features:

• Affiliate Intelligence
• Affiliate Reports

Affiliate Intelligence

This tool gives you access to look at products and ads being used by 132 affiliate networks.

You can click through on any network and be shown their offers by URL with searchable affiliate ads for those products.

Affiliate Reports gives you access to big players in the affiliate marketing space such as CJ, LinkShare, Clickbank, ShareAsale, etc. Here you can access top affiliate products and top affiliates by product id and affiliate id respectively. You can also use affiliate product and affiliates id’s to search in the destination URL field to try and find additional products/ads they may be promoting.

Keyword Spy mentions something about “Anti-cloaking” technology but they do not elaborate on it. However, color me skeptical that these affiliate options are able to uncover properly cloaked links by top affiliates. So while this may be good for help in looking at potential affiliate products, as well as finding affiliates who do not cloak their links, I’m really not overly impressed with these features but they can be somewhat useful when first starting out.

In Closing

Keyword Spy is a feature rich membership and they have a deep database. For me, if I had to pick just one tool I would opt for either SemRush or SpyFu as both supply solid PPC/SEO competitive intel at a much more reasonable price. Although, if I were a serious PPC player their tracking account might be quite nice (still have reservations about giving a spy tool company my campaign data though.

Related posts:

1. SpyFu Review
2. ISpionage Review
3. A Review of Spy Tools

SpyFu is one of the more feature rich tools, but probably has the least attractive interface out there. SpyFu offers SEO and PPC spy tool options along with their own keyword research tool.

The SpyFu toolset covers US and UK markets.

SpyFu’s toolset includes:

• SpyFu Kombat
• SpyFu Classic
• Keyword Ad History
• Domain Ad History
• Keyword Smart Search
• A Variety of Top 100 Lists

SpyFu Kombat

With SpyFu Kombat you can look at overlapping and site specific keywords for up to 3 websites. For the PPC version you can also see a chart which goes back over a period of a few years showing the overall amount of keywords being bid on by all three sites. You can also rollover the chart to see keywords specific to just 2 of the sites if you feel the 3rd site may not be doing as good a job (or vice versa) as 2 of the other sites. It will also show you the PPC budgets of the sites as well as the number of organic keywords ranking in the top 50 results for said keyword.

When you click on an area of the circle chart it will show you the keywords in whatever bucket you click, to the right of the chart. You can view and download those keywords for your own use. As you can see I am on the ads tab but the options are similar when you click on the organic tab (on the top box, the organic one on the bottom shows you total organic keywords).

Switching between the organic tab and the ppc tab (as well as the overall # of organic keywords + PPC ad budget should also give you an idea of which of the bigger sites are more into the PPC or SEO side of things which can be a good barometer to look at if you happen to be concentrating on one area over the other.

SpyFu Classic

SpyFu Classic is the “flagship” section so to speak. This is where you enter one domain on the home page and are presented with a TON of data including:

• Daily AdWords PPC Budget
• Links through to SpyFu Kombat
• Average Position of Ads vs # of Advertisers
• Estimated Value of Organic Traffic (estimated traffic with a variable of CPC factored in)
• Paid Traffic Compared with Organic Traffic Estimates
• Subdomains (useful for looking at how a site might break out parts of the main domain, perhaps a good spot to look for niche keywords???)
• Top Ten Paid Keywords w/ Keyword Ad History (links through to full Keyword Ad History tool)
• Total Paid Keywords
• Total Organic Keywords
• PPC Competitors (with a link to overlapping keywords)
• Organic Competitors (with a link to overlapping keywords)
• Category

In addition to searching for a domain SpyFu let’s you search by keyword as well, as shown below:

The data here can be useful, as you can see the:

• Estimated PPC, Clicks, Cost Per Day, Total Advertisers…all with trend data
• Top Ten Domains Advertising on the Keyword, with Domain Ad History
• Additional Keywords Purchased By Relevant Domains
• PPC Ad Copy with a Link to Keyword Ad History
• Top Ten Organic Results with Title, Meta Description
• Related Terms
• Related Concepts (based on semantic relationships)
• Categories

Keyword Ad History

Keyword Ad History will show you, via color coded bars, how often the keyword appeared in a domain’s PPC campaign along with any changes in the ad copy (all of which can be exported to excel). It shows a year’s worth of data up front and goes back to 2006 via the Bonus History Button.

So it’s pretty straightforward, which is what I like about SpyFu Tools. No over-reliance on “in-house metrics” it’s just “here’s the ad history of the keyword”, plain and simple. Typically, if you see a keyword being advertised on by a good PPC advertiser consistent then you can look to apply that ad copy technique to a niche market of that larger keyword. If I were advertising for “hotels in Oklahoma” I might pay attention to what ad copy has been successful, over time, for that main/core keyword “hotels”.

Domain Ad History

Domain Ad History is similar to Keyword Ad History except it shows the keyword history of a particular domain:

This tool is useful in looking at keywords that have been successful for your competitors (or larger players in your niche) and which ones they tried and abandoned (which could be ones for you to avoid out of the gate). All of this assumes the domain you are researching is competent PPC advertiser.

Keyword Smart Search

The Keyword Smart Search tool in SpyFu uses semantic word relationships, publicly available keyword data, and PPC campaign data to return a list of keywords related to the keyword(s) (up to 10) you enter. As you can see, you can also filter by CPC, search volume, and you can also exclude keywords:

Here is a screen shot of the results page for Keyword Smart Search:

For me, I prefer to use the PPC keywords and the Organic keywords found in either SpyFu Classic or SpyFu Kombat. I like to use other tools for pure keyword research (Google tools, Microsoft Ad Center Intelligence, and Wordtracker). Primarily, I feel SpyFu is at its best when used as a competitive research tool versus a keyword research tool.

A Variety of Top 100 Lists

They have a list of all there Top 100 Lists here.

In Closing…

I find their tools pretty useful for competitive research. I don’t use their Keyword Smart Search much as described above but the amount of data that they give (in a straightforward fashion) at the price points they give is quite a nice combination. SpyFu makes its way into my toolbox on just about every project.

Related posts:

1. Keyword Spy Review
2. ISpionage Review
3. A Review of Spy Tools

In the past I have not been a fan of certain outing policies, but of late I have seen that practice has went away…and if it stays that way how could I not recommend the above tool?

Sometimes it is hard to appreciate how spoiled we are as SEOs with cheap to free keyword data, cheap to free great link data, and lots of useful tools to help us organize and make sense of it all. And even lots of charts

One area where some of our tools could be better is on the usability front…we tend to presume some level of knowledge and/or the willingness to work through things to figure them out, but the presentation on OSE is very easy to grasp & understand at first glance. Part of this challenge comes from limited resources…and the most limiting one being time. It is so hard to make money servicing the SEO market (because there are so many great free options). As the market continues to open up more with more tools and options, at the same time the SEO *process* keeps getting more complex, with more competitors jumping into the SEO market.

It certainly feels like it will keep getting easier to make money as a publisher rather than a person servicing the SEO market.

But not all forms of publishing will get easier & more profitable. Companies like Demand Media and Aol sharing their results publicly will saturate some segments, but there are many areas where bullshit content won’t be enough to compete. And some thin operations will see margin contraction as the investment needed to stay competitive increases.

But we are quite literally drowning in opportunity. If a person can’t make money as a publisher with SEO knowledge, it is simply because they are not willing to put in the effort (or they are part of an old bureaucratic publishing company which moves slowly, is debt laden, and has a high cost structure).

I have always avoided scaling as a company, but there is so much opportunity that I might have a resolution for 2010

Related posts:

1. I Stopped Caring About Links (Well, Almost)
2. A Review of Spy Tools
3. Is SEO Science or Art?

For December 20, StatCounter has IE 7 at 21.2% market share, and Firefox 3.5 at 21.93%. Microsoft’s IE 8 is closing in though. Its share has been steadily increasing, presumably making up for most of the lost IE 7 share, and has approached 20.33%.

The top browsers are as follows:

1. Firefox 3.5
2. IE 7
3. IE 8
4. IE 6
5. Firefox 3.0
6. Other
7. Safari 4.0
8. Firefox 2.0
9. Opera 9.6

Last month Mozilla celebrated the fifth birthday of the Firefox browser. They said that in the first four days of launch, over a million people downloaded Firefox 1.0. Firefox is available in over 70 languages and offers over 7,000 add-ons.

Firefox 3.5 also had over a million downloads in its first day of launch, and Mozilla surpassed a billion downloads in late July. Firefox has come a long way in five years and has really given Microsoft a run for its money in the web browser space.

Related Articles:

> Mozilla Celebrates The Fifth Birthday Of Firefox

> 8 Firefox Add-Ons to Boost Productivity

> Firefox Nears 1 Billion Downloads

Related posts:

1. Mozilla Says New Firefox 20% Faster Than Last Version
2. Featured FireFox Addon: See Which Software the Site Uses with Wappalyzer
3. Yahoo to Be Default Firefox Search Engine on Ubuntu

First of all I think it is a bad idea because if Google owns the search engine and the browser then maybe that is not the best spot to have your SEO research stuff hooked up, but even beyond that I don’t think we would make $1 more by creating those tools. Why? Because the people who use Google Chrome for SEO research are not the type of people who want to pay for anything related to SEO (outside of buying links perhaps).

My buddy Patrick from BlogStorm mentioned ‘Imagine all the links you would get from people writing about the “Top 10 Chrome Extensions for SEO”‘ but when you think about it, what kinds of “customers” would those links bring? Entitled demanding and rude non-customers who pollute our sales funnel and waste our time. Eh…not really worth it.

Today a person running a COMMERCIAL SEO company told me “One of my employees loves the hubfinder and is now distraught that it’s not free anymore. What would it take to get access to that tool?” And I responded with “if they are distraught over $300 then frankly they are quite pathetic, IMHO.” He wants to CHARGE his clients, PAYS his employees, and then wants my time FOR FREE to ask how he could get FREE access to the fruits of our labor.

Distraught? Really? I couldn’t imagine having the audacity to send that message.

And the truth is…that is 99%+ of the SEO market…everything should be free except whatever they sell. But we have to PAY $1,000+ a month for a web host, PAY for our vBulletin license, PAY for our SupportSuite license, PAY to license data from other sources, PAY to create tools to collect data, PAY to create new tools, PAY to maintain tools, PAY to advertise, PAY for a design + redesign, PAY for additional servers working creatively on future projects, PAY for the risks associated with being a well known public SEO, PAY to fly out to speak at SEO conferences & share information, PAY for upgrades to the site, SPEND lots of time on creating content for the blog, PAY PAY PAY etc etc etc

We have subscriptions with services like Compete.com and WordTracker because to us they are worth it.Which is why we buy AdWords ads, certain links, access to other sites and services, desktop software like AdvancedWebranking, etc. I have easily spent $100,000’s on consulting, tools, and info-products. Was every purchase profitable? No. But in aggregate, there was plenty of profit to be had.

The people who are selling stuff but who are afraid to spend any money themselves often sell trash. They are not convinced in the value of what they sell (often because it is lacking). Or as Seth puts it…

Money is more than a transfer of value. It’s a statement of belief. An ad agency that won’t buy ads, a consultant who won’t buy consulting, and a waiter who doesn’t tip big—it’s a sign, and not a good one.

You don’t create a real business by being the free infrastructure for someone else’s business while giving it away AND providing 1:1 support. That is why open source works so well…give away the software, but if they want 1:1 support from the source they pay for it. $$$

Yes we could use more links, but that is not a weakness in our business right now…we have something like a quarter million people using our stuff. If anything, I would love to donate some of this site’s links to a few of our affiliate websites.

Imagine having a quarter million+ non-customers. If you are at that scale your problem is not finding a way to get more people at the top of the funnel. At that scale the issue more becomes filtering out the bottom portion of the market without offending the people who might potentially become customers. Assume 5% of the 250,000 people are entitled ___holes. Assume another 5% of them are great people who just happened to have a minor issues in the conversion process (forgot their username, picked the wrong username, registered under the wrong email address, didn’t get the welcome email, etc.). Could you imagine handling 25,000+ personal emails a year? Add in paying customers & media inqueries and now your up above 30,000. And that doesn’t even include making close to 1,000 posts a month in our member forums and reading the nearly 100,000 posts that have been made there! I love the work I do (and love helping people), but I think this really expresses the sentiment nicely.

I had to add the following to our support feedback section to help make the pollution from non-customers more manageable

Free SEO Tool Issues?
A Polite Warning for Non-customers

We run the best SEO website with the deepest and richest customer engagement. But our resources are finite and our time is valuable.

We Give Away Lots of Value, But Our Company is Small

At the same time we have given away some of our free SEO tools to over 100,000 webmasters. We can not provide 1 to 1 support to an audience that large while still providing the amazing customer experience that our paying customers have grown to appreciate and expect. If you are not a PAYING member then we expect you to read the installation and usage documentation before filing a ticket.

Did You Read The Usage Instructions?

Please note that if you are not a PAYING customer AND your issue is with our free SEO tools then we will NOT respond to ANY requests where you have not read the installation and operating instructions from the associated download pages.
SEO Tool Usage + Configuration Instructions

For your convenience here are links to the official resource centers for SEO 4 Firefox, Rank Checker, the SEO Toolber, & the download page. (The download page requires you set up a free account and login to it).

Need to Uninstall a Tool?

If you would like to uninstall something here are 2 ways to do that.

In his book Here Comes Everybody, Clay Shirky stated something along the lines of popularity being an imbalance between supply and demand of attention. Which is precisely why filtering is so important. If you don’t filter out the laggards and freetards you are only holding back your own potential while giving your paying customers an inferior service to what they deserve.

Eventually it gets to where filtering through that noise becomes nearly a full time job. (Lucky for me I work twice as long as just about anyone I know), but anything that makes the sign up process more complex creates more noise (which potentially eats your time + harms your brand while creating 0 income). Plus the above quoted piece from our support section might turn off some potential clients…but it is always a game of filtering…help as many people as you can and hopefully try not to offend many people.

What we are scarce on is time. And that is also what many people new to SEO are scarce on. And what people who are willing to pay for correct information with rich context are short on.

Working through all the hype and misinformation and scammy offers can be a bit overwhelming. To a person who is new to SEO, it is already confusing enough to decide…

• which tools offer real value
• if they should use any software
• if they should use free or paid tools
• what combination of software to use

And if we duplicate our tools for many different browsers then that ends up increasing the maintenance cost WHILE harming conversion rates (because people don’t know what they should chose, or why you have so many tools doing the same thing).

The capital and efforts spent creating (and maintaining) a second or third set of duplicative free SEO tools would be better spent creating more paid tool & content for our paying subscribers.

Related posts:

1. Xenu’s Link Sleuth – More Than Just A Broken Links Finder
2. Links Not Always the Best Indicator of Relevance
3. Strategies to Spend Money & Earn ROI from Link Campaigns (without buying links)

There are literally a bazillion SEO tools on the internet (literally!), this post discusses just one such tool; Xenu’s Link Sleuth. Many people in the SEO industry are already aware of this tool but many people I’ve spoken to only treat the tool as a broken link finder. It’s so much more than that.

This post is aimed at those who haven’t heard of it before and those who do use it regularly – there are lots of nifty features that solve all kinds of SEO-problems and hopefully beginners and advanced alike will learn something from this post.

What is Xenu?

Xenu’s Link Sleuth is a FREE download (everyone loves free) that runs on all versions of Windows (but not quite on Macs unfortunately). It’s a lightweight download and I’ve never had issues with it crashing or hanging. In a nutshell it’s a site crawler and once you point it at a URL it will crawl around the site and spit out a report at the end. It’s main focus and branding is all about finding broken links on your site (so where you link internally to a 404 error) but I’ve found that I use it to solve a whole host of different SEO-related issues which I will explain below.

Problem – How do I find broken links on my site?

This is the most basic use of Xenu in my opinion, but also the most common use. Simply point the program at the homepage of your site, check ’skip external’ to avoid it crawling the entire web, and set it going!

Click here to view a sample report provided by Xenu for the Distilled site (note that this is a sample report only, not run across the whole site).

You can see that there is a handy section which reports any broken links that it finds, though in this case I’ve chosen a rather poor example since there are no broken links on the homepage of Distilled

Problem – How do I get a crawl of my site into microsoft Excel?

The answer to this one, as you may have guessed is also Xenu! Simply choose the following menu option once the report is run:

Click here for a google docs of a sample report from the Distilled site. As you can see you get some really useful data such as:

• The status code of all pages crawled
• The type of page crawled
• The title tag of each page crawled

Problem – How do I check the length of my title tags across my whole site?

Looking at the above data sheet – simply filter for html pages and then check the length of the column titled “Title” – this will give you the length of the title tag. Filter for any above 65 and bingo – there’s your to-do list!

Problem – How do I analyse my site’s information architecture?

Yep, you guessed it – Xenu will do this too. This one requires a little more explaining however. Firstly, you see that in the spreadsheet above there is a column for “level” – what this column tells you is the number of links away from the initial link that you entered the crawled page is. So in the example sheet all the pages have a level of 1 since I restricted the crawl to just those pages 1 link away from the homepage.

This is really useful information as it tells you how many clicks it takes to get to a given page on your site from the homepage. Useful information! Especially in a large site where you have multiple levels of information architecture and several different types of navigation. Below is a quick screenshot of a report run 3 levels deep on the site. I’ve pivot-tabled the data (zomg – excel ftw) and selected the following options:

Of course, the beauty of pivot tables is that I can double click each of those rows and see which pages are contained within each level. This is of course, a pretty basic application of the data. But you see that once you start getting more data you can do more powerful things.

The second application of the very same data is the useful links in/links out column which looks like this:

There are other ways of getting this data for your site, Linkscape does it for example, but the good thing about Xenu is that you get the data structured in Excel and you have all the other page metrics alongside it. There’s plenty more you can do with this but at a very crude level you can use it to identify pages with more than 100 links on the page across your site!

Taking this data to the next level – here’s a glimpse at what’s possible, an analysis of type of page vs number of internal links shows you that for this site (not the distilled site) the money pages are getting very few internal links compared to top level pages and something is broken in the information architecture:

Problem – How do I find any 302 redirects on my site?

Xenu to the rescue! In order to catch redirects on your site you need to modify one of the settings on the crawl preferences to “treat redirects as errors”:

Then, when you run the report and export to excel redirects will no longer get the status code 200 but will get the true status code, be it 301 or 302! Perfect.

Problem – How do I check the indexing of a test version of my site?

Xenu of course! If your test version lies at a public URL such as testsite.distilled.co.uk then you can just point Xenu at that URL. However, if that’s not an option then you can even run Xenu off a local HTML file which is pretty nifty:

Problem – How do I generate an XML sitemap for my site?

Although there are many many ways of generating an XML sitemap for your site, Xenu does this in a quite nice (if not particularly customisable) way. This is perfect for small site owners with limited technical knowledge I think:

Problem – How do I find images missing alt text?

If only Xenu would do such a thing…. Wait, it does! Simply filter your excel download to image files, then the “Title” column is the alt text of the image:

Well that’s just a few of the many many applications of the Xenu tool – hopefully it’s inspired you to go out and give it a try – I know I use it a lot for all kinds of things. I mean, once you get your data into Excel the world is literally your oyster. Mmmmmm data oysters.

But wait! That’s not all – I reached out to Rich Baxter as I know he’s a very knowledgeable and smart SEO and he uses Xenu a lot. I asked him if he had any killer tips and here’s his killer tip. Thanks a lot Rich for getting me this at short notice:

Crawling web directories, looking for errors (By Rich Baxter)

Xenu’s not just a great tool to look inside your own site, it’s also pretty powerful for crawling external resources like directories, particularly if you’re looking for a domain to buy.

Try crawling dmoz.org, being sure to restrict Xenu’s access to “editors.dmoz.org”, but allow the crawler to “check external links”.

Quite quickly you’ll start finding “not found” URL errors from directory entries that might have been forgotten, on domains that may not yet have expired. Just sort by “status” in the crawl results table in Xenu. Here’s one I found earlier. I’m pretty sure that with the right offer via SEDO, the owner of fridgemagnet.org.uk (with its 634 sub domain links) might be interested in selling before the domain expires.

I’ve always found the “Copy URL”, Google cache and Wayback Machine links invaluable on a right mouse click on the results you’re interested in:

As a side note: If you are crawling external resources, try to be a good citizen and crawl slowly. Set your maximum threads to a very low level, so as not to get your IP banned by your target host.

Thanks Rich! Great tips. Let’s get link sleuthing! If anyone has any other creative/useful uses for Xenu please share them in the comments.

In other news, there’s still time to get your FREE SES Chicago Pass by purchasing a year of PRO! (If Chicago’s not your thing, SES will let you exchange the pass for any SES Event in 2010). SES just raised their prices to $1995 for a pass, so $799 for an entire year of PRO and a full-access SES Pass is an awesome deal

You’ve also still got 7 days left to get super-low release pricing on the brand-new SEOmoz Advanced SEO Training Series: Tips, Tricks & Tactics six-disk DVD set! Until December 6th, you can get this killer new series for 20% off and Free Shipping anywhere in the world. Hundreds of people have already ordered and supplies are limited, so take advantage while the price is low, and this limited-edition set is still available.

Do you like this post? Yes No

Related posts:

1. Advanced Link Analysis Charts
2. Strategies to Spend Money & Earn ROI from Link Campaigns (without buying links)
3. Google Link: Command – Busting the Myths

And here’s one more tool that retrieves search results from more sites: Kikin is the FireFox addon that “enriches and personalizes your browsing and search experience by adding results from your preferred social media websites (Facebook, Twitter, YouTube, eBay, etc).

Install:

Just click “Download”, allow FireFox to show you the pop-up from this site and install the tool. Re-start the browser and you are done.

Customize:

Go to TOOLS FireFox menus and select “My kikin“. You’ll be taken to the the kikin website where you will be able to customize your settings. To always see results from sources, select the green button. To only see results when they are relevant to your search, select the yellow button. To never see results from a source, select the red button.

Note 1: you will need to allow access to Kikin Facebook application to get the results from Facebook.

Note 2: you will also be able to search your Twitter stream if you connect the tool to your Twitter account.

Use:

Just search for anything that you are interested in and notice a cute box above the Google search results. Click through the tabs to see relevant results from the sources you set up in the options:

Warning:

The tool won’t work for me when I am using Google toolbar to search. I am not sure why. So if this happens to you, try deleting extra URL parameters in the Google search string appended by the toolbar. Hopefully, this bug will be fixed or clarified.

The tool was reviewed under SEJ policy.

Check out the SEO Tools guide at Search Engine Journal.

Enrich Your Google Search Results Page with Kikin

Related posts:

1. Google Testing a Revamp of the Search Results Page
2. 3 Tools That Make Posting Links to Wordpress Easy and Fun
3. Google: a new signal in search ranking algorithms!