A common standard for the Robots.txt protocol

Microsoft, Yahoo and Google have developed a detailed documentation about how they implement the Robots Exclusion Protocol (REP).

The search engines’ robots go wherever they want as long as they find links to follow. This means that they may index everything they find on your site or server unless you tell them explicitly not to.

There are several reasons for doing so. You may have pages with duplicate content, in which case it makes sense to tell the search engines what pages to index and what pages not to include in the search engine database.

There are two ways of telling the search engines where to (or not to) go:

(1) You can add a meta tag to each and every page explaining what to do, or you may (2) add a text file called robots.txt to the top directory of your site.

Our Search Engine Marketing 101 tutorial explains how to use the meta tags.

See our article on the robots.txt file to learn how to set up such a file.

The following table is adapted from the Live Search Blog, and includes the new standards. The rules described in the guides referred to above still work.
Click here to read the rest of this article!

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • blogmarks
  • Blue Dot
  • Bumpzee
  • Furl
  • Ma.gnolia
  • MisterWong
  • Propeller
  • Reddit
  • Simpy
  • StumbleUpon
  • TwitThis
  • Wikio
  • YahooMyWeb
  • BlinkList
  • NewsVine
Posted on Tuesday 15 July 2008
Filed under: All (summaries) and Search engine marketing and The search engine industry | Permalink

Scour, social search with a twist

scour logoScour is a brand new meta social search engine that encourages voting and commentary on each of its query results drawn from Google, Yahoo! And Live Search. And they will pay you to use their site.

You score points for searching, voting and commenting and if you invite your friends, Scour will add 20% of your friends’ points to your balance.

From my point of view, the money is the least interesting part. For one thing, in Scour’s own scenario, an average of 4.5 searches per day translates into 25 US dollars a year. So unless you invite lots of friends, you won’t get rich using Scour.

I’m more interested to see if the social search concept of Scour is an improvement on standard web search. If it is, I’ll return even if they don’t pay me. So how does it work and is it any good?

Click here to read the rest of this article!

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • blogmarks
  • Blue Dot
  • Bumpzee
  • Furl
  • Ma.gnolia
  • MisterWong
  • Propeller
  • Reddit
  • Simpy
  • StumbleUpon
  • TwitThis
  • Wikio
  • YahooMyWeb
  • BlinkList
  • NewsVine
Posted on Monday 14 July 2008
Filed under: All (summaries) and Online search tools and services and Social media | Permalink

Pandia Weekend Wrap-up July 13

Recent search engine intelligence gathered for our readers:Crayons

Click here to read the rest of this article!

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • blogmarks
  • Blue Dot
  • Bumpzee
  • Furl
  • Ma.gnolia
  • MisterWong
  • Propeller
  • Reddit
  • Simpy
  • StumbleUpon
  • TwitThis
  • Wikio
  • YahooMyWeb
  • BlinkList
  • NewsVine
Posted on Sunday 13 July 2008
Filed under: All (summaries) and Weekend | Permalink

No means no, Yahoo! says to Microsoft

The never ending Microsoft courtship has led to another loud “No!” from the prospective bride Yahoo!

Microsoft officially gave up its quest for Yahoo! weeks ago. Disgruntled Yahoo! stock owners, seeing their potential wind fall disappear, then made heroic efforts to force CEO Yang & Co to agree to a buyout.

The Icahn rebellion

Microsoft responded by saying to stock owner and rebel Carl Icahn that they might consider a new offer if he could get rid of the present Yahoo! board and get a more sensible one. (We are quoting freely here. They did not actually say “a more sensible one”, although that is clearly what was implied).

Icahn wants to replace Yahoo’s board with nine of his own choosing, himself becoming the director of the board at Yahoo shareholder meeting next month.

The Yahoo! counterattack

Yahoo board chairman Roy Bostock calls this and “an odd and opportunistic alliance of Microsoft and Carl Icahn has anything but the interests of Yahoo’s stockholders in mind,”

“While this type of erratic and unpredictable behavior is consistent with what we have come to expect from Microsoft, we will not be bludgeoned into a transaction that is not in the best interests of our stockholders.”

A Microsoft/Yahoo merger makes sense, doesn’t it?

Some may find Yahoo’s vehement refusal to consider a marriage between Microsoft and Yahoo! odd. After all, it seems like a match made in heaven. Here you have the financial clout of Microsoft coupled with the relatively popular search tool of Yahoo and Yahoo’s content rich portal features. And Yahoo! definitely needs the capital.

The fate of the board

Cynics may say that this is a board and a directorship desperately struggling to keep their positions. Indeed, the board argues that:
Click here to read the rest of this article!

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • blogmarks
  • Blue Dot
  • Bumpzee
  • Furl
  • Ma.gnolia
  • MisterWong
  • Propeller
  • Reddit
  • Simpy
  • StumbleUpon
  • TwitThis
  • Wikio
  • YahooMyWeb
  • BlinkList
  • NewsVine
Posted on Sunday 13 July 2008
Filed under: All (summaries) and The search engine industry | Permalink

5 basic tactics for improving your search engine rankings

At sign on blackboardIf you have done nothing to improve the search engine rankings of your web pages, here’s some advice that gives results.

The web is full of sound (and unsound) advice on how you can improve your search engine rankings and increase your online visibility. There is a whole industry out there catering to companies that try to get more customers and more sales online.

But what if you can’t afford that? You don’t have the money and you don’t have the time to read books on search engine optimization, attend search engine marketing conferences of hire search marketers.

Actually, unless your site or blog is targeting a very competitive area, you can achieve a lot search engine wise by applying the following basic tactics.

1. Identify the right keyword phrases and put them in your headlines

Find out what search terms your target audience will use when looking for the kind of information, goods or services your site provides.

These phrases are never self evident. Do not take for granted that others will use the phrases you use yourself when searching for the same topic.

Google has a free service for identifying relevant search queries. You can also test the excellent Wordtracker service for free.

Include keywords in a headline that tells the search engines what the page is about: When you write articles or blog posts, try to include relevant keyword phrases in the headline of the post. The headline is by far the most important source of information for the search engines when they try to find out what the page is about.

Most blogging software will make sure that this headline also appears in the TITLE field of the page (the text that appears on top of the browser window), which is good.

Then make sure that the keyword phrase is included in the text itself, at least once near the top.

The web is not the place for poetic and creative headlines. Puns and jokes, entertaining as they are, do not help the search engines and they do not help the reader understand what the page is about. Web readers are very impatient and will leave your site if they don’t immediately “get it”.

Our search engine tutorial has more on selecting keywords.

2. Cover relevant sister sites to get inbound links

Identify the most important high quality sites covering your topic, and include them in a resource section on your site.

Do not put up a large link collection! Make it a valuable catalog over useful online sources of information, adding one or two informative sentences to each link.

Your readers will love you for this section. You have done some really hard research for them, and they may bookmark this page and come back.

The sites you link to may discover your page via their web analytics software, and if they like it they may link back.

You need links like that, as incoming links from high quality authoritative sites will give you a big boost in your search engine rankings over time.

3. Write reviews of relevant sites

Which is why you should write reviews about the sites you like the most. Good, honest, useful reviews that help your readers and impress the owners of the sites you are reviewing. Their willingness to link back to you increases tenfold if you give them good coverage.

Send them a mail as soon as your review is up. Do not beg them for a link back. Popular web sites get a large number of link requests every week, and will normally ignore that kind of requests. Just add one or two paragraphs in your mail giving a summary about what your site is about. They will often link to it if they like it.

4. Keep track of your traffic

You definitely need some kind of software that informs you about your traffic, and especially where your traffic is coming from. You need to know what pages succeed, and which pages don’t, and you need to know what kind of search phrases people use when they find you using Google, Yahoo!, Live or Ask.

Your web hotel will normally provide such services for free, installing the relevant software for you.

You may also make use of online services like Google Analytics. It is free.

Many low scale bloggers find that Feedburner gives them all the data they need. If you select the Pro version, which is free, Feedburner will help you publish your web feeds and give you simple statistics on web traffic. If you want to make a living on the web, you need more, though.

5. Provide useful content

None of these tactics help if you do not provide useful high quality content, however.

If you have a news site or a blog, you should publish regularly. For a blog at least once a week. Two to three times a week is good, and once every day is perfect.

It is better to publish fewer high quality posts than many of low quality, though. If you publish too much, readers may find it hard to identify the gems and get frustrated. You don’t want that!

Remember that for each page you add to your site, the chances of people finding your site through the search engines increase. As each page has a different headline or “key keyword phrase” you add new gateways to your site with each page or post.

These gateways may also be used by webmasters looking for relevant info. If they find your page valuable, you may gain a link.

Adding pages and increasing the number of inbound links takes time. It often takes months before you see results, and at least a year before your site becomes a search engine winner.

See Pandia’s Search Engine Marketing 101 for more tips and more detailed advice on how to improve your search engine rankings.

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • blogmarks
  • Blue Dot
  • Bumpzee
  • Furl
  • Ma.gnolia
  • MisterWong
  • Propeller
  • Reddit
  • Simpy
  • StumbleUpon
  • TwitThis
  • Wikio
  • YahooMyWeb
  • BlinkList
  • NewsVine
Posted on Tuesday 8 July 2008
Filed under: All (summaries) and Pandia top 5 and Search engine marketing | Permalink

Pandia Weekend Wrap-up July 6

It is summer in the Northern Hemisphere and the search engine industry catches its breath. No big news this week, but we did found some interesting articles all the same:

Click here to read the rest of this article!

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • blogmarks
  • Blue Dot
  • Bumpzee
  • Furl
  • Ma.gnolia
  • MisterWong
  • Propeller
  • Reddit
  • Simpy
  • StumbleUpon
  • TwitThis
  • Wikio
  • YahooMyWeb
  • BlinkList
  • NewsVine
Posted on Sunday 6 July 2008
Filed under: All (summaries) and Weekend | Permalink

Clip, save and search text and images with Evernote

Evernote logoEvernote is a cool tool has the potential to become my outsourced brain, a place to store anything I want to remember, tag it, annotate it and make it easy to find when I need it.

I have been testing Evernote for about a month now. This week it is available to anyone in open Beta.

Evernote allows you to capture information in any environment using whatever device you prefer, and makes this information accessible and searchable at any time, from anywhere. Here’s an intro to some of the features and my verdict:

Click here to read the rest of this article!

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • blogmarks
  • Blue Dot
  • Bumpzee
  • Furl
  • Ma.gnolia
  • MisterWong
  • Propeller
  • Reddit
  • Simpy
  • StumbleUpon
  • TwitThis
  • Wikio
  • YahooMyWeb
  • BlinkList
  • NewsVine
Posted on Friday 4 July 2008
Filed under: All (summaries) and Online search tools and services | Permalink

100 useful niche search engines

Laura Milligan has put up a useful list of niche search engines over at College@Home.

She points out that though the general Google site is often touted as the number one search engine online, college students sometimes need more specific tools to help them uncover quality information on the Web that they can use for class projects, research papers, and even job and apartment searches.

This list features a huge variety of search engines that can be useful to many of us, including tools that find photos, sound effects, summer internships, health and medical information, reference guides, and a lot more.

The links are sorted into the following categories:

Extracurricular
Quick Answer Guides
City Guides and Travel
Shopping Search Engines
Business
Academic and Reference
Social Media and People
Multisearch
TV, Video and Radio
Medical Students and Health Search
Law Students
Metasearch and Megasearch Engines
Photos, Images and Visual Search Engines
News Searches
Jobs and Real Estate

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • blogmarks
  • Blue Dot
  • Bumpzee
  • Furl
  • Ma.gnolia
  • MisterWong
  • Propeller
  • Reddit
  • Simpy
  • StumbleUpon
  • TwitThis
  • Wikio
  • YahooMyWeb
  • BlinkList
  • NewsVine
Posted on Wednesday 25 June 2008
Filed under: All (summaries) | Permalink