Improved search engine query parser and with that more accurate search results, that’s what you can expect from last days updates.
I just had the privilege of spending a day programming instead of doing marketing. I used the time to improve the search engine query parser to find long keyphrases in a query and to remove redundant shorter phrases that are really just a sub phrase of one of the long phrases.
The problem with the old search engine query code was that it would sometimes omit common words even though they were essential to the search engine query. It could omit “best” in the query best hosting providers, and that changes the meaning of the query.
It also split search engine queries into far too many sub-queries.
Like in this search for Hotels in New York City, it’s split the old query parser into “hotels”, “in”, “new”, “York”, “City”, “Hotels in”, “New York”, “in New York”, and “New York City” which is a lot of redundant queries and may end up giving New York too much weight at the cost of Hotels
With the new search enging query parser in place the query for “Hotels in New York City” will omit “in”, “city”, “New York” and even “New York City” as they are sub phrases of the longer “in New York City”
Not that the new query parser is perfect, it still fails to search for “Location: New York City”, Hotels in Manhattan or an address in New York City.
The solution to this of course is the planned synonym logic that will make a search for “in New York City” also query alternatives like “in NYC”, “of New York City” “in Manhattan” etc.
It will be some time though before I have synonyms running as a fair amount of processing power is needed to determine synonyms form the main index. I currently have one Linode working on it and I expect to be able to test the algorithm in a month or two.
Update: I found an interesting article from SEO by the Sea describing a Google patent on Search Engine Query Statistics. The huge volume of search queries that Google has in it’s logs lets the draw a lot of useful conclusions on word relationships and user behavior. I would love to have that much data in my query log
Please leave you comment and tell me what you think about the new search engine query parser and please also leave a comment if you notice any interesting phenomenons with the new search query analyzer
There’s phone, website, marketing, legal advice and a lot more for free or almost free.
Don’t however fall into the trap of looking to get everything free. If you run a real business a good paid service will many times be the best alternative. It might give you better support, less time spent tinkering with it or better overall quality.
Some tools, like OpenOffice.org, though are pro quality even if free and there is no need searching for a paid laternative.
The feed I promised you last time is now ready and you can now get our search results directly on your own website, allowing you to make your own search engine or use the results as additional content on your pages.
You can test it by simply including this URL in you website as an iframe or directly form PHP or some other scripting language.
In this example q is the query “obama”, start is 2 meaning we skip the first result and start from result 2 (not page 2), num is 3 meaning we show only three results, namely the results number 2, 3 and 4
Yesterday I completed the promised server upgrades, upgrading the main server from a Linode 720 to a Linode 1080 and adding a new Linode 360 server to host the API (I’ll tell you more about that later) and for doing special data processing that is outside the normal indexing cycle.
So what does this mean for you?
The main thing you’ll notice is that the index will get bigger. I can’t tell you how much though as this is impossible to predict, but I really hope we will be able to reach 300,000 pages with this setup.
There’s two things that will allow the undex to grow:
1. With more RAM, 1080MB instead of 720MB the updating of the index database will be quicket letting the robot index more pages in a month.
2. As there is less VPS nodes sharing the same server there should be more disk and CPU cycles available to Us. This is not cut in stone though as it depends a lot on what usage profile the other nodes have and I don’t know that yet.
The index will grow, great. What about that other server then?
The other server, the Linode 360, will be used to host the new API/Feed that I will announce soon. I will make a feed of the search results available for free letting you make your own search engine, use it as additional content for your directory or anywhere that you want to give your visitors some relevant websites to visit. But more about this later.
The other mission for the new server will be to do calculations of data sets that will support the main indexing.
Mission one will be to make a related keywords database to allow us to find sites about “New York City” when someone writes “NYC” or “Win XP ” when someone writes “Windows XP”.
I know I promised you an index of 300,000 pages the last time we upgraded servers but things changed and I implemented a few new algorithms that improv quality but slows down the indexing process.
The main slowdown this year has been the addition of the search cache, a separate database arranged to make search queries ultra fast. You can now expect to get your results in under a second, sometimes in two, while before the cache searches was anywhere between 4 and 40 seconds as they had to dig through the main index.
Now lets just wait and see how much the index grows.
Simon Byholm
CEO and founder,
Secret Search Engine Labs
P.S. If you order a Linode through the links in this post Linod will give us $20 in free server time.
I have the pleasure to announce that late yesterday the final steps was taken to integrate the advertising feeds from ISEDN.org into the search results pages. (It’s the green text ads to the far right)
On December 26th (between eating leftovers from Christmas) I signed the partner agreement with Jayde Online Inc to join the Independent Search Engine and Directory Network (ISEDN.org) also known as the ExactSeek featured listing program.
The feed is now live (for an example see the search for webmaster) and you can get a featured listing for your website targeting a specific keyword. Your ad will show up on Secret Search Engine Labs and on over 375 other search engines in the ISEDN network. See the advertising page for details.
Joinig the ISEDN will allow us to increase revenue from the search result pages which in turn mean we will be able to get bigger and better servers to increase the index size and improve the search results.
The press release about this just went live on PRWeb and I’d like to say welcome to all new visitors, please try the search results and tell me what you think by leaving a comment below.
I found a new (actually old but I didn’t know about it before) search engine by some graduates from Southern Illinois University. Founded in 1999 it has had a recent facelift as looks nice providing meta search results combined from Google, Yahoo, Ask and Bing.
They include “Did you mena…” type suggestions and related searches and I like the clean and professional looking interface.
I forgot to blog about it but in case you didn’t notice we started the official beta testing period last week and we even had a press release distributed through PRWeb.
Thanks to the press release we had about 400 extra visitors on last Thursday and Friday, most spending more than the average number of click on site.
The number if visitors isn’t exactly earth shattering but nevertheless it was great to have you all here and I hope many of you will be back for more searches.
The Beta testing period is planned to run all year and end in a public launch in January 2011 with the target of having over a million relevant pages indexed by then.
I also want to take the opportunity to apologize if you have stumbled upon some less than relevant results. We had a bit of a Cuil experience when in anticipation of a stampade of visitors from the press release we added a new cache database to speed up searches.
Yes the searches are now about 30 times faster with most searches ready in less than half a second.
The downside is it will take a whole month before the cache is fully populated with data and in the meantime the search results are based only on a subset of the whole index. For some search terms this means junk and garbage from the bottom of the barrel will show up.
Please enjoy the new super quick search results and check out the ping tool in the webmaster tools section.
It’s amazing how much junk there is on the Internet. You let the spider loose and it just wanders so deep into a site that you never find it again.
I first made a system with levels to prevent the spider to go more than x link jumps (3 or 4) from the domain homepage of a site. This wasn’t enough though.
As I have a really small index I don’t want that many pages from every domain but as many different domains as possible, so I invented something I call CashRank which is a somewhat controversial way to rank pages.
Each page gets a dollar value indicating how much money has to be paid annually to keep that page online. This is domain registration fees, hosting fees, cost of IP address, advertising costs etz.
A page in only included in the index if it’s worth at least $1/year, this effectively limits auto generated content that has no value, because no one is paying for it’s upkeep.
The CashRank is also propagated through links, a page keeps $1 for it’s own
upkeep and then sends $1 through every link until it has used up all of it’s cash.
This means pages with many inbound links can have more pages in the index and only the first links on a page are counted.
Please give me your opinion on CashRank by leaving a comment below.
I just found out that Stinky Teddy is following me on Twitter.
Stinky Teddy is a real-time gossip search engine which gives your search term a buzz-o-meter score and if the buzz is high enough it will show you the hot news stories relating to your search term.
This search engine is developed by David Hardtke, a “Physicist turned search engine innovator” according to his Twitter profile, and was launched on the 15th of December after half a year of development.
Stinky Teddy is a meta search engine and uses data from Bing, Yahoo, VidoSurf, Twitter, Oneriot and Collecta which it then mashes up to become gossip results.
