For the moment, let’s leave aside the question of what makes a site “better” than Google (other a conceit to attract clicks like mine). And while we’re at it, the elevation of Bing or Twitter (or heck, any of these five sites) to the rarified stratum that Google occupies. (It is Fox News, after all.)

What got me blogging was the fact that he singled-out three Q&A services: Aardvark, Wolfram, and Siri. Tell us, Harry, what makes these sites better than Google?

[Aardvark] works well when you’d rather get quick advice from a few real knowledgeable people than scour Google results for relevant links on a question such as “Should I buy a mountain bike, a road bike, or a hybrid to ride around San Francisco?”

Wolfram|Alpha calls itself a “computational knowledge engine,” but I think of it as a 21st-century equivalent of a thick, fact-packed paperback almanac. It’s a vast repository of knowledge skewing towards the mathematical and scientific that you can explore by entering questions.

[Siri]’s a “virtual personal assistant” that uses voice recognition, your GPS location, and links to local information and services to respond to requests you speak into an iPhone 3GS.

The answer? Their ability to deliver precise bits of information — without having to “scour” Google results. That’s also where he sees value in Bing:

[With Bing Travel], You can enter dates and locations for plane tickets or hotel stays, then get a grid of results that you can further refine — to direct flights only, for instance, or to hotels with swimming pools.

Despite the attention-grabby headline, I love articles like these because it suggests conscious is changing regarding Web search. While traditional search engines aren’t going away anytime soon, consumers are beginning to see value in services which can get you to the content you’re really looking for — or, at the very least, can hook you up with experts who can help you get on the right track.

What’s particuarly interesting about Aardvark, Wolfram, and Siri is that they’re all providing access to information that has been created — and vetted — by humans. Vark is brilliant because it lets you ask questions to your social network en masse — without having to wait for anyone friend to pick up and tell you “I have no clue.” Siri makes sure you don’t have to search Yelp or Gayot or yes, Google, to be able to find if Boom Noodle closes at 8 or 10 pm on Sundays. Wolfram saves you the trouble of trying to cram the latest Information Please almanac in your pocket.

Yes, all of this information is great. And yes, I use all of three of these services every day. But truth be told, I find all of them a little unsatisfying: I’ve got so many more questions that these services can’t answer. Here are just a few that I thought of this morning while watching the Winter Olympics:

• What’s Sidney Crosby’s number?
• What country did the Winter Olympic sport of skeleton originate in?
• Who is the fastest female luger? (And where can I meet her?)

Like I said, I’m an enthusiastic user of Q&A services. (And yes, I’m building one of my own.) But I want these services to be transformative: to do things that I could never do myself (even if I had the time and energy). Don’t just improve access to data sources that I could likely manipulate myself. Give me access to knowledge that I could never have.

I know, I’m impatient. And lazy: I could probably spend time pouring over search results to find my answer. Or go to Wikipedia. But I want what Harry wants: content at my fingertips.  I may be more greedy than Harry, though:  I want all of it.

I had the good fortune on Thursday to be a part of a panel on semantic search at the Web 3.0 Conference. The panel was organized Mark Johnson(Bing/Powerset) and featured the likes of Connie Kenneally (TextWise), Will Hunsinger (Evri), Tim Musgrove (TextDigger), and yours truly (LCC, Swingly, Extractiv, etc.).

Mark put on an absolutely great panel. In addition to being one of the most knowledgeable people in our industry, he’s a natural-born moderator and a talented discussion leader. He’s got great journalistic chops too: definitely not one to shy away from asking the tough questions.

Since I wasn’t able to capture video of the panel, I thought I’d try to recreate my side of the discussion. Here are some of the questions that Mark asked — and the gist of the answers I gave. (Or would have given.)

More after the jump…

Mark Johnson: So, semantic search. A few years ago, this panel was made up of companies like Powerset and Hakia — companies with the stated goal of taking market share from Google/Yahoo/Bing. Now, it’s hard to find anyone who would even claim that they’re doing “search” anymore. Is search even the right word anymore? Would anyone consider what they’re doing to be search?

I think we’re seeing the diversification of semantic search. What did “semantic search” mean a few years ago? Beating Google/Yahoo!/Bing at their own game, using some as-of-yet untapped “semantic” technology. But heck, while we knew what the app looked like — pan-galatic web search — but we had no idea which semantic tech would actually make a difference. (Or what “semantic” meant, for that matter.)

Startups are now exploring how semantic search can be used to improve other kinds of apps, ones that are much more micro-scale than traditional search. That’s not to say that the current generation of semantic search startups have less ambition than the Powersets and Hakias of past years. We’re just as hungry — probably more so.

However, it does have a lot to do with the fact that traditional search (or retrieval) tech works just so darn well most of the time. If you’re interested in figuring out the name of the song that’s going through your head (as Google’s Johanna Wright was doing at Web 3.0), there’s nothing in particular about semantic search that’s going to help match the lyrics you know to a page with the rest of the song on it. And furthermore, while traditional search is by no means perfect, it’s generally at least mediocre all the time. Need to know how big labrador retrievers get? While a question-answering engine (like Swingly or WolframAlpha) might be able to interpret your question using completely snazzy semantic technology, it doesn’t matter how sophisticated their approach is if they don’t get you the right answer. In most cases, people will settle for mediocre and reliable over totally sexy but occasionally flaky.

So, it’s incumbent on us semantically-oriented startups to find the right set of use cases. (I hesitate to call them “markets” as of yet.) Ones where the sexiness is totally worth any potential flakiness. One where you can do things that you weren’t ever able to do before. Want to interact with lots and lots of structured data using natural language? There’s a semantic app for that: WolframAlpha. Need to find people who talk about the same things that you do on Twitter? There’s gonna be a semantic app for that.

Is semantic search dead? No, not in the least. However, we’ve realized that it’s time for us to show what we (in particular) do best — and that may not be pan-galatic gargleblasting search (in the way we know it now).

MJ: What scares you most?

Me, I’m most scared of Mechanical Turk. Yeah, that’s right: I’m afraid of people.

Here’s why. Any of us who invest in semantic technologies have a deep, unshakeable belief that we can build machines which can get meaning from text faster and better than any human ever could. And we’ve made a heckuva lot of progress these days: we’re beginning to talk about machines being able to “read” texts, take AP exams, translate a text in any language into any other language, etc. And that’s largely without the contributions of the Semantic Web community. Without using linked data. Without taking advantage of semantic interchange formats and standards, like RDF.

However, here’s the catch. Our algorithms aren’t perfect. In fact, they’re far from it. We still need humans to “train” our algorithms — that is, to give them cookies when they do well, and to hit them with a rolled up newspaper when they mess up the living room. And that costs money. And takes plenty of time for experimentation and analysis to get things right. And of course, that costs money, too.

Companies like mine continue to invest in R&D because we’re looking to minimize — or ultimately to get rid of — this kind of human input to our systems. R&D is expensive, sure — but it pales in comparison to the costs we’d have to incur if we had to go out any pay humans to perform the same task without any automation.

Turk is really disruptive because it makes it possible for humans to “fight back”. It’s cheap. It’s fast. It’s got the quality benefits that come from crowds checking (and re-checking) each other’s input. Does that mean that we’re going to see humans replace NLP systems? Well, no. But if it’s more cost-effective to let humans do an NLP task — like a name annotation task, say — that’s going to potentially jeopardize future investment in automation.

MJ: What tech do you use? Where does it come from? Do you use 3rd party software tools?

My two start-ups, Swingly and Extractiv, use technology that’s been developed by their parent company, Language Computer Corporation. We don’t use any 3rd party tools — largely due to licensing issues. Using GPL components can make it tricky if you ultimately want to license software yourself.

MJ: Who are your customers?

Swingly’s definitely designed for the web user. Our goal is to provide access to that 1% of knowledge that’s already out there — and really hard to get to through traditional search techniques. We’re also attracting some serious attention from folks with lots of domain-specific data: call centers, customer support centers, any service that has to maintain an FAQ, etc.

With Extractiv, we’re looking to become an “authoritative” provider of semantic content. Not just semantic annotations — mind you, although we definitely will do that, too. We want to establish that we are that definitive source for high-quality data (that no one else can get their hands on).

MJ: Is the popularity of “free-mium” causing companies to monopolize each others’ revenue streams?

To an extent. It’s also important to recognize that “free-mium” services (of which OpenCalais is probably the best example) have done a tremendous amount to set the market for semantic apps. It’s probably safe to say that without the success of OpenCalais’s more-or-less free service, we’d not being having as many mainstream discussions about the value of semantic apps.

I think free-mium models will begin to make a lot more sense in the not-so-distant future. Consumers’ appetites for content are only going to grow. And while we don’t see that many “power users” who need more capacity than they can get from a free service now, things are going to change. Whether we’re gonna see free-mium providers expand what they offer for free is the real question, however…

MJ: How do you measure how good you are? How do you communicate about measurements to your customers?

I’m a big fan of “open”, impartial, community-wide evaluations. I’ve participated in a bunch during my time at Language Computer: TREC (for question-answering), DUC (for summarization), ACE (for information extraction), and TAC (for textual inference). Yes, participating in these evals requires significant investment. But it’s tremendously satisfying to be able to point to a real benchmark, especially in space as competitive as ours. Frankly, I think there should be more opportunities for tech companies to show off what they can do.

We have to realize, however, that precision and recall aren’t enough. While benchmarks attract customers, they also can set up unrealistic expectations. If you don’t frame the discussion in terms of the real impact of your technology, it really doesn’t matter if your system can correctly answer 80% of questions users ask — they’ll only focus on the 20% where you left them high-and-dry.

MJ: Where do you see your business in 5 years?

I have two hopes for my companies. First, I’d like them to be contributors to the major search providers. There might be other viable ways forward, but aggregating search tools together into a single portal seems to be the way we’re all headed these days. Second, I’d count us as successful if we’re also actively shaping the discussion about how semantic apps should evolve. There are a lot of open questions out there. And I’d like us to have a crack at answering them.

Posted via web from andyhickl’s posterous

The Good

Here are the top 8 topics people are positive about (ranked in order of how strongly people are feeling).

1. People: So happy,  favorite,  fun,  great,  incredibly interesting,  interesting,  really nice,  smart,  super smart,  way more interesting

2. Day: good,  great,  lovely,  really fun

3. Dinner: Awesome,  Excellent,  absolutely delightful,  lovely

4. Topics: good,  positive

5. Discussions: Liking,  interesting,  lively

6. Talks: interesting,  really interesting,  visually engaging

7. Wifi: as good as the weather,  enjoying,  hella sweet

8.  EventVue, livestream, backchannel: Really enjoying, nice

(Ranking was determined on the inherent expected strength of the sentiment (e.g. hella sweet >> okay) and the number of tweets we found which expressed the same sentiment.  We’ve only listed the unique sentiments up above.)

The Bad

Far fewer negative sentiments.  In most cases, we couldn’t find something that three people were griping about.  Here are the two that reached that threshold:

1.  Kessler’s talk:  dangerous, doesn’t care, stupid, bad, provocative, pointless, useless

2.  language on stage: offended, unnecessary, pointless

The Day 1 “Winners”

After Day1, here are the most positively regarded Twitterers using the #defrag or #defragcon hashtags:  @bpm140, @stoweboyd, @sacca, @benkepes

The Overall Score

So far, tweets are running 88% positive overall for Defrag 2009 after day 1.

Want more info on sentiment tracking on Twitter using Positively?  Contact me at andy@languagecomputer.com.

Like you, we’ve heard a lot this summer about the challenges facing America:  the financial crisis, healthcare reform, and worst of all:  search overload.

Well, here at Swingly HQ, we’ve been doing our part.  We’ve been trying to find new ways to figure out what kinds of information are most relevant to a particular search topic.

While relevance modeling isn’t exactly new, it’s becoming an increasingly important problem for semantic search applications.   Information Extraction apps are rapidly increasing the amount of factual information that’s available from the Internet.  That’s good.  Unfortunately, instead of being buried under mountains of irrelevant information, we’re now being overwhelmed with gigabytes of factual information which may (or may not) be exactly what we’re looking for.  That’s bad.

So, what’s a new semantic search app to do?  Full details after the jump.

Let’s imagine you’re interested in learning more about Jesse “The Body” Ventura.  Well, if you’ve got access to a named entity recognizer (like the one we use with Swingly), you might be able to infer that he’s a:

• person
• Navy SEAL
• professional wrestler
• actor
• politician
• mayor
• governor
• talk show host

While these classes might not tell you anything you don’t already know, they (in theory) can provide semantic search apps with some of the intelligence needed to provide better, more informative results.

Since we know he’s a talk show host (among other things), we could have our app focus on finding information that’s relevant to any talk show host, such as:

• the station he’s on
• the format / length / medium of his show
• when his show started / ended
• how big his audience is
• …

But since he’s a former politician-turned-talk show host, we might want to go further and find other talk show host-related facts that may only be relevant for talk show hosts with this kind of background, such as:

• his political affiliation
• his endorsements
• the blogs that cover him
• …

These kinds of facts are most likely not relevant for other kinds of talk show hosts:  e.g. about your local sports-talk radio jock, the CarTalk guys, etc.

But we’re not out of the woods just yet.

While information extraction apps now able to capture lots and lots of different types of facts (including some of the ones I’ve listed above), they still require a human to tell them which classes of facts are relevant (e.g. political affiliation for a former politician-turned-talk show host) — and which ones aren’t (like home runs hit for a hockey player).

What’s worse?  Even though extractor ontologies are growing, most of the facts that we’ll need coverage for won’t be covered by an extractor.  Despite lots of efforts to reduce the cost of creating (and maintaining) extractors, ensuring adequate coverage across multiple domains requires serious investment in time and money.

So, what’s a semantic search app to do?  Start small — and work your way up.

At LCC, John Lehmann (and his team) recently developed a new algorithm which determines the most relevant predicates for each of individual names — or classes of names — mentioned in Swingly’s index.

For example, if you’re interested in information on NASCAR great Richard Petty, we think you might want be most interested in sentences which contain any of the following predicates.  (I’ve bolded some of the ones that I think are particularly interesting.)

{ win=46, appear=13, take=11, lead=10, drive=6, hold=6, race=6, finish=6, make=6, step=5, qualify=4, begin=4, compete=3, announce=3, mark=3, managed to qualify=3, fill=3, tie=2, visit=2, own=2, run=2, leave=2, return=2, come=2, suffer=2, remain=2, be part of=2, provide=2, drop=2, crash=2, follow=2, participate=2, spend=2, wear=2, miss=2, is currently=1, was back=1, serve=1, host=1, duplicate=1, form=1, log=1, tangle=1, record=1, start=1, discuss=1, running eleventh=1, put=1, give=1, use=1, bump=1, express=1, recognize=1, donate=1, unveil=1, stay=1, edge=1, slam=1, produce=1, send=1, remark=1, pull=1, become=1, snap=1, overcome=1, rebound=1, rivaled only=1, get=1, enter=1, raced alongside=1, collaborate=1, sign=1, failed to qualify=1, wave=1, match=1, release=1, was formerly=1, collect=1, trying to pass=1, allow=1, voice=1, set=1, retire=1, achieve=1, circle=1, established to honor=1, supply=1, develop=1, dominate=1, chose to run=1, begrudge=1, feel=1, tried to pass=1, outlast=1, sliding sideways=1, pit=1, claim=1, …}

Or, if you’re interested in wizards (yes, LCC provides access to a named entity type #wizard), you might be preferentially interested in answers which talk about:

{ appear=29, use=24, tell=22, take=21, have=18, find=17, give=15, make=14, become=13, ask=11, create=10, leave=8, visit=7, turn=7, reveal=7, send=7, place=6, choose=6, discover=6, hide=5, cast=5, put=5, hear=5, arrive=5, summon=5, defeat=5, recruit=4, meet=4, possess=4, provide=4, narrate=4, was once=4, destroy=4, confront=4, see=4, help=4, allow=4, order=4, offer=4, begin=4, air=3, go=3, agree=3, entrust=3, learn=3, lives backwards=3, inform=3, argue=3, hold=3, imprison=3, return=3, fall=3, kill=3, instruct=3, live=3, warn=3, appoint=3, die=3, conjure=3, begging to die=2, associate=2, search=2, drink=2, serve=2, convince=2, face=2, subdue=2, witness=2, transform=2, look=2, seek=2, prophesy=2, change=2, lead=2, watch=2, battle=2, advertise=2, catch=2, interpret=2, revive=2, throw=2, returned to punish=2, ally=2, locate=2, rush=2, mature=2, approach=2, involve=2, avoid=2, sacrifice=2, flee=2, announce=2, acquire=2, remind=2, save=2, travel=2, request=2, hire=2, …}

We then use LCC’s dependency parsers in order to expand each of these predicates into a set of semantically-typed triplets.  We then use submit these triplets to a modeling framework to learn the triplets which are most relevant for the individual entity, the entity type, or the user’s query:

Richard Petty:

• #driver – win – #race, #person – win – #raceEvent
• #driver – take – #ordinalNumber
• #driver – passed – #driver
• …

#Wizards:

• #wizard – cast – #spell
• #wizard – killed – {#person, #monster}
• #wizard – suffered – #quantity
• …

We then use the output of this modeling to pick out Q&As from Swingly’s index which are expected to be most relevant with respect to the query.  This means that for a generic query like Richard Petty, the top Q&As that Swingly returns goes from:

• Where can I buy Richard Petty die-cast replica cars?
• Which number did NASCAR retire in honor of Richard Petty?
• What type of car did Richard Petty race in 1964?

to

• What racing championship did Richard Petty win 7 times?
• What races did Richard Petty win more than once?
• How many races did Richard Petty win over his career?

Have we eliminated the pernicious problem of search overload?  Nah. (Well, not yet.)  But we expect techniques like these will be able to push high-quality, relevant content to the top of search results — even when users don’t give us enough information to figure out what they’re really looking for.

Author’s Note:  “Search Overload” is one of the taglines used in the Bing marketing campaign. I like Bing.  And anything that’s going to reduce the amount of crap I have to deal with from my search engine.