Scaling up personalized query results for next generation of search engines

At issue is how search engines handle complex or confusing queries. For example, if a user is searching for faculty members who do research on financial informatics, that user wants a list of relevant webpages from faculty, not the pages of graduate students mentioning faculty or news stories that use those terms. That's a complex search.

"Similarly, when searches are ambiguous with multiple possible interpretations, traditional search engines use impersonal techniques. For example, if a user searches for the term 'jaguar speed,' the user could be looking for information on the Jaguar supercomputer, the jungle cat or the car," says Dr. Kemafor Anyanwu, an assistant professor of computer science at NC State and senior author of a paper on the research. "At any given time, the same person may want information on any of those things, so profiling the user isn't necessarily very helpful."

Anyanwu's team has come up with a way to address the personalized search problem by looking at a user's "ambient query context," meaning they look at a user's most recent searches to help interpret the current search. Specifically, they look beyond the words used in a search to associated concepts to determine the context of a search. So, if a user's previous search contained the word "conservation" it would be associated with concepts likes "animals" or "wildlife" and even "zoos." Then, a subsequent search for "jaguar speed" would push results about the jungle cat higher up in the results -- and not the automobile or supercomputer. And the more recently a concept has been associated with a search, the more weight it is given when ranking results of a new search.

Search engines have also tried to identify patterns in user clicking behavior on search results to identify the most probable user intent for a search. However, such techniques are impersonal and are applied on a global basis. So, if the most frequent click pattern for a set of keywords is in a particular context, then that context becomes the context associated with queries for most or all users -- even if your recent search history indicates that your query context is about jungle cats.

"What we are doing is different," Anyanwu says. "We are identifying the context of search terms for individual users in real time and using that to determine a user's intention for a specific query at a specific time. This allows us to deal more effectively with more complex searches than traditional search engines. Such searches are becoming more prevalent as people now use the Web as a key knowledge base supporting different types of tasks."

While Anyanwu and her team developed a context-aware personalized search technique over a year ago, the challenge has been how to scale this approach up. "Because running an ambient context program for every user would take an enormous amount of computing resources, and that is not feasible," Anyanwu says.

However, Anyanwu's research team has now come up with a technique that includes new ways to represent data, new ways to index that data so that it can be accessed efficiently, and a new computing architecture for organizing those indexes. The new technique makes a significant difference.

"Our new indexing and search computing architecture allows us to support personalized search for about 2,900 concurrent users using an 8GB machine, whereas an earlier approach supported only 17 concurrent users. This makes the concept more practical, and moves us closer to the next generation of search engines," Anyanwu says.

The paper, "Personalizing Search: A Case for Scaling Concurrency in Multi-Tenant Semantic Web Search Systems," will be presented at the 2013 IEEE International Conference on Big Data being held Oct. 6-9 in Santa Clara, Calif. Lead author of the paper is Dr. Haizhou Fu, a former Ph.D. student at NC State. The paper was co-authored by Hyeongsik Kim, a Ph.D. student at NC State. The research was supported by the National Science Foundation.