Posts tagged Ultrasphinx

Presented faceted search using ultrasphinx

Nov 25th

Posted by Akhil Bansal in ROR

5 comments

The first ruby funday held on 22nd Nov 2008 in New Delhi, India. I presented on faceted search using ultrasphinx. I also gave a live demo there. Here are the slides:

Facet Searching using Ultrasphinx
View SlideShare presentation or Upload your own. (tags: sphinx ultrasphinx)

You can ask me if you have any doubt.

, , , ,

When Ultrasphinx is used with polymorphic associations…

Aug 7th

Posted by Akhil Bansal in ROR

No comments

Lets first consider simple has_many and belongs_to associations as:

class Article < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :article
end

Now if you wish to index the title of associated article with comment for searching, you just need to add ” is_indexed :fields => :body, :include => [{ :association_name => 'article', :field => 'title', :as=> 'article_title'}] ”

So, your comment model will look like:

class Comment < ActiveRecord::Base
  belongs_to :article

  is_indexed :fields => :body,
             :include => [{ :association_name => 'article', :field => 'title', :as=> 'article_title'}]
end

Setup ultrasphinx by issuing:

rake ultrasphinx:bootstrap

Now at rails console try:

>> article = Article.create(:title => "This is the first article's title", :body => "Here comes first article's body.")
=> #<article id: 1, title: "This is the first article's title", tagline: nil, type: nil, body: "Here comes first article's body.", created_at: "2008-08-07 07:47:53", updated_at: "2008-08-07 07:47:53">
>> article.comments
=> []
>> article.comments<<comment.new(:body=>"first comment on first article. which says yahooo !!")
=> [#<comment id: 1, body: "first comment on first article. which says yahooo !...", article_id: 1, created_at: "2008-08-07 07:48:47", updated_at: "2008-08-07 07:48:47">]
>> Comment.find :all
=> [#<comment id: 1, body: "first comment on first article. which says yahooo !...", article_id: 1, created_at: "2008-08-07 07:48:47", updated_at: "2008-08-07 07:48:47">]
>> search = Ultrasphinx::Search.new(:query => "first article's title", :class_names => ['Comment']).run.results
=> [#<comment id: 1, body: "first comment on first article. which says yahooo !...", article_id: 1, created_at: "2008-08-07 07:48:47", updated_at: "2008-08-07 07:48:47">]

Simple, we have article and comment models with has_many belongs_to associations. Comments are indexed with their associated article title. So it can return comments if query matches with article’s titles.

Now, consider a case when we wish to change comment model to make it polymorphic. In that case our models will be look like:

 class Article < ActiveRecord::Base
   has_many :comments, :as => :commentable, :dependent => :destroy
 end

class Comment < ActiveRecord::Base
  belongs_to :commentable, :polymorphic => true
end

Check at rails if associations are working fine:

>> article = Article.create(:title => "First article title", :body => "here comes first artile's title")
=> #<article id: 1, title: "First article title", tagline: nil, type: nil, body: "here comes first artile's title", created_at: "2008-08-07 08:12:31", updated_at: "2008-08-07 08:12:31">
>> comment = Comment.new(:body => "hello there, I am a comment")
=> #<comment id: nil, body: "hello there, I am a comment", commentable_id: nil, commentable_type: nil, created_at: nil, updated_at: nil>
>> comment.commentable = article
=> #<article id: 1, title: "First article title", tagline: nil, type: nil, body: "here comes first artile's title", created_at: "2008-08-07 08:12:31", updated_at: "2008-08-07 08:12:31">
>> comment.save
=> true
>> comment.reload
=> #<comment id: 1, body: "hello there, I am a comment", commentable_id: 1, commentable_type: "Article", created_at: "2008-08-07 08:14:29", updated_at: "2008-08-07 08:14:29">
>> article.reload
=> #<article id: 1, title: "First article title", tagline: nil, type: nil, body: "here comes first artile's title", created_at: "2008-08-07 08:12:31", updated_at: "2008-08-07 08:12:31">
>> article.comments
=> [#<comment id: 1, body: "hello there, I am a comment", commentable_id: 1, commentable_type: "Article", created_at: "2008-08-07 08:14:29", updated_at: "2008-08-07 08:14:29">]
>> comment = Comment.new(:body => "hi there, this is second comment")
=> #<comment id: nil, body: "hi there, this is second comment", commentable_id: nil, commentable_type: nil, created_at: nil, updated_at: nil>
>> article.comments<< comment
=> [#<comment id: 1, body: "hello there, I am a comment", commentable_id: 1, commentable_type: "Article", created_at: "2008-08-07 08:14:29", updated_at: "2008-08-07 08:14:29">, #<comment id: 2, body: "hi there, this is second comment", commentable_id: 1, commentable_type: "Article", created_at: "2008-08-07 08:15:34", updated_at: "2008-08-07 08:15:34">]
>> article.comments
=> [#<comment id: 1, body: "hello there, I am a comment", commentable_id: 1, commentable_type: "Article", created_at: "2008-08-07 08:14:29", updated_at: "2008-08-07 08:14:29">, #<comment id: 2, body: "hi there, this is second comment", commentable_id: 1, commentable_type: "Article", created_at: "2008-08-07 08:15:34", updated_at: "2008-08-07 08:15:34">]
>> comment.commentable
=> #<article id: 1, title: "First article title", tagline: nil, type: nil, body: "here comes first artile's title", created_at: "2008-08-07 08:12:31", updated_at: "2008-08-07 08:12:31">

Now, the point is to index article title with comments. Here we can get associated article using ‘commentable’ i.e. comment.commentable.

So, change ultrasphinx is_indexed code in comment model accordingly:

class Comment < ActiveRecord::Base
  belongs_to :commentable, :polymorphic => true

  is_indexed :fields => :body,
            :include => [{ :association_name => 'commentable', :field => 'title', :as=> 'article_title'}]

end

Note that we have changed associan_name to ‘commentable’.
Next, when we reconfigure ultrasphinx using ” rake ultrasphinx:bootstrap”(since we have changed db schema), it starts throwing errors:

** Invoke ultrasphinx:bootstrap (first_time)
** Invoke ultrasphinx:_environment (first_time)
** Invoke environment (first_time)
** Execute environment
rake aborted!
uninitialized constant Commentable
/Library/Ruby/Gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:278:in `load_missing_constant'
/Library/Ruby/Gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:467:in `const_missing'
/Library/Ruby/Gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:479:in `const_missing'
/Library/Ruby/Gems/1.8/gems/activesupport-2.1.0/lib/active_support/inflector.rb:283:in `constantize'
/Library/Ruby/Gems/1.8/gems/activesupport-2.1.0/lib/active_support/core_ext/string/inflections.rb:143:in `constantize'
/Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/associations.rb:19:in `get_association_model'
/Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/fields.rb:128:in `configure'
/Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/fields.rb:124:in `each'
/Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/fields.rb:124:in `configure'
/Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/fields.rb:101:in `each'
/Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/fields.rb:101:in `configure'
/Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/configure.rb:32:in `load_constants'
/Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/autoload.rb:8:in `after_initialize'
/Library/Ruby/Gems/1.8/gems/rails-2.1.0/lib/initializer.rb:148:in `process'
/Library/Ruby/Gems/1.8/gems/rails-2.1.0/lib/initializer.rb:93:in `send'
/Library/Ruby/Gems/1.8/gems/rails-2.1.0/lib/initializer.rb:93:in `run'
/Users/akhilbansal/work/sti/config/environment.rb:13
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:27:in `require'
/Library/Ruby/Gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:509:in `require'
/Library/Ruby/Gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:354:in `new_constants_in'
/Library/Ruby/Gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:509:in `require'
/Library/Ruby/Gems/1.8/gems/rails-2.1.0/lib/tasks/misc.rake:3
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:546:in `call'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:546:in `execute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:541:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:541:in `execute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:508:in `invoke_with_call_chain'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:501:in `synchronize'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:501:in `invoke_with_call_chain'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:518:in `invoke_prerequisites'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1183:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1183:in `send'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1183:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:515:in `invoke_prerequisites'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:507:in `invoke_with_call_chain'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:501:in `synchronize'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:501:in `invoke_with_call_chain'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:518:in `invoke_prerequisites'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1183:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1183:in `send'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1183:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:515:in `invoke_prerequisites'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:507:in `invoke_with_call_chain'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:501:in `synchronize'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:501:in `invoke_with_call_chain'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:494:in `invoke'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1931:in `invoke_task'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1909:in `top_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1909:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1909:in `top_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1948:in `standard_exception_handling'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1903:in `top_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1881:in `run'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1948:in `standard_exception_handling'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/lib/rake.rb:1878:in `run'
/Library/Ruby/Gems/1.8/gems/rake-0.8.1/bin/rake:31
/usr/bin/rake:19:in `load'
/usr/bin/rake:19

After spending some time on research, I was able to make it work by making some changes in comment model:

class Comment < ActiveRecord::Base
  belongs_to :commentable, :polymorphic => true

is_indexed :fields  => :body,
           :include => [{:class_name => 'Article',  :association_sql => 'left outer join articles on articles.id = comments.commentable_id and comments.commentable_type = "Article"', :field => 'title', :as=> 'article_title'}]
end

Lets check it on rails console:

>> search = Ultrasphinx::Search.new(:query => 'first article', :class_name => "Comment").run.results
=> [#<comment id: 1, body: "hello there, I am a comment", commentable_id: 1, commentable_type: "Article", created_at: "2008-08-07 08:14:29", updated_at: "2008-08-07 08:14:29">, #<comment id: 2, body: "hi there, this is second comment", commentable_id: 1, commentable_type: "Article", created_at: "2008-08-07 08:15:34", updated_at: "2008-08-07 08:15:34">]

In such cases we need to define class_name and association_sql in is_indexed statement instead of association_name.

Hope it helps…

, , , , , , , ,

When Ultrasphinx is used with STI…

Jul 31st

Posted by Akhil Bansal in Rails

No comments

Hi Guys, it has been a long time since I last posted. I had worked on several things since then, and have couple of posts pending/draft. One of those posts is related to Ultrasphinx, when it is used with STI models.

For those who are new to Ultrasphinx: Ultrasphinx is a rails plugin and client to the Sphinx(full text search engine) written by Evan Weaver. More about Ultrasphinx here.

Lets get into the situation. Consider a STI case where we are using ultrasphinx to index several fields:

class Post < ActiveRecord::Base

end

class Article < Post
  is_indexed :fields => [:title, :body]
end

class Story < Post
  is_indexed :fields => [:title, :body]
end

Now assume we have following data in posts table:
picture-4.png
and now in application root:

rake ultrasphinx:configure
rake ultrasphinx:index
rake ultrasphinx:daemon:start

By this point we have created sphinx configuration file, indexed all records from ultrasphinx models and started sphinx search daemon.

Now open script/console and create and fire a query to find some articles:

>> search = Ultrasphinx::Search.new(:query => "second article", :class_names => 'Article')
=> #<ultrasphinx::Search:0x2105bec @subtotals={}, @response={}, @facets={}, @options={"indexes"=>"main", "class_names"=>["Article"], "sort_by"=>nil, "parsed_query"=>"second article", "sort_mode"=>"relevance", "weights"=>{}, "filters"=>{}, "per_page"=>20, "query"=>"second article", "page"=>1, "facets"=>[], "location"=>{"units"=>"radians", "long_attribute_name"=>"lng", "lat_attribute_name"=>"lat"}}, @results=[]>
>> search.run
ActiveRecord::RecordNotFound: Couldn't find Article with ID=5
	from /Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:308:in `reify_results'
	from /Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:286:in `each'
	from /Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:286:in `reify_results'
	from /Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/search.rb:357:in `run'
	from /Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:352:in `perform_action_with_retries'
	from /Users/akhilbansal/work/sti/vendor/plugins/ultrasphinx/lib/ultrasphinx/search.rb:337:in `run'
	from (irb):3

Here when you run the query you get an exception, because it is trying to find an article with id 5, which actually a story type not article. So why it is trying to find such article?

Now have a look into config/ultrasphinx/development.conf file. Under section “source articles_main” you’ll get “SELECT (posts.id * 3 + 0) AS id, ” AS article_title, posts.body AS body, ‘Article’ AS class, 0 AS class_id, posts.title AS title FROM posts WHERE posts.id >= $start AND posts.id <= $end GROUP BY posts.id" Which is a SQL to get record to index.

If you check it carefully you'll findout that it should select all articles record to index but unfortunately it is selecting all records from table and considering them as articles. To fix it you need to modify this query and add "posts.type = 'Article'" in where condition. So the query should be "SELECT (posts.id * 3 + 0) AS id, '' AS article_title, posts.body AS body, 'Article' AS class, 0 AS class_id, posts.title AS title FROM posts WHERE (posts.type = 'Article') and posts.id >= $start AND posts.id <= $end GROUP BY posts.id"

But still there is a problem. If you do this manually you have to do it again and again whenever you issue "rake ultrasphinx:configure" because this configuration file will be overwritten.

Better option is to add conditions in models as is_indexed options like:

class Article < Post
  is_indexed :fields => [:title, :body], :conditions => "posts.type = ‘Article’"
end

and it worked fine.

Manik gave me an idea to write a patch for ultrasphinx to add such conditions automatically in case of STI. So may be another post related to ultrasphinx will be soon ;-)

Update: Here is the patch Git patch: Fix STI Issue. After applying this patch you need not to add conditions explicitly for such case. It will automatically check and add conditions for STI.

, , , , , , ,