Recently I was browsing Django documentation deliberately looking for some ideas to cross-pollinate to Rails. F() and Q() objects looked like nice little hacks so I decided to implement them for Rails.

Q() Object

Even though it’s not nearly as powerful as his Star Trek namesake it’s still pretty interesting to look at (original docs available here). It tackles the problem of complex SQL conditions building. Usually most issues arise from the fact that simple hash based conditions don’t allow for negated or ORed conditions. There are plenty of other projects aimed to mitigate these restrictions among others (a while ago I assembled the collection of these): Squirrel, Ambition, dm-sugar-glider, ez_where, RQuery, Searchlogic, Sequel, probably most recent one is MetaWhere.

There are different tricks above libraries employ

1. instance_eval and/or method_missing letting you write

           # Squirrel
           Playlist.find(:all) do
             any do
               name == "Party Mix"
               total_length > 3600
             end
           end
           

   or

           # RQuery
           User.where do |user|
             (user.age > 20) | (user.age.in 16,18)
           end
           

2. core objects extensions (extending Hash and/or Symbol objects)

           # Sequel
           Artist.filter(:name.like('Y%') & ({:b=>1} | ~{:c=>3}))
           # SELECT * FROM artists WHERE name LIKE 'Y%' AND (b = 1 OR c != 3)
           

3. hooking the interpreter (accessing syntax tree of the condition specifying block)

           # Ambition
           >> SQL::User.select { |m| m.name == 'jon' && m.age == 21 }.to_s
           => "SELECT * FROM users WHERE users.name = 'jon' AND users.age = 21"
           

While there is nothing technically wrong with the approaches listed above, it’s interesting to see what can be done without resorting to any of these tricks. The Q() object fits there very well. It let’s you build ORed, ANDed, and negated conditions without dropping to writing SQL (drawing on simple and well known hash-based syntax). Below is the illustration of Q() based syntax and the corresponding SQL fragments generated by it

    ~Q(:user_id => nil) # => user_id IS NOT NULL

    Q(:user_id => nil) | Q(:id => nil) # => user_id IS NULL OR id IS NULL

    ~(Q(:user_id => nil) & Q(:id => nil)) # => user_id IS NOT NULL AND id IS NOT NULL
    

    !Q(:user_id => nil) # => user_id IS NOT NULL
    

F() Object

Original docs. It let’s you reference your database columns and do simple calculations on them from your conditions and update statements. The concept is pretty simple and can be illustrated with a few examples

    # Simple column reference
    User.where(:updated_at => F(:created_at))
    # instead of writing it in SQL like: User.where("updated_at = created_at")

    # Column reference with calculation
    User.where(:updated_at => F(:created_at) + 1)
    # instead of User.where("updated_at = created_at + 1")

    # Calculation with references also work
    User.where(:updated_at => F(:created_at) + F(:updated_at))
    # instead of User.where("updated_at = created_at + updated_at")

    # Use in update statement
    User.update_all(:id_copy => F(:id))
    # instead of User.update_all("id_copy = id")
    

      user = User.find(1)
      user.views = F(:views) + 1 #
      user.save
    

Installation

You can get the code from my eponymous gem like this

    > gem install dolzenko
    > irb
    ruby-head > require 'dolzenko/django_q_object'
    ruby-head > require 'dolzenko/django_f_object'
    

So there you have your command line prompt from Bash/Zsh/cmd.exe and there is so called Rails console which is basically IRB prompt with the Rails application loaded.

My tip suggests that you invoke some Rails commands, namely generator related commands and rake tasks, from the running Rails console. The reason to do so is clear and simple – it’s fast.

Real World Proof

Let’s do some math to prove the point. The last project I was involved with contains 171 models, 158 controllers and 572 migrations. Models and controllers were mostly generated together by single ./script/generate scaffold command. Migrations were generated with ./script/generate migration followed by rake db:migrate command. This accounts for about 1300 command line invocations.

Now sad statistics – running generator command from command line loads application environment which is not that cheap even on empty project

    rails3_app/ > /usr/bin/time -p rails generate migration just_testing
          invoke  active_record
          create    db/migrate/20100702124738_just_testing.rb
    real 7.54
    user 3.40
    sys 4.09
    

On the other hand running the same command from Rails console (i.e. with environment already loaded) is instantaneous

    rails3_app/ > rails console
    Loading development environment (Rails 3.0.0.beta4)
    MyIrb loaded
    ruby-head > time { migration 'test9' }
          invoke  active_record
          create    db/migrate/20100702125251_test9.rb
    Time elapsed: 0.07000 user, 0.08000 system (0.14645 wall clock seconds)
    

Getting/Using It

Helpers that bring generator and rake commands to IRB prompt are available as part of my IRB configuration in console_in_console.rb file. Assuming it’s saved at ~/console_in_console.rb

    rails3_app/ > rails c
    ruby-head > load '~/console_in_console.rb'
    ruby-head > include MyIrb::Rails::ConsoleInConsole
    # Have fun now generating and destroying migrations, models, controllers
    # much faster than before

    ruby-head > model 'user login:string password:string'
          invoke  active_record
          create    db/migrate/20100702132016_create_users.rb
          create    app/models/user.rb
          invoke    test_unit
          create      test/unit/user_test.rb
          create      test/fixtures/users.yml
    
    ruby-head > migration 'migration_name'
          invoke  active_record
          create    db/migrate/20100702130723_migration_name.rb

    ruby-head > destroy_migration 'migration_name'
          invoke  active_record
          remove    db/migrate/20100702130723_migration_name.rb

    ruby-head > rake 'db:migrate'
    ==  Test11: migrating =========================================================
    ==  Test11: migrated (0.0000s) ================================================
    

Age-Old Problem

Suppose you want to authenticate user from the login/password entered on the web from

    user = User.first(:conditions => "login = '#{ login }' AND password = '#{ password }'")
    

    User.find_by_login_and_password(login, password)
    # or
    User.first(:conditions => ["login = ? AND password = ?", login, password])
    # or
    User.first(:conditions => ["login = :login AND password = :password", { :login => login, :password => password }])
    

Many Solutions (Haskell Included)

Parameterized queries cover most of the cases but what if you have large query with multiple parameters? You can quickly lose track of all the question marks and order in which parameters appear. Most likely you will prefer the latter, hash-based notation.

Side note: This stuff is actually analyzed in depth by Jim Weirich in his The Building Blocks of Modularity presentation where he talks about the connascence in software.

Back to our topic what we really want is just to write SQL queries injecting properly quoted values where needed. In other words we would like to be able to customize the way string interpolation works. In cool languages like Haskell this concept is natively supported and called Quasiquotation. While not really on par with such rigorous achievements it can be simulated everywhere where eval is available. Here is for example the solution from the Google Caja team.

Hack For Better Future

There is eval in Ruby and there is also one peculiar feature about it. You can pass binding to it, and when the block is created it’s binding is captured and therefore can be used from anywhere (providing access to the lexical scope captured by the block). Let’s illustrate that

    ruby-head > def m
                  local = 42
                  proc {} # return empty block to get binding from it later
                end
    ruby-head > eval("local", m.binding) # binding gives access to the scope where block was created
    => 42 
    

Using that feature we can hide eval and implement our own safe interpolation routines like I did in safe_interpolate.rb. Use it like

    ruby-head > require 'dolzenko/safe_interpolate'
    ruby-head > include SafeInterpolate
    ruby-head > login = "' or 1=1"
    ruby-head > sql_interpolate { 'login = #{ login }' }
     => "login = ''' or 1=1'" # single quote got turned into double and the whole expression is quoted itself
    

    ruby-head > content = "<script>alert(1)</script>"
    ruby-head > html_interpolate { '<p>#{ content }</p>' }
    => "<p>&lt;script&gt;alert(1)&lt;/script&gt;</p>" # injection prevented!

    ruby-head > query = "&admin=1"
    ruby-head > uri_interpolate { 'http://example.com?q=#{ query }' }
    => "http://example.com?q=%26admin%3D1"
    

Further Reading

A type-based solution to the “strings problem”: a fitting end to XSS and SQL-injection holes? by Tom Moertel. This article got me interested in the subject.
Haskell features I’d like to see in other languages by Tim Carstens. Check the «Quasi-quoting» section for collection of Haskell quoting related links.

In the previous post I introduced Reflexive gem which lets you browse the classes and modules for running application with Web UI. But we’re really often just goofing off at the IRB prompt, aren’t we? For this case I have extracted some portions of Reflexive to help with code navigation right from the interactive prompt.

Let’s see how these may help us look under the hoods of Rails 3 framework. Disclaimer: this posts is intended as the demonstration of navigation techniques that can be used on any code base, not necessary Ruby on Rails, also if you’re looking for detailed explanation of the architecture of particular Rails components Maxim Chernyak compiled Rails 3 Reading Material list is the great place to start.

Side note: it may look like things demonstrated below could be done with debugger and more accurately so. My idea is that debugger basically follows push model so program state is pushed to you and you just follow along. On the other hand navigating manually is more like a pull model which is much more engaging and more effective when you try to understand and remember patterns in foreign code (well at least that’s how it is for me )

ActiveRecord find method

Suppose we want to find out what happens when we call ActiveRecord find method. That’s the easy one. First install method_extensions gem, add it to Gemfile and start console

    root@ubuntu:~/ > rvm use ruby-1.9.2-head 
    root@ubuntu:~/ > rvm gemset create rails3 && rvm gemset use rails3 # sandbox our Rails 3 experiments in rails3 Gemset (h/t to Mark Carey for this note)
    root@ubuntu:~/ > gem install rails --pre
    root@ubuntu:~/ > gem install method_extensions
    root@ubuntu:~/ > rails rails3_test_app
    root@ubuntu:~/ > cd rails3_test_app
    root@ubuntu:~/rails3_test_app/ > echo 'gem "method_extensions"' >> Gemfile
    root@ubuntu:~/rails3_test_app/ > bundle install
    root@ubuntu:~/rails3_test_app/ > rails c
    ruby-1.9.2-head >
    

Now ActiveRecord find is a class method so to retrieve corresponding Method object we’re using method method

    ruby-1.9.2-head > ActiveRecord::Base.method(:find)
    => #<Method: ActiveRecord::Base.find> 
    

    ruby-1.9.2-head > puts ActiveRecord::Base.method(:find).source
    ArgumentError: failed to find method definition around the lines:

      delegate :find, :first, :last, :all, :destroy, :destroy_all, :exists?, :delete, :delete_all, :update, :update_all, :to => :scoped
      delegate :find_each, :find_in_batches, :to => :scoped
    

Not very impressive, but we can see at least that find is delegated method, and it’s delegated to the result of scoped method. That’s how delegate works, but what if we had no idea about delegate method and how it works?

source_with_doc method will also return the comment preceding the method definition

    ruby-1.9.2-head > puts ActiveRecord::Base.method(:delegate).source_with_doc
    # Provides a delegate class method to easily expose contained objects' methods
    # as your own. Pass one or more methods (specified as symbols or strings)
    # ... snip
    def delegate(*methods)
      options = methods.pop
      unless options.is_a?(Hash) && to = options[:to]
        raise ArgumentError, "Delegation needs a target. Supply an options hash with a :to key as the last argument (e.g. delegate :hello, :to => :greeter)."
      end
    # ... snip
    end
    

Now that we know that find is delegated to scoped indeed, what does scoped return?

    ruby-1.9.2-head > puts ActiveRecord::Base.method(:scoped).source_with_doc
    # Returns an anonymous scope.
    # ... snip
    def scoped(options = {}, &block)
      if options.present?
        relation = scoped.apply_finder_options(options)
        block_given? ? relation.extending(Module.new(&block)) : relation
      else
        current_scoped_methods ? unscoped.merge(current_scoped_methods) : unscoped.clone
      end
    end
    

Here we need to stop for a moment and consider what this piece of code may potentially do. We can see that in presence of options hash function recursive call is made without options passed which takes us to the second branch which ultimately calls unscoped. Let’s see what it does.

    ruby-1.9.2-head > puts ActiveRecord::Base.method(:unscoped).source
    def unscoped
      @unscoped ||= Relation.new(self, arel_table)
      finder_needs_type_condition? ? @unscoped.where(type_condition) : @unscoped
    end
    

    ruby-1.9.2-head > ActiveRecord::Base.method(:unscoped).reflexive_url
    => "http://localhost:3000/reflexive/constants/ActiveRecord::Base/class_methods/unscoped" 
    

Rewinding a little back to the scoped definition we see that find is in the end called on an instance of ActiveRecord::Relation. Let’s confirm that using full_inspect method also added by method_extensions gem. It is just the plain inspect, source_location, and source_with_doc put together

    ruby-1.9.2-head > puts ActiveRecord::Relation.instance_method(:find).full_inspect
    #<UnboundMethod: ActiveRecord::Relation(ActiveRecord::FinderMethods)#find>
    ["~/rails3_test_app/vendor/rails/activerecord/lib/active_record/relation/finder_methods.rb", 90]
    # Find operates with four different retrieval approaches:
    # ... snip
    def find(*args, &block)
      return to_a.find(&block) if block_given?
      # ... snip
    end
    

ActionController render method

Let’s get into it

    ruby-1.9.2-head > puts ActionController::Base.instance_method(:render).full_inspect
    #<UnboundMethod: ActionController::Base(ActionController::Instrumentation)#render>
    ["~/rails3_test_app/vendor/rails/actionpack/lib/action_controller/metal/instrumentation.rb", 36]
    def render(*args)
      render_output = nil
      self.view_runtime = cleanup_view_runtime do
        Benchmark.ms { render_output = super }
      end
      render_output
    end
    

    ruby-1.9.2-head > require "method_extensions/method/super"
    ruby-1.9.2-head > puts ActionController::Base.instance_method(:render).super.source_with_doc
    # Check for double render errors and set the content_type after rendering.
    def render(*args) #:nodoc:
      raise ::AbstractController::DoubleRenderError if response_body
      super
      self.content_type ||= Mime[formats.first].to_s
      response_body
    end
    

    ruby-1.9.2-head > puts ActionController::Base.instance_method(:render).super.super.full_inspect
    # Normalize arguments, options and then delegates render_to_body and
    # sticks the result in self.response_body.
    def render(*args, &block)
      self.response_body = render_to_string(*args, &block)
    end
    

    ruby-1.9.2-head > puts ActionController::Base.instance_method(:render_to_string).source_with_doc
    # Raw rendering of a template to a string. Just convert the results of
    # render_to_body into a String.
    # :api: plugin
    def render_to_string(*args, &block)
      options = _normalize_args(*args, &block)
      _normalize_options(options)
      render_to_body(options)
    end
    ruby-1.9.2-head > puts ActionController::Base.instance_method(:render_to_body).source_with_doc
    def render_to_body(options)
      options[:template].sub!(/^\//, '') if options.key?(:template)
      super || " "
    end
    ruby-1.9.2-head > puts ActionController::Base.instance_method(:render_to_body).super.source_with_doc
    def render_to_body(options)
      _handle_render_options(options) || super
    end
    ruby-1.9.2-head > puts ActionController::Base.instance_method(:render_to_body).super.super.source_with_doc
    # Raw rendering of a template to a Rack-compatible body.
    # :api: plugin
    def render_to_body(options = {})
      _process_options(options)
      _render_template(options)
    end
    ruby-1.9.2-head > puts ActionController::Base.instance_method(:_render_template).source_with_doc
    # Find and renders a template based on the options given.
    # :api: private
    def _render_template(options) #:nodoc:
      view_context.render(options)
    end
    

    ruby-1.9.2-head > puts ActionController::Base.instance_method(:view_context).source_with_doc
    # An instance of a view class. The default view class is ActionView::Base
    #
    # The view class must have the following methods:
    # View.new[lookup_context, assigns, controller]
    #   Create a new ActionView instance for a controller
    # View#render[options]
    #   Returns String with the rendered template
    #
    # Override this method in a module to change the default behavior.
    def view_context
      view_context_class.new(lookup_context, view_assigns, self)
    end
    ruby-1.9.2-head > puts ActionController::Base.instance_method(:view_context_class).source_with_doc
    def view_context_class
      @view_context_class || self.class.view_context_class
    end
    

    ruby-1.9.2-head > puts ActionController::Base.method(:view_context_class).source_with_doc
    def view_context_class
      @view_context_class ||= begin
        controller = self
        Class.new(ActionView::Base) do
          if controller.respond_to?(:_helpers)
            include controller._helpers

            if controller.respond_to?(:_router)
              include controller._router.url_helpers
            end

            # TODO: Fix RJS to not require this
            self.helpers = controller._helpers
          end
        end
      end
    end
    

    ruby-1.9.2-head > puts ActionView::Base.instance_method(:render).source_with_doc
    # Returns the result of a render that's dictated by the options hash. The primary options are:
    #
    # * <tt>:partial</tt> - See ActionView::Partials.
    # * <tt>:update</tt> - Calls update_page with the block given.
    # * <tt>:file</tt> - Renders an explicit template file (this used to be the old default), add :locals to pass in those.
    # * <tt>:inline</tt> - Renders an inline template similar to how it's done in the controller.
    # * <tt>:text</tt> - Renders the text passed in out.
    def render(options = {}, locals = {}, &block)
      case options
      # ... snip
    end
    

While this won’t replace proper docs/debugger in any way it was nice to see how self-reflection capabilities of Ruby can be improved with just a few hacky helper methods. Hopefully the travel wasn’t too boring , comments/suggestions/patches are welcome indeed!

Added note: I’ve just started using this for real and it holds up pretty well so I decided to add some shortcuts to my .irbrc and method_extensions which will make the IRB print result of full_inspect verbatim by default. Check this method and comment ]]> http://dolzhenko.org/blog/2010/05/travel-to-the-core-rails-3-methods-without-leaving-irb-prompt/feed/ 8 http://dolzhenko.org/blog/2010/05/reflexive-live-class-and-source-code-browser/ http://dolzhenko.org/blog/2010/05/reflexive-live-class-and-source-code-browser/#comments Thu, 20 May 2010 16:45:33 +0000 Evgeniy Dolzhenko http://dolzhenko.org/blog/?p=150 Background

Recently I was trying to understand how the Arel (Relational Algebra for Ruby) internals work.

The project has strong OO design and uses some metaprogramming techniques. Main code base consists of ~ 80 files, with files having 30 LOCs on average.

I’m totally cool with that kind of modular/small classes design, but for new comer (like me) it could be a little daunting and I had to draw some diagrams on paper before finally getting it instead of getting lost in inheritance chains, compositional patterns and so on.

Solved In SmallTalk 20 Years Ago

Indeed as with many other problems it’s been solved in SmallTalk for many years. Having live classes reside in live virtual machine is one the SmallTalk features that makes rich reflection facilities possible and which is missing from Ruby.

So Can We Have A VM Please?

While there is no Ruby VM in a strict sense what’s that Rails development server process we have running if not a VM with classes kept in sync with their «on the disk»/plain-text representation? Well at least it’s pretty close, also the fact that at least half of the metaprogramming stuff happens at load time definitely helps for better reflections (think klass.included hook, attr_reader, delegate methods)

What IDE?

Really I would love to have something like this built-in into my favorite IDE, but the problem is that I’m not even sure what’s mine favorite IDE is, and there is even less agreement on the subject in the community.

What Ruby Offers? Or Quick Recap Of Ruby Reflection Techniques

• Kernel#method and Kernel#instance_method let’s you determine where the method defined on an object (instance of the object) comes from (an old trick taught by Dave Thomas). Less known is that with 1.9 you can get the file and line number of method definition with Method#source_location
• Module#ancestors because of Ruby unified object model let’s you determine class ancestors, modules included in class, and modules included by another module. This also works seamlessly for singleton classes (i.e. when you have something like C.extend(M) module M becomes an ancestor of C.singleton_class)
• Object#methods and Module#instance_methods when passed false as first parameter can be used to enumerate methods owned by class (as contrasted with returning owned and inherited methods by default)
• Ripper Ruby parser although not directly reflection related Ripper is part of a standard library and is based on parse.y (the grammar file which Ruby interpreter uses) which automatically makes it always up-to-date with any Ruby syntax changes.

Get Reflexive

With the set of aforementioned tools in hand I went and created sort of proof-of-concept good old web-app with virtually no UI and enjoyed the process a great deal. It browses classes and modules, and the corresponding source code files with some interactivity twists (and a little dose of pink color).

Visit the GitHub project page to learn more about what it does and how to get it for your applications.
You can also check out the demo Rails 3 app with Reflexive installed running on Heroku but be warned that Heroku has old Ruby 1.9.1 which has some issues with the reflection capabilities Reflexive uses.

In Conclusion

It seems to me that live reflection is the really powerful way to get precise code navigation/autocompletion/refactoring tools in Ruby IDEs. It also should be possible to integrate Reflexive in RDoc or YARD to get more interactive source code listings, but I scratched my own itch now and have to move on

• Use Enumerable#chunk to recreate Facebook “Show N Similar Posts” feature
• Use Enumerable#slice_before to easily parse mysqldump, Rails log, and INI files
• Tail Rails log files like a pro targeting only specific actions/clients

Enumerable#chunk

class UpdatesController
  def index
    # since we're not interested in the property we're chunking the array on (user here),
    # we get just the chunks themselves with +map(&:second)+
    @chunked_updates = Update.order("created_at DESC").chunk(&:user).map(&:second)
  end
end

<!-- views/updates/index.html.erb -->
<% @chunked_updates.each do |update_chunk| %>
  <%= render update_chunk.first %>
  <% if update_chunk.size > 1 %>
    Show <%= pluralize update_chunk.size - 1, "similar update" %>
    <!-- ... render additional updates here ... -->
  <% end %>
<% end %>

Enumerable#slice_before

Section 1 Header
Section 1 contents
...
Section 2 Header
Section 2 contents
...

• INI files. Gist which parses INI file into Hash
• mysqldump files. Gist which extracts particular tables from whole database dump
• Rails application log files. Gist which calculates cumulative running time per action

Structure aware log grep

Show only local requests: 
tail -f log/production.log | grep_rails_log_file.rb 127.0.0.1

Show only GET requests: 
tail -f log/production.log | grep_rails_log_file.rb GET

Show only requests to /updates path: 
tail -f log/production.log | grep_rails_log_file.rb /updates

Use it right now

IRB.CurrentContext.irb.instance_eval do
  def output_value
    printf @context.return_format, y(@context.last_value)
  end
end