As we make the push toward releasing the platform that I'm working on we've installed exception_notification in our Rails app. With the increased visibility of all the exceptions it became apparent quite quickly that there were numerous hits against the server from automated vulnerability scanners. These attempts were causing routing errors as they looked for paths like '/user/soapCaller.bs' – thankfully not targeting Rails applications. 

The arrival of this sort of scan was not particularly surprising to me as I've seen similar scans in the past. I've even actively dabbled in some security research by running a few honeypot projects, I digress.

Even though these scans usually go after large installations such as WordPress, Drupal, Joomla and phpMyAdmin, it isn't stupid to take it as a reminder to keep up to date on security vulnerabilities. In the case of Ruby on Rails the starting point would be the rubyonrails-security google group and the Ruby on Rails blog. 

Another great resource is Rails Inside. Rails Inside usually picks up any serious flaws and relays them to the community. In addition to this, they follow new releases of popular plugins/gems that may form part of your app. The site provides an important service because keeping up-to-date is a good way of reducing the risk of being caught by a vulnerability that has been dutifully patched by the maintainer.

The above is certainly not a complete list, so I'd like to hear if there are any other rails/ruby security resources that you find useful. 

I've spend most of the afternoon working on a complex regex in order to parse command line argument forms (for lack of a better term). If you've ever run the man command you'll know what I'm talking about. Take the tar command as an example:

tar  [ - ] A --catenate --concatenate | c --create | d --diff --compare
       | --delete | r --append | t --list | u --update | x --extract  --get  [
       options ] pathname [ pathname ... ]

If you're already familiar with regular expressions you'll know that  doing something like:

[\[].*[\]]

trying to match: 

[ - ] 

won't accomplish what you think. Instead of getting just the first set of braces you'll end up with the whole remainder of the string. This is because of a feature, we'll give it that title, called greedy matching. Greedy matching means that it takes the largest possible chunk that your regex will match, which in this case is the ']' on the end of pathname.

I was aware of what was going on, but not being a particular master of regular expressions, I wasn't sure how to get it to stop being greedy. As it turns out its quite easy.

• .*    - Greedy matching
  
• .+   - Greedy matching
  
• .*?  – Non-greedy matching
  
• .+? – Non-greedy matching
  

It could not be easier, once you know about it of course.

With JRuby and Warbler it’s possible to get Sinatra, or any WebApp based on Rack, running on a myriad of different Java application servers. There are of course gotchas when it comes to using Warbler with the many different app servers, so this is a definitive guide to everything you have to do to get a simple Sinatra app running on the various application servers.

Why Sinatra?

There are examples of how to get Rails running on Tomcat and Websphere floating around the web, but I find Rails overkill for small projects. With that in mind, it’s worth looking at how to get Sinatra running on java application servers. Besides the weight of rails, Sinatra is a nice, easy to learn framework.

Start by installing Sinatra and Warbler. You don’t have to be using JRuby to install Warbler, the install will download a gem of the jruby jars.

Install Sinatra and Warbler

Lets start by installing the required gems.

$ sudo gem install sinatra warbler haml

Haml isn’t strictly required, but the template I’m going to use has views generated in haml, so if you’re following the tutorial closely you’ll want to install it.

Create a project folder (and structure)

I usually keep a sinatra project template handy. So I’m going to clone my existing template off github. You can create a more minimal example than the one I’ll download, this will get the job done though.

$ git clone git://github.com/vertis/sinatra-example.git deploy_test

TODO: Details about the project

Check that our page is displayed

$ rackup

Go to http://localhost:9292/ and you should see our default page.

Generate the warble config

$ mkdir config && warble config

Lets look at the config file that was generated:

If you tried packaging and installing this now, it would fail miserably because, the ‘init.rb’ file would not be included. The generated warble.rb only includes the following

config.dirs = %w(app config lib log vendor tmp)

In addition to this the gems that we installed above would not be install. Here is same config with the lines we need (and the other cruft removed).

Package up the War file

$ warble

From here on in we’ll be looking at any gotchas, when deploying to the different Application Servers.

Deploy to Glassfish and test (effort: moderate – working: yes)

You can get Suns open source application server from http://glassfish.org. The current stable version of Glassfish 2.1.1, though Glassfish 3 is in active development. The installer comes packaged as a jar file. You can run the installer with

$ java -Xmx256m -jar glassfish-installer-v2.1.1-b31g-linux.jar

After accepting the license it should put all the files in a folder called ‘glassfish’ in the current directory.

$ cd glassfish
You need to run the following commands to finish the setup:

$ chmod -R +x lib/ant/bin
$ lib/ant/bin/ant -f setup.xml

Once the software is installed you can start the domain with:

$ bin/asaadmin start-domain domain1

And use either the admin console or the autodeploy directory to deploy the war file.

Glassfish now has a working copy of our application.

Deploy to JBoss and test (effort: n/a – working: no)

JBoss has a community and an enterprise edition. For the purposes of this test we’ll be using the community edition. The current stable version of JBoss AS is 5.1.0 GA. You can get a copy of JBoss from http://www.jboss.org.

Getting started is as simple as unzipping the archive and running:

$ cd jboss-5.1.0.GA

$ bin/run.sh

You can then use the admin console to deploy the application. One gotcha here, the first time I deployed the application using the console I got the following nasty message:

Application initialization failed: no such file to load — rack
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/vendor/rack.rb:1
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/vendor/rack.rb:22:in `require’
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/jruby/rack/booter.rb:22:in `boot!’
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/jruby/rack/boot/rack.rb:9
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/jruby/rack/boot/rack.rb:1:in `load’
from <script>:1

Turns out that after some digging there is an open jruby bug about the issue -  http://jira.codehaus.org/browse/JRUBY-3935

I also did a bit of digging through the logs and found:

16:27:50,703 ERROR [STDERR] Warning: JRuby home “/opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-stdlib-1.4.0.jar/META-INF/jruby.home” does not exist, using /tmp

I’ve not managed to find a solution to this problem. I will revisit this at some point in the future. After googling a little it may be possible to just revert to a few older versions that seemed to work.

Deploy to Jetty and test (effort: easy – working: yes)

The current version of Jetty is 7.0.1.v20091125, though the version that comes as part of your Linux distro may not be so up to date. You can either install it using your favorite package manager, or if you’re on Windows get it from the homepage at http://www.mortbay.org

Once you’ve installed Jetty copy the generated war file to the webapps folder, and run:

$ bin/jetty.sh

You should be able to go to http://localhost:8080/deploy_test

Congratulations you now have a working copy of your sinatra app on Jetty.
Deploy to Tomcat and test (effort: easy – working: yes)

The current stable version of Tomcat is 6.0.20. You can either install it using your favorite package manager, or if you’re on Windows get it from the homepage at http://tomcat.apache.org

You shouldn’t have to make any changes to get our web app to work on Tomcat. Once you’ve installed Tomcat copy the generated war file to the webapps folder, and run:

$ bin/startup.sh

You should be able to go to http://localhost:8080/deploy_test

Tomcat really is the bread and butter of Java Application Servers, especially outside the Enterprise.

Deploy to Websphere and test (effort: hard – working: yes)

NB: Websphere 6.1.0.11 was the first application server I ever deployed Sinatra too, it failed miserably. I spent a long time debugging and playing with it to make it work properly. The biggest problem stems from the fact that the default way of using rack as configured by warbler doesn’t work.

Start by logging into the administration console, it should be something like – http://localhost:9043/ibm/console

Click on ‘Servers’ and when it expands select ‘Application Servers’. From here you can setup a new server instance that we can use for our testing. Call the instance something like ‘deploy01′. You can follow the default creation steps all the way through.

Once you have a server instance to test on, you can deploy a new application. The big gotcha as mentioned above is that you can’t use filters, the good news is that it’s quite easy to switch out the rack filters for a rack servlet. Rather than duplicate information that already exists, I’ll link to the place I learned to deploy warbler to websphere, http://clint-hill.com/2008/11/26/jruby-on-rails-and-websphere/.

Websphere is not the easiest application server to setup in general, but once you get it all configured it is fairly robust. Worth the effort if you want an application server you won’t have to restart constantly (as can be the case with Documentum on Tomcat).

Deploy to Weblogic and test (effort: n/a – working: no)

Setting up Oracle Weblogic 10.3.2 was nothing short of awesome. The install process is intuitive and speedy, though the size is quite large, at ~600Mb,  compared to smaller cousins such as Tomcat. There is a wizard that walks you through the process of setting up your first domain, what Tomcat would call an instance and Websphere would call a profile, once the software is installed. I chose the default options for everything and had a running Weblogic server in about 20mins (including download).

My previous experience with Weblogic, was the version bundled with Documentum D6SP1. I’ve found both that version and the current fully fledged Oracle version to be a joy to work with.

Once the installation process is finished you can find the administration console at http://localhost:7001/console, you can then login using the username/password you picked during installation.

From the admin console it is a simple matter of clicking on ‘Deployments’ on the left and then when the screen loads clicking ‘install’, browse to the directory with the deploy_test.war file in it and start the install.

You should now be able to access the deployed application at http://localhost:7001/deploy_test…

Application initialization failed: no such file to load — rack from C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/vendor/rack.rb:1 from C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/vendor/rack.rb:22:in `require’ from C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/jruby/rack/booter.rb:22:in `boot!’ from C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/jruby/rack/boot/rack.rb:9 from C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/jruby/rack/boot/rack.rb:1:in `load’ from

Clearly thats not going to be the case though. This issue is very similar to the error message received for JBoss.

Conclusion

Not a bad scorecard really. Of all the Java application servers that I tested, only JBoss and Weblogic proved to be a problem. I’ll be retesting these two periodically to see if support has been improved (there are open tickets with JRuby). Until then I hope that this has been useful.

Up until now I’ve been using a fairly standard capistrano deploy.rb. The problem is that as we speak I’m trying to deploy a patch from my Windows work machine and it’s not working. The ethics of fixing one of my project from home aside, this is a problem. When I try and deploy the following error  comes up

can’t convert Net::SSH::Authentication::Pageant::Socket into IO (TypeError)

The change I was trying to deploy stops in its tracks. After quite a bit of searching I found a thread about the error that dates back to Capistrano 2.5.3… from 2008. What a shining example of open source.

To be fair the error isn’t necessarily in Capistrano, it may in fact be in Net::SSH. The sad thing is that we’re over a year later, and nothing has been done to fix the problem. Is the number of people that use Ruby from Windows machines so low that no-one has managed to fix it in a year?

I’m not expecting Jamis to fix it. I get that he has too much to do, and doesn’t have time to give out a bunch of freebie support, but I’m now faced with the choice of either trying to fix the problem, a task I don’t currently have time for, or ditching capistrano.

I’ve considered switching to heroku in the past and just never made the leap. For a start the app in question has to many moving parts for heroku. One thing I did like though was the notion that to deploy all I had to do was ‘git push target master’  and the app would be updated and deployed.

After toying around with rolling my own solution, I stumbled upon mislav’s gem. It lacks some of the features that I’m looking for, but its a good deal closer to the level that I need. It lacks the bloat of capistrano, which is important, because the biggest barrier to me getting in and fixing capistrano would be the size of the library and knowing where to start.

I very quickly migrated my existing application to use git-deploy. It’s not perfect for every problem, particularly if you’re doing multi stage deployments, etc, but at least I’ll be able to do a deployment everywhere I can get access to git now.

I’ve finally decided to release ActiveDocumentum.  ActiveDocumentum is a Ruby Gem that I created to bring some of the goodness, learned by pulling apart ActiveRecord, to accessing Documentum. Its nowhere near as mature, but I has been doing the job pretty well so far for the scripts and sites I’ve been using it for. It has a dependency on JRuby because it hooks into the DFS client libraries to facilitate connecting to Documentum. I’m going to post some further examples, but for the time being, here it is.

http://www.github.com/vertis/active_documentum

You’ll need a Documentum repository, JRuby and a copy of the DFS sdk to play with it.

Update: I’ve also done up a quick sample which you can find on my github (right next door to the actual library).

http://www.github.com/vertis/active_documentum_sample

One of my favorite finds this year is Programmable Web, a site dedicated to APIs and Mashups that are the lifeblood of web 2.0. Having just watched the movie Julie & Julia, which I highly recommend, I think that its only appropriate that I set myself a similar challenge. Take the top 50 apis, and within the next year develop a non trivial application for each of them (in no particular order). I’ve set myself a couple of rules:

• You are allowed to use any client libraries if they exist.
• Using multiple APIs in the one application is allowed (yay for mashup’s)
• Any libraries/applications that are developed in the course of the challenge must have tests/specs and documentation.
• And of course blog about each api.

First up:

• API: Google Maps
• Deadline: Thursday the 29th of October

You’ll be hearing from me soon. In the meantime I welcome anyone that wants to participate. Just leave a comment on this post, with your details. If it proves to be popular then I’ll get some kind of signup/participant website going.

Turns out Rack doesn’t read minds. I just spent the better part of the morning troubleshooting a stupid error, convinced there was something wrong with JRuby/Rack/Sinatra. As it turns out the error was caused by having a blank config.ru file,  I was editing one that was in a different directory (and believing that it was the correct file). My own stupidity aside, the error message wasn’t helpful, mostly because google didn’t return any results related to the same error. So while this admits my mistake, hopefully the next person who comes across this error will spend less time scratching their heads.

I was chatting to Craig Randall on twitter a little while back and he let me know that they were just about ready to go into Early Access for their new web service platform -  Documentum RESTful Services. I was excited then, and I’m even more excited now that I’ve had my first taste.

RESTful web services are hard to explain to people. I usually fall back on explaining them by linking them to Ajax, which is a Web2.0 technology that people are more familiar with. If you wanted to use Ajax to make your Documentum webapps more friendly before Documentum RESTful Services, then you’d be out of luck (short of creating a custom server side component).

RESTful web services are easily accessible because you don’t need any funky client library to access them. Pop open any web browser and type in:

http://localhost:8080/dctm_rest/resources/core/repositories/<RepoName>/folders

And you’ll get back an XML representation of all the root folders (after you’re prompted for a username/password). RESTful web services use only standard HTTP, which means that you’re cutting out the middleman of having to encode requests/responses in Soap (or some other container).

They’re automagically more language agnostic. Gone is the need for a complex client side library. For example, while it should strictly be possible to access DFS in any language, this doesn’t translate into a reality. So Documentum RESTful Services will allow us to step away from relying on Java/.Net and expand into using high level dynamic languages (python, ruby, javascript).

Let’s do a simple example in Ruby to demonstrate:

require 'rubygems'
require 'httparty'

class DctmRest
  include HTTParty
  base_uri 'http://localhost:8080/dctm_rest'
  def initialize(username, password)
    @auth = {:username => username, :password => password}
  end

  def repositories
    options = { :basic_auth => @auth }
    repository = "test_repo"
    self.class.get("/resources/core/repositories.json", options)
  end
end
d = DctmRest.new('username', 'password')
puts d.repositories.inspect

The example is probably more complex than it even needs to be. If I look at the output:

{"repository"=>[{"queriesUri"=>"http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/queries.json", "repositoryType"=>"managed", "typesUri"=>"http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/types.json", "foldersUri"=>"http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/folders.json", "checkedoutUri"=>"http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/objects/checkedout.json", "uri"=>"http://localhost:8080/dctm_rest/resources/core/repositories/test_repo.json", "name"=>"test_repo", "objectsUri"=>"http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/objects.json"}]}

I can see a very succinct set of other URIs that I can use to get other information. In the next post I’ll go through how to browse around a Docbase using Documentum RESTful Services. For now, take the time to sign up for the early access program, so you can help give feedback on what is a giant leap forward for Documentum in the web2.0 world.

Having stepped away from trying to get Documentum running on linux, I’ve been working on bringing the permissions through into our custom Lotus Notes/Documentum client. At the moment when a user imports a document to a given folder it does no permission checking before the fact, just errors nastily when it can’t write the document. A short stop by Prasad’s blog gave me a query to start from.

SELECT i_all_users_names as users FROM dm_group
WHERE group_name IN (SELECT r_accessor_name FROM dm_acl
WHERE object_name IN (SELECT acl_name FROM dm_sysobject
                       WHERE r_object_id = '0c0xxxxx80009e16'))
ORDER BY i_all_users_names

This is superb, but it doesn’t actually go far enough. If I was to attach it to my application and tell it not to let me import anything into the folder if my name wasn’t on the list I wouldn’t be able to copy anything into my home cabinet. It enumerates the members of any group within the ACL but doesn’t deal with named users & also the owner.

This will give me a chance to show off what you can do with my ActiveDocumentum project. ActiveDocumentum is a working name for a JRuby gem written to provide functionality similar to what can be found in ActiveRecord, with the addition of repeating attributes and other DQL specific items. While the final code for the below function may end up being in Java or .Net. Doing it using ruby allowed me to very quickly scope out the logic of what would need to be done to get the permission that a user had on an object (NB: If there is a way to do this simply with DQL I’m not aware of it):

require 'config/environment'

def get_effective_permission(r_object_id, user = nil)
  return 0 if user.nil?
  folder = DmFolder.find(:first, :columns => "owner_name, owner_permit, acl_name", :conditions => {:r_object_id => r_object_id})
  current_level = 0
  if folder.owner_name.eql? user
    current_level = folder.owner_permit
  end
    acls = DmAcl.find(:all, :columns => "r_accessor_name, r_accessor_permit, r_is_group", :conditions => "object_name = '#{folder.acl_name}' ENABLE(ROW_BASED)")
    acls.each do |acl|
      if acl.r_accessor_name.eql? user
        current_level = acl.r_accessor_permit if acl.r_accessor_permit > current_level
      elsif acl.r_is_group.to_s.eql?('1') and not acl.r_accessor_permit.nil?
        group = DmGroup.find(:first, :columns => 'r_object_id, i_all_users_names', :conditions => {:group_name => acl.r_accessor_name})
        current_level = acl.r_accessor_permit if acl.r_accessor_permit > current_level and group.i_all_users_names.include?(user)
      end
    end
  end
  return current_level
end

r_object_id = '0c0xxxxx80009e16'

user = 'John Doe'

puts get_effective_permission(r_object_id, user)

As you can see the above function takes a user and returns the highest permission that they have, by checking if they’re either directly listed on the ACL or in one of the groups on the ACL. The bottom line is that with ActiveDocumentum I can write less lines of code for one off scripts that I have to create, and include complicated logic that is lacking in straight DQL scripts. I will be posting more details on this in the near future.

I accept planetmcd’s criticism of my previous post. I’m aware that I’m less than eloquent and my arguments less than logical at times.

I can’t say for sure whether anyone has ever done a presentation like that here. I do know that something like that wouldn’t be accepted not just because of the images (sexual or not), but because it doesn’t conform to corporate standards. Ruby/Rails/Web2.0 has no such standards, more,  the culture is one of being risky, on the edge, and of pushing the limits.

There are probably many ways that it could have been done better, but it wasn’t. The problem stems from not going to Matt, and expressing that they didn’t like the presentation. They could have suggested using ‘Fragstar’ next time (via Renae Bair). Instead choosing to drag Rails through the mud publicly, “Here is a professional community that doesn’t respect women”.

I’m aware that Matt has defend his position, and the DHH may have made it worse, but I don’t condone the method this was approached in the first place. It’s sensationalist and unnecessary. Do people actually think they’ve improved the community by acting in this manner?

The dress code is only one facet of what I was trying (albeit poorly) to express. If you asked a programmer whether he would prefer to wear jeans and t-shirt or suit and tie to work which would he pick? What is the dress code at the Web2.0 development houses (not having worked at one I don’t know)? If it is jeans and t-shirts then that workplace is different to mine. My current employers wouldn’t consider them very professional either – This is where it goes to the heart of the community. You can do development the traditional, non agile way, any time you want to put on a suit and tie and forget you know techniques like metaprogramming/bdd/tdd (and don’t forget how to use windows, because that’s what corporate professionals use).

I have worked for a few industrial clients where staff had nudes as desktop wallpapers (we’re not talking partial nudity either), and pinups scattered around the sheds. There were certainly females around, though how they felt about it never came up. They would consider themselves professional, in that they provide top notch solutions to their clients. Warranties & quality assurance, etc. I doubt my current employer would find them very professional either, sweaty, greasy, and not very formal.

One of the things that I’ve heard raised when Australian corporate entities deal with overseas counterparts, is that we’re a good deal less formal and respectful than they are. Socially and culturally Australians are more laid back, some might say unprofessional. Different people are always going to have differing opinions on professionalism, I find it unlikely that Matt felt he was being unprofessional in using the pictures and analogy that he did. I would hazard he still doesn’t feel he was unprofessional, though he undoubtedly realizes that it was a mistake.