Google App Engine, JRuby, Sinatra and some fun!
Yesterday I was experimenting with Google App Engine for a little project that I was working on. I literally started from ground zero and could do my thing after a long night. I'm blogging about it to share you the idea and save you some time googling solutions. So, here is the thing in brief: I wanted to parse some feed on periodical basis and send an email with new entries.
Looks like a 10 minutes with Nokogiri and cron jobs. Actually that's true except of the fact that I need to pay for a VPS so that I can run the script with various gems (since I needed to do some work on the feeds before sending, but that's for another topic) and for using cron jobs. Thus, I decided to try something new this time and I went with GAE since it has memecached that I can use as an LRU cache, also it has cronjobs and finally it has a mail system. I'm using JRuby and Sinatra for this project.
Here are the steps:
Install JRuby, I'm using RVM on my machine:
rvm install jruby-1.5.5
rvm use jruby
Install needed gems, those versions are the ones which worked for me:
gem install jbundle
gem install appengine-sdk -v "1.4.0"
gem install google-appengine -v "0.0.19"
Create a simple app:
appcfg.rb generate_app notifier
cd notifier
Now let's create the following files (Please note that the following code can be further enhanced):
Gemfile
# Critical default settings:
disable_system_gems
disable_rubygems
bundle_path ".gems/bundler_gems"
# List gems to bundle here:
gem "appengine-apis"
gem "appengine-rack"
gem "sinatra"
gem "sinatra-reloader"
gem "jruby-rack", "1.0.4"
gem "nokogiri","1.5.0.beta.3"
config.ru
(fill the sender/receiver emails and the feed URL)
require 'sinatra'
set :sender_email, 'SENDER_EMAIL'
set :receiver_email, 'RCEIVER_EMAIL'
set :feed_url, 'FEED_URL'
require 'app'
run Sinatra::Application
app.rb
require 'digest/sha1'
require 'appengine-apis/memcache'
require 'appengine-apis/logger'
require 'appengine-apis/mail'
require 'appengine-apis/urlfetch'
require 'sinatra'
require 'sinatra/reloader' if development?
require 'nokogiri'
class Fetcher
def initialize(sender_email,receiver_email,feed_url)
@sender_email = sender_email
@receiver_email = receiver_email
@feed_url = feed_url
end
def act
send_email(create_message(get_results))
"done"
end
private
def memcache
@memcached ||= AppEngine::Memcache.new(:namespace=>"cache")
end
# get the new results only
def filter_results(results)
res = []
results.each do |hashed_guid,info|
if memcache.get(hashed_guid).nil?
memcache.set(hashed_guid, 1)
res << info
end
end
res
end
# parse the feed
def get_results
results = {}
doc = Nokogiri::XML(AppEngine::URLFetch.fetch(@feed_url).body)
doc.css('item').each do |item|
title = item.at_css('title').content
link = item.at_css('link').content
description = item.at_css('description').content
guid = item.at_css('guid').content
results[Digest::SHA1.hexdigest(guid)] = {:title => title,:description => description, :link => link}
end
filter_results(results)
end
# create email message
def create_message(results)
template = ""
results.each do |info|
template += <<-DOC
Title: #{info[:title]}
Description: #{info[:description]}
Link: #{info[:link]}
--------------------------------------------------------------------------------------------
DOC
end
template
end
def send_email(message)
AppEngine::Mail::send(@sender_email,@receiver_email,"New updates for the site",message) if message.length > 0
end
end
get '/' do
"Hello World!"
end
get '/notify' do
Fetcher.new(settings.sender_email,settings.receiver_email,settings.feed_url).act
end
cron.xml
(You can set the period here, I'm using 15 minutes intervals)
<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
<cron>
<url>/notify</url>
<description>fetch new data</description>
<schedule>every 15 minutes</schedule>
</cron>
</cronentries>
Now, create an application-id at appspot.com. Then go to Administration >> Permissions and make sure the sender/receiver emails play some role in the app. Personally, I granted them the developer role.
Now, go back to the source and modify the application-id in WEB-INF/app.yaml
application your-app-id
We should be ready now, start development server, watch the console and make sure no exceptions are there:
dev_appserver.rb .
If everything is OK, go to http://localhost:8080/notfiy and you should see "Done".
Let's go live now:
(the first time you execute this, you will prompted to submit the GAE email/pass combination)
appcfg.rb update .
It should now be running at http://your-app-id.appspot.com, and in few minutes you should be receiving the first email. If not, go to app engine page then to Main >> Logs and check what's the problem.
I like this stack cause it's Zero cent cost, Zero deployment time, flexible, and it gives me exactly the freedom I want in terms of needed gems. Give it a shot, you won't regret it!