Pezholio

My Response to BBC Watchdog: A Chrome Plugin for Just Eat

| Comments

A couple of nights ago, the BBC’s Watchdog ran an investigation into restaurants on just-eat.co.uk. Their main focus seemed to be on ‘fake’ takeaways set up in residential premises or on industrial estates, but they also touched on how many premises listed on the site had low food safety ratings.

This gave me a bit of an idea. I’d fiddled around with Greasemonkey scripts back when I used to use Firefox (which basically use Javascript to alter the contents of pages if you have that script enabled), and I wondered if I could do the same in Chrome to display food safety inspection ratings next to each premises on Just Eat.

It turns out that you can (Chrome has the concept of content scripts), and with a few lines of Javascript, I had an extension that iterates over each premises listed on the search page of the Just Eat website and uses the Food Standards Agency API to search for the result of the take-away’s last food safety insepction.

It doesn’t catch every single premises (mainly because the spelling on the Food Standards Agency Website and Just Eat may not match exactly), but it seems to catch most of them. Interestingly, I also found out that the restaurant I used to use on Just Eat (who used to have a 5) have just gone down to a 2. I’ll be using somewhere else in future then.

You can install the plugin via the Chrome web store, if you’ve got any issues with it, please let me know in the comments. The code is also on Github, so if you’ve got any ideas for improvments, please fork and send a pull request. I’d love to see this extended to support other take-away delivery sites, so if you fancy having a crack, please do!

Lost the Source to My Octopress Blog (Ooops)

| Comments

As I use Octopress to publish my blog, there’s no database and no admin panel for my blog. It’s all generated on my local machine, and pushed to Github pages.

On one hand, this is great, hosting is free (on Github), and the site is hella lightweight (just flat HTML pages). I also write my posts in Markdown (which makes writing so much more enjoyable than squinting into a WYSIWIG editor on a webpage), save them on my local machine and then run a couple of commands in my terminal to pubish my new blog post.

The way it’s supposed to work is you should have two branches in GitHub, one for the posts that Octopress generates, and one for the actual source of your blog, so if you forget to push your source to Github (like me), and get a new computer (like me), and don’t BACK EVERYTHING UP (like me), things sort of fall apart at the seams.

I posted a question to StackOverflow about this and, after a day, I got no response, so, with a new blog post I was eager to publish, I decided to get to hacking.

First, I pulled down a new copy of the Octopress source and changed the settings to (more or less) represent what I had before. I also have a modified vesion of the oct2 theme, so I pulled that down and made the modifications I’d done on my old blog.

Next came the hard bit, getting my posts back into the source. I came across this code to import a Blogger export into Octopress, so decided to use this as a basis. I then pulled down my Github pages repo, and made some tweaks to the Blogger code I grabbed to scrape the title, content, published date, and categories out of the published HTML. I didn’t need to worry about comments, as these are hosted on Disqus (because flat HTML remember?)

I then saved my script in the source directory of the cloned Octopress repo, ran it, and all the pages were generated automagically (in HTML, rather than Markdown, as Octopress supports bog standard HTML in case you’re mad and would rather write blog posts in raw HTML). I then ran rake generate and then rake preview to see my new site in all it’s glory.

To my amazement, it worked, and, after setting up Github Pages, I ran rake deploy, and the site was live!

I also remembered to commit my source this time (documented quite clearly(!) here), so hopefully will never have the same problem again.

If you’re an idiot like me, and have done the same thing, you can see my script here, and please, remember to commit your source next time!

Automated Testing With Cucumber and Phantom.js

| Comments

Since I last spoke to you, I’ve been very busy. I’ve recently changed jobs and am now working at the Open Data Institute. It’s a massive change from working at my old place, and one of the many new things I’ve learnt about is the joys of automated testing.

Working, as I did, on my own, in isolation (almost as a one man skunkworks) meant I didn’t really need to worry about the quality of my code, and, as far as testing went - I could just test it in my browser and it’d work, right?

Well, yeah, I suppose, but this isn’t The Right Way of doing things. Also, if I changed a line of code, how could I be sure that what I changed wouldn’t have unintended consequences in some obscure part of my application without testing the whole application myself? Also, as I would often not have a development environment (at least in my non-Ruby apps), this would mean introducing a whole load of test data into my live environment.

In short, writing code without tests is a Very Bad Idea, and, as we move to the holy grail of continuous deployment at the ODI, this would mean that my code will be getting pushed into the live environment regularly by robots, so we need to make sure that any changes made aren’t going to break anything.

This is where Cucumber comes in. I won’t go into this too much, but this is where the magic happens. I can write expectations (in a language called Gherkin), and write code in Ruby that attempts to match these expectations. I can call code that queries the database, spins up browsers and acts like a real user and records HTTP interactions.

This is all well and good, but if I’m building an app that makes heavy use of JavaScript, then, out of the box, Cucumber (via Selinium) needs to spin up a real browser (such as Firefox or Chrome). This is OK when developing on my Mac at work, but once the code is being tested remotely in Jenkins before deploy, this isn’t possible.

However, there does exist a neat tool called Phantom.js - a fully functional WebKit browser (like Chrome), but without one crucial thing, a window. All user interactions are carried out via JavaScript, and you can do everything a normal user would do, but without having to have a GUI. This mean you can automate all sorts of tasks on the command line, screen scraping, filling out forms, and yes, testing.

As Phantom.js is not like other browsers, getting Phantom.js and Cucumber to play nicely together is not, a straightforward task. However, there is a nice tool called Poltergeist, which allows you to do just that.

Getting Poltergeist set up in a Rails app is easy, but not trivial, and I had to jump through a few hoops to get it sorted, so I thought I’d document it here for the ages.

I’m assuming you’ve got Cucumber-rails set up in your Rails app already, so if you haven’t, take a look at this Railscast to get you started.

Once you’re all set up, the first thing to do is install Phantom.js - if you’re on a Mac and running Homebrew, it’s as easy as running brew install phantomjs on the command line, otherwise take a look at these instructions.

Next, add poltergeist to your gemfile (probably in your :test group) like so:

gem 'poltergeist'

and run bundle install

The next thing to do is register Poltergeist as a new browser in Cucumber, and make it run as the default driver for all your JavaScript tests. Open up your features/support/env.rb file and add the following lines:

require 'capybara/poltergeist'

Capybara.register_driver :poltergeist do |app|
    Capybara::Poltergeist::Driver.new(app, {debug: false})
end

Capybara.javascript_driver = :poltergeist

Then you should be good to go! Crucially, you need to make sure you add the @javascript tag to all your tests, so Capybara knows to use Poltergeist for your tests, but other than that, when you now run your tests, rather than a browser window being fired up, everything happens in the background like magic!

There are a couple of other cool things you can do with Poltergeist that may help you when you’re writing initial tests, like taking screenshots and executing arbitrary JavaScript (like in the JS Console in Chrome). Take a look at the docs for more info, and have fun!

One very small gotcha, is that, as Capybara fires up a real browser when running tests, if you’re recording API interactions with VCR, it will try and record your browser interactions too. To get around this, you’ll need to make VCR ignore local requests by adding this line to your features/support/vcr.rb file:

VCR.configure do |c|
    c.ignore_localhost = true
end

Council Websites: Time for a New Approach to Navigation?

| Comments

If you spend any amount of time on different council websites (like I do, sad as I am), you’ll noticed that most of them will have a similar navigation structure, starting with categories like Advice and Benefits, Environment and Planning, Transport and Streets etc.

This is known as the Local Government Navigation List (or LGNL), and it’s been going since the early days of eGovernment (remember that?), backed up by a steering group of people who shape and mould the list as time goes on.

I’ve been of the opinion that it is in serious need of reexamining for some time now - I don’t feel it’s customer focussed, there are too many levels and it can, in some cases be far too detailed for a council website navigation.

That said, I’ve shrugged my shoulders and put up with it, as for some time I’ve believed that the future of website navigation is search. Why put up with clicking endless links when you can just type a few words in and (generally) get to what you’re looking for. It’s what I do and what (I assumed) most people would do too.

However, you know what they say about assumption, and I was surprised to see that, in their latest redesign, the gov.uk team are dropping the idea of a big search bar on the homepage, and going back to good old fashioned navigation.

You can see more about their rationale on the blog post, but, suffice to say, it’s backed up by hours of testing with real users, the same sort of users that we have on our council’s website, and it does makes sense.

This leaves us with a problem, if search isn’t the way forward, where does that leave LGNL? If, as I think, LGNL is in serious need of looking at, then what can we do?

I put up a Google doc yesterday to kick around a few ideas, and already, I could see it was becoming monolithic, exactly like the LGNL I would like to see killed off. So what is the future?

I believe we need a user tested, people focused navigation system, with no more than two levels, but one that isn’t prescriptive, and can be moulded and shifted to meet the needs of a particular council. Maybe this is something that any potential local GDS can do?

I don’t have all the answers, but I think it’s a discussion we need to have. As always, let’s have those thoughts in the comments.

Twitter Kills RSS: How to Roll Your Own Feed With Scraperwiki

| Comments

As announced this week, Twitter are continuing their war on developers by killing off RSS feeds.

Add this to the fact that, once the old API is killed off around the middle of next year, all endpoints will require authentication, it’s going to be a bit harder to do simple stuff with Twitter (such as display the latest tweets from your Twitter feed).

You could, of course, use Twitter’s own widgets, but what if you (quite reasonably) want control over the look and feel of your own website? You’re either screwed, or you have to sign up for a developer account, create an application and use OAuth authentication just to display a couple of tweets, which is possibly overkill.

However, there is a third way, by using Scraperwiki to scrape the tweets from your profile page you can create an RSS feed which will also act as a drop in replacement for any feeds you already use.

It’s pretty simple, just go to my scraper on Scraperwiki, click copy, and change the username to whatever username you want a feed for. Save the scraper and then run it.

Next, copy this URL, replacing YOUR_SCRAPER_NAME with the name of your scraper (this will be in the URL when you save the scraper):

https://views.scraperwiki.com/run/twitter_rss_feed/?scraper=YOUR_SCRAPER_NAME

And there you have it! The URL you have will then display an RSS feed of your latest tweets. An added advantage of this approach is that the scraper runs every day and saves the tweets in a database, so you’ve also got a daily archive of your tweets.

One disadvantage is, however, as the scraper only runs once a day (unless you upgrade your Scraperwiki account so you can run scrapers every hour), it may not be particularly up to date, but it’s better than nothing!

Local GDS: A Skunkworks for Local Government?

| Comments

Skunkworks logo

A bit of history

GDS has received a lot of good press by going against the grain of how we in the public sector traditionally do things and acting more like a startup, using agile project management techniques, shying away from the traditional Microsoft stack by using Ruby and other open source programming languages and publishing (most of) their code on Github.

The problem(s)

As someone who works in local government and is an advocate of these practices, I would love to see this sort of thing cross over into local government. There are pockets of local government webby innovation around the country, and this isn’t often shared - meaning that there are some councils who have great websites, and some councils who have not so great ones.

This is exacerbated further by the fact that some councils have no technical resource at all, meaning they’re beholden to suppliers who sometimes (not always!) charge the earth for simple solutions and tweaks to existing products.

The solution(s)

Therefore, it would seem that the simple solution would be to abolish all local government web teams and have one central web team for all local councils, but to do this would be to miss the point of local councils completely. We could just as easily say let’s just have one national waste service, or one national planning department. By being close to their residents, and having councillors who live in an area, councils can better adapt to their residents’ needs.

I think, therefore, a better solution would be to have something closer to a local government skunkworks.

What is a skunkworks?

For the uninitiated, the first skunkworks was set up in 1943 at the defence company Lockheed, and was tasked with working on highly secret and advanced projects, unheeded by bureaucracy and given a high degree of autonomy. The Lockheed skunkworks came up with number of famous aircraft designs, including the U-2, the SR-71 Blackbird, the F-117 Nighthawk, and the F-22 Raptor.

This has been tried before in the public sector with the HMG Skunkworks (presided over by one Mark O’Neill), who worked on (amongst other things) the latest version of the government ePetitions site. The Skunkworks has since been moved into GDS, but it could be argued that without the skunkworks, there would have been no GDS.

So, how would a local government skunkworks work?

There are two possible solutions, either a centralised team that works out of an office, or a collection of individuals working for local councils seconded for a couple of days a week.

However it works, these people will already be doing great stuff in the field of web development, design, UX, sysadmin etc.

The skunkworks could work on modular projects (like planning systems, ‘find my nearest’ tools, bin collection day finders etc), as well as actively looking for projects from councils who don’t have technical resource, but also don’t have the budget to use big suppliers.

All these projects would be open source and published on Github for those with the technical nous to self-host, but could also be sold / given away via the G-Cloud, so the headache of setup and hosting is taken away from them and taken care of by the skunkworks team.

This is sort of the thing we’ve already talked about with Project Maple, but I think I’ve struggled to communicate these ideas properly, and, to be honest, they’ve only just crystalised in my head recently.

What do you think?

I’m keen to get people’s thoughts on this, so let me know what you think, either in the comments, or join the Project Maple Google Group and have your say there.

data-updated=”true”>Sep 4th, 2012

Take a screenshot of any tweet using Casper.js Twitter kills RSS: How to roll your own feed with Scraperwiki

Take a Screenshot of Any Tweet Using Casper.js

| Comments

While Twitter do offer a way of embedding tweets, there’s no guarantee that this feature will stay in its current form, and equally no guarantee that if the tweet is deleted, the embed will continue to work, which is a potential headache for journalists and bloggers.

With this in mind, I’ve developed a quick solution to the problem, using the fantastic headless browser Phantom.js and Phantom.js utility Casper.js.

Simply put, all the solution does is allow you to take a simple screenshot in png format of any tweet you give the URL for.

To get Phantom.js and Casper.js running, first install phantom.js and then casper.js (I won’t go into the details here).

Next, download the script to your machine by running:

wget https://raw.github.com/gist/3518514/51c87bc6e59040948b7aebe08cb73e500e0783e5/tweetshot.js

You will then be able to grab a screenshot by running the command:

casperjs tweetshot.js {tweet permalink} {output filename}

Your screenshot will then be saved with the filename you requested (with a .png extension).

And that’s it really! Ideally I’d like to wrap it in a simple Sinatra app, so you don’t have to have any technical knowledge to get a link, but that’s something for another day.

If anyone has any improvements to suggest, or would like to build on this, please let me know. I’d love to hear what you can do with this.

Make Your Own Food Safety Twitter Bot With Scraperwiki and Ifttt

| Comments

Food safety is a subject very close to my heart, and something I can get very geeky and passionate about.

This isn’t because I’ve ever been struck down by a particularly nasty case of the runs, but because I’ve been very heavily involved in the publication of online food safety inspections reports since 2006, when I worked on Ratemyplace, the food safety inspection website for Staffordshire councils.

Because of my work with Ratemyplace, I’ve had a lot of people asking “can we get Ratemyplace in my council area?” It’s not really up to me, but I have put something together that will replicate one aspect of Ratemyplace’s functionality - Twitter alerts, for your council area without you having to write one line of code.

Step 1 - Get the data

The first thing you’ll need to do is look on the national Food Ratings website, to see if your council is there. More are being added as time goes on, so if your council isn’t there, there’s a chance they’ll come on board in the near future. The best way to do this is have a look on the open data page and scan down the list (it’s divided into regions, which makes it easier).

If your council is there, click on the link marked ‘English language’ (which is the link to the data), and copy the address from the address bar. Armed with this link, you’re then ready for the next step.

Step 2 - Make your scraper

I know what you’re saying, ‘but you said this wouldn’t involve coding!’ Worry not, dear reader, this won’t hurt a bit.

Next, you’ll need to create an account on ScraperWiki. Once, you’ve done that, go to this scraper on ScraperWiki, and click ‘Copy’ as highlighted below:

Copying the scraper

You’ll then be presented with the inner workings of the scraper, but don’t worry, all you need to do is replace the text that says {ENTER YOUR URL HERE} with the url you copied earlier. Once that’s done, you can rename the scraper to something more meaningful by clicking on the title (as shown), and then click ‘Save Scraper’ at the bottom right of the screen.

Editing and saving the scraper

Once your scraper is saved, you can click ‘Back to scraper overview at the top right of the page’ to see your scraper’s page. Now click edit, under the ‘Schedule’ section as highlighted below, and choose the ‘Run every day’ option.

Setting the schedule

Then click the ‘Run now’ button just above the ‘Schedule’ section, and go an make a cup of tea. Depending on how busy Scraperwiki is, this should take around half an hour.

Step 3 - Making your feed

Once you’ve had a cup of tea, refresh the webpage with your scraper in, and it should look something like this:

A working scraper

This means the scraper has done its magic and you can now more on to the next step. Click the ‘Explore with API’ button as marked below:

Explorer with API link

Now, copy this code below:

select name || ", " || address as title, "Rating: " || rating  as description, link as link, lng || " " || lat as "georss:point", date from swdata  order by date desc limit 10

and paste it into the box marked ‘query in SQL’, then click ‘copy’ as shown below:

Making your feed

Paste it into a text editor and add format=rss2 to the end of the URL. You now have an RSS feed, ready to take along to Ifttt

Step 4 - Tweet your feed!

Next, pop along to Ifttt and create an account. You’ll also need a Twitter account, either your own, or a specific Twitter account for this purpose (I recommend the latter).

In true Blue Peter style, I’ve prepared a ‘recipe’ for you to follow on Ifttt. Once you’re logged in to Ifttt, go to the recipe, click ‘Activate’ under the Twitter Channel heading to connect your Twitter account (make sure you’re logged in to the right account!), follow the instructions on the window that pops up, then click ‘Done’.

Next, scroll down and in the box marked ‘Feed URL’, and copy and paste the URL you created earlier.

Ifttt

The next box shows the format that the Tweet will be posted in, you can fiddle with this if you want, or just leave it as is.

That’s it! Just click ‘Use Recipe’ and every time the ScraperWiki scraper picks up a new inspection, your Twitter account will tweet it!

If you want to see it in action, check out @EatSafeWalsall. I’ve been running this for a while, and it seems to be picking up quite a few followers!

If you do use this recipe, or have any problems following my none-too-lucid instructions, please let me know in the comments. Have fun! :)

New Blog, New Beginning!

| Comments

Shamefully, it’s been almost a year since my last blog post, but I’ve finally decided to get my act together and get cracking with a brand new blog!

As much as I love Wordpress, I’m starting to think it’s becoming a bit too bloated to be a blogging platform, and, with this in mind (and just because I like fiddling), I’ve switched to Octopress.

Octopress is most definitely for hackers. It’s built in Ruby, and, rather than relying on a database like Wordpress does, it generates pages on your local machine, which you then push to a remote repository.

The benefits of this is you can get free hosting (I use Github pages, but you can also use Heroku), and, as the pages are static HTML, you’ll always have a super fast blog (so no more Slashdot effect if you get a lot of traffic!).

Another benefit is that you can write your pages in Markdown, which is a great way of writing text that converts to HTML with the minimum of fuss.

I’ve used a (slightly customised) version of the Oct2 theme, as although I like the default Octopress theme, as Octopress gets more prevelant, so does the default Octopress theme, and I like to stand out from the crowd. There’s not a lot of themes out there, but I’m told the next version of Ocotopress makes theming a lot easier, so expect to see more kicking around soon.

I’ve also imported all my blog posts from Wordpress using Exitwp, so my old brain dumps will always be available for all the world to see. Some of the Markdown is a bit funky (particularly on the code side of things), so that needs sorting. I’m also working on importing my old comments too.

All in all, for me, Octopress is a more enjoyable way to blog, so hopefully you’ll be seeing much more blogging from me. I’m also interested to know any tips or gotchas on blogging from Octopress users (I’m particularly keen to know about good Markdown editors for OSX), so let me know in the comments!

Generating QR Codes Using PHP and the Goo.gl API

| Comments

QR codes are all the rage at the mo, after a bit of a slow start, they’re popping up everywhere, even appearing in cookery programmes.

I’ve decided to take advantage of this, and have started a project to get QR codes on planning notices, so if someone is out and about and they see a planning notice attached to a lampost, they can scan the code with their phone and immediately find out more about the application.

I’ve still not finished this, but the first hurdle was actually generating the QR codes themselves. There are plenty of APIs out there, but, the more information you put in a QR code, the more complex it is, making it more difficult it is to scan and more prone to errors.

What I really wanted to do was shorten a URL and then create a QR code from that. If this is going to be automated, I’d need a little script to do the legwork for me.

I chose Google’s goo.gl url shortener, as it provides an easy option for creating a QR code, just by putting .qr at the end. Here’s the code, with comments along the way:

// We're going to be outputting this as an image, rather than as plaintext, to make it easier to include in, for example a webpage
header("Content-Type: image/png");

// We put the long URL in an array, as we're going to encode our data as JSON for the Goo.gl API 
$request = array(
    'longUrl' = $_GET['url']
);

// As we need to POST the data, we're using cURL
$ch = curl_init(); 

// This tells cURL to output the result of the request
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// This is the URL of the Goo.gl API
curl_setopt($ch, CURLOPT_URL, "https://www.googleapis.com/urlshortener/v1/url"); 

// We're setting the content type of our request as JSON
curl_setopt($ch, CURLOPT_HTTPHEADER, Array("Content-Type: application/json"));

// It's a POST request
curl_setopt($ch, CURLOPT_POST, true);

// And here is the contents of the request we set earlier, encoded as JSON
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($request));

// Now we're getting our data, again, as per the request, it's JSON, so we're using json_decode() to convert to JSON to a PHP object
$data = json_decode(curl_exec($ch));

// Closing the cURL request 
curl_close($ch);

/*
The response is in this format:

{
 "kind": "urlshortener#url",
 "id": "http://goo.gl/fbsS",
 "longUrl": "http://www.google.com/"
}

and it's 'id' we're intested so:
*/
$shorturl = $data-id;

// We now add .qr to the end of the URL and get the raw contents of the resulting QR code
$image = file_get_contents($shorturl.".qr");

// Now echo the result out! As we've set the header of the document to be a PNG image, the browser (or whatever) knows to display an image, rather than a meaningless string of characters
echo $image;

And there you have it! Any questions, give me a yell in the comments. Next step is getting the codes on the notices, which I’m nearly there with - will let you know how I get on.