Using RESTful Web Services

You made it to part four of our series on REST APIs, so are you ready to get your hands dirty with a little bit of coding?

We can call a REST API from the web browser (client) using Javascript, or from the server using a language of our choice (such as Perl, PHP, Python, Ruby etc.) I’ll cover both in this article, server-side first, but first some good practices that apply to both.

Best Practices

Read The Terms & Conditions Carefully

Every web service comes with a set of terms and conditions attached that detail how you may (and may not) use the service. Read these very carefully and if you are building a business based on services provided by others, get a lawyer to check over the agreements before committing time and resources.

There are many potential pitfalls that can be hidden deep inside a service’s Ts & Cs. For instance you can’t mix Google search API results with results from other services, and you can’t use Yahoo’s FireEagle API to control nuclear facilities!

Be Nice To The Servers

Many web service providers implement rate limits. For example, Twitter limits clients to about 70 requests per minute, whilst Yahoo’s Search API limits you to 5000 requests per IP per day.

These numbers might seem high but a popular service can easily exceed them so ensure that you have adequate caching and throttling in place to minimise load - just as you would with your own database.

Security Is Job Number One

If you’re developing a web application then you know you can’t trust the data input by your users. You must strip it, sanitise it, scrub it and validate it to prevent those naughty hackers from playing havoc with your servers and disrupting your legitimate users.

The same applies to third party APIs. Whilst the likes of Google, Yahoo, FriendFeed et al are trustworthy, you can’t trust their users. Service providers do their best to ensure that the data they serve to you is clean but there can always be another backdoor or unpatched security hole, so validate and clean any data that comes from a RESTful API.

Subscribe to relevant security mailing lists as well as developer mailing lists for the APIs that you are using and be prepared to patch your application quickly. Get process in place to deal with security issues before it is needed.

Using REST Interfaces From Server-Side Code

So, you’ve chosen your API and your development language and are ready to start designing and coding your application. Where to start?

Re-Use An Existing Wrapper

The first stop I would recommend is to look for third party interface libraries in your language of choice:

Perl coders should check CPAN
PHP users should check PEAR
Python wranglers look to PyPI
Ruby hackers probably already know about RubyGems

Popular APIs often have third party wrappers available which will do a lot of the heavy lifting for you, like these libraries for the delicious.com API:

Almost every modern scripting language has a large repository of open-source libraries available, so if I didn’t mention your language of choice don’t despair, Google around and let us know about your findings in the comments.

Many interfaces are not 100% REST compliant or have other quirks so using a third-party wrapper will insulate you from having to know too much about the API. They usually have good community support and frequent updates too.

Roll Your Own

If you can’t find a wrapper library for the API you want, you’ll need to go in at a lower level, preparing the HTTP connection, sending the request and parsing the response. I’m not going to provide examples in every language because there are already some great articles explaining just how to do this:

Dive Into Python has a great tutorial on calling calling web services over HTTP.
Perl hackers just need to use LWP and then parse the response in XML or JSON.
There’s a nice tutorial on using PHP to call REST resources by George Papayiannis.
Similarly, for Ruby coders, here’s a lovely example of writing a a RESTful API wrapper by ActsAsFlinn.

A good place to find further examples and tutorials is the delicious.com search engine.

Server-Side Designs

Think carefully about the design of your server-side code, here are some ideas that you might want to take into consideration:

You might want to switch service providers (from del.icio.us to diigo, for example) so abstract your calls to third party services.
Handle failures gracefully. If a third-party service provider fails, it should not make your application look bad.
Think about timeouts, how long should you wait for a response? Can you send feedback to your users whilst they wait?
Record all success/failure rates. You’ll need to examine this data for trends.
Can you use multi-threading or asynchronous network calls to improve the response time of your application?
You can use XML schema definitions to validate XML responses, but beware that this gets pretty slow for complex interfaces.

Using REST Interfaces From Client-Side Code

Client-side coding generally means just one language - Javascript. As with server-side coding, you can find a lot of Javascript wrappers to add the output of a RESTful webservice to your web pages. One example of this is Google’s AJAX Search API.

Sending The Request

Using a RESTful API from Javascript is complicated by the same origin policy of modern web browsers which seeks to prevent XSRF by ensuring that Javascript from one domain cannot modify a document from another domain. This means that trying to POST to the del.icio.us API from your webpage (not hosted at del.icio.us) will always fail unless you do it with code loaded from del.icio.us via a <script> tag.

If, however, you have control of the domain which provides the service, then you can use an XMLHttpRequest object to send your HTTP request to the server. You can send GET, POST, PUT and DELETE requests using this object.

Note that XMLHttpRequests are asynchronous. That means that your code will not wait until it receives a response from the server, it just carries straight on. You need to provide a callback that will handle the response some time in the future. This is a common stumbling block for people starting out with AJAX.

Handling The Response

RESTful interfaces typically send their responses as either XML or JSON. Javscript can handle both although JSON is the easier of the two to work with.

In the case of an XML response, you can grab the document’s DOM object from the XMLHttpRequest’s responseXML member. From there you can use DOM navigation to access child or sibling nodes, jsut as you would with an HTML document, or you could even use XML Stylesheets to transform the document into the format of your choice.

JSON responses are easy to work with since they can be interpreted into a Javascript object but don’t be tempted to just eval() the response text - remember “security is job number one”.

Instead, ensure that invalid characters are not accepted by using a parsing function like this:

String.prototype.parseJSON = function () {
    try {
      return !(/[^,:{}\[\]0-9.\-+Eaeflnr-u \n\r\t]/.test(
        this.replace(/"(\\.|[^"\\])*"/g, ''))) &&
        eval('(' + this + ')');
    } catch (e) {
      return false;
    }
};

End Of Part Four

Phew, we covered quite a lot during this post but still there was hardly any code in there. I recommend that you have a good look through the links for code and examples that apply to your project, then sit down and do some design before you start coding your own projects.

You’ll probably have noticed that being a client of a RESTful API is very much like being a client of any published API - don’t trust the input, handle errors gracefully, etc - so there wasn’t too much influence from the RESTful side of things. That will all change in the next post, when we come to designing RESTful interfaces.

As always, feel free to ask questions below and thanks for reading.

Creative Commons licensed photos by Nat W and davestfu respectively.