MMMeeja

We’re finally up to part five in our series on REST APIs, so by now you should be familiar with URLs, HTTP, understanding and using RESTful APIs.

If you’ve made it this far on our journey, I think it’s safe to assume that your you’re a bit of a coder, which is good because in this installment we talk about how to provide a RESTful API to your own web services.

URLs Again

You might have your own ideas of the kind of services or data you want to expose via an API or maybe you’re reading this to understand the motivations behind some APIs that you already know. Either way, we’re going back to the start of the series to take another look at the URL, which, you remember, identifies a unique resource. This post uses an example of a shop with products in different categories so a product URL might look like this:

http://myshop.com/categories/toys/products/steam-train-0023/

Performing a HTTP GET on this URL would return a representation (in HTML, XML or JSON) of the data that we have on the “steam train” toy.

If we wanted information on the “toys” category itself, we would GET this URL:

http://myshop.com/categories/toys/

Now, take a look at this URL, which returns a set of all the products that are classed as “toys”

http://myshop.com/categories/toys/products/

Does this break the rule that a URL identifies a unique object? No, because the collection is itself an object, much like the “shopping basket” that Brian Hendrickson and I had a useful discussion over in the comments on REST examples earlier in this series. We can create, read, update and delete collections - they are just a little more abstract than products or baskets.

RESTful URL Patterns

Notice the pattern in all the RESTful URL examples that I’ve been using in this series? No? Then have a look at these:

base_url/name/id/name/id/

base_url/name/id/name/

base_url/name/id/

In the shopping example, the base_url is just the domain name, but it’s very common to see sub-directories of domains, like https://api.del.icio.us/v1/.

After that we see combinations of names (“categories”, “products”) and identifiers (“toys” “stream-train-0023”) If the RESTful URL ends in an identifier, it returns a single object. If it ends in a name it returns a collection of objects. That is a beautifully simple yet powerful pattern.

Not Names But Axes

OK, now I want you to forget about names and identifiers and instead think about axes and values. Yes, axes, like on a graph:

base_url/axis1/value1/axis2/value2/axis3/value3/

Any mathematicians reading are hopefully saying “Aha!” right now, but the rest of us might need a bit more help with this concept. Try this URL:

http://mygraph.com/x/1/y/2/z/1/

It represents the point (1,2,1) in 3D, whereas this...

http://mygraph.com/x/1/y/2/z/

...is a plane, or the collection of all points on the Z-axis constrained by x=1 and y=2.

I’m trying to get you to think at a higher level than web coders typically have to do. A well designed architecture should have a basic model which makes sense at the most abstract level, so think in terms of relational algebra, set theory and matrix arithmetic.

Returning The Data

You’ll have respond to requests with either a HTTP error code (such as 404 Not Found) or a representation of the resource. The last named axis in the URL defines what type of resource you’ll be returning. Outputting a collection is there’s no subsequent value or the object on its own if there is.

You need to sit down and do some interface design now, paying particular attention to:

• Calculate (and document) exactly what data should be returned for each type of resource you’ll be returning.
• What formats will you support when describing resources? XML, JSON, HTML, PHP?
• You should provide an error document with a HTTP error. What data will you include to describe the error?
• Think about paging and rate limiting now, not when your servers collapse under heavy loads.
• Your interface will change over time. Yes it will. How will you manage the transition from version 1 to version 2?
• URL parameters, specifying things like output format, application keys, paging offests, search parameters etc. Include these in your specifications.

There’s a lot to think about in that list and you should realise that it’s not good to just build a SQL statement out of each URL component. Take the API design you created above and make a matrix, plotting each type of URL against the four HTTP request methods, like this:

	GET	POST	PUT	DELETE
http://myshop.com/categories/
http://myshop.com/baskets/
http://myshop.com/baskets/1/
http://myshop.com/categories/toys/products/steam-train-0023/
http://myshop.com/categories/toys/products/
http://myshop.com/categories/toys/

Now, for each of the blanks you need to specify a handler or decide that the handler is innappropriate (for example, if the API cannot delete a category). Each handler will be a function or class in your server application. If you are using a Model-View-Controller framework, the handlers will not be a Model, View or Controller - they will be a bridge.

Remember that your API must remain stable no matter how much your application architecture changes.

Practicalities

Any API will come in for some serious abuse, not just from malicious users but from prototype, untested applications so you’ll need to plan ahead. Consider implementing these features now:

• Banning users by username, IP address or referrer
• Rate limiting & timeouts
• Caching
• Paging
• Returning HTTP code 304 Not Modified
• Supporting the HTTP If-Modified-Since header

Testing your API is vital but at least it should be pretty easy. Using a tool like curl or wget to automate calls and then validate the output against XML schemas etc. Don’t forget to run some load tests too.

Conclusion

Phew, finally at the end of this series and it’s been fun to write and stirred up some great discussion.

Did I cover everything you wanted? If there’s still some uncertainty, then please ask in the comments.

---

Creative Commons licensed photos by briankennedy and Khalil Sawant.

This article series will take you through RESTful web services, introducing the technologies behind them, what they are, how to use them and how to implement them.

The series is presented in five parts, each of which will be posted over the next few days so sign up to the MMMeeja RSS feed to make sure you don't miss a single post!

REST (or Representational State Transfer) is fast becoming the de facto standard for exposing APIs on the web, beating more complex SOAP/RPC services. It's easy to understand why - it's a logical extension to how we all use the web already - as you'll see as the series continues.

I start off with a couple of introductory articles to the underlying technologies to get you up to speed:

1. What Is A URL?
2. An Introduction To HTTP

Then, let's get to the meat with:

3. Examples Of REST Interfaces
4. Using RESTful Web Services

And for the more advanced:

5. Designing And Implementing A RESTful Interface

So, the series will be a trip from n00b to l33t h4xx0r in five installments. Don't forget to subscribe.

---

Creative Commons licensed photo by Bobby ~ Lawcrow911.

You made it to part four of our series on REST APIs, so are you ready to get your hands dirty with a little bit of coding?

We can call a REST API from the web browser (client) using Javascript, or from the server using a language of our choice (such as Perl, PHP, Python, Ruby etc.) I’ll cover both in this article, server-side first, but first some good practices that apply to both.

Best Practices

Read The Terms & Conditions Carefully

Every web service comes with a set of terms and conditions attached that detail how you may (and may not) use the service. Read these very carefully and if you are building a business based on services provided by others, get a lawyer to check over the agreements before committing time and resources.

There are many potential pitfalls that can be hidden deep inside a service’s Ts & Cs. For instance you can’t mix Google search API results with results from other services, and you can’t use Yahoo’s FireEagle API to control nuclear facilities!

Be Nice To The Servers

Many web service providers implement rate limits. For example, Twitter limits clients to about 70 requests per minute, whilst Yahoo’s Search API limits you to 5000 requests per IP per day.

These numbers might seem high but a popular service can easily exceed them so ensure that you have adequate caching and throttling in place to minimise load - just as you would with your own database.

Security Is Job Number One

If you’re developing a web application then you know you can’t trust the data input by your users. You must strip it, sanitise it, scrub it and validate it to prevent those naughty hackers from playing havoc with your servers and disrupting your legitimate users.

The same applies to third party APIs. Whilst the likes of Google, Yahoo, FriendFeed et al are trustworthy, you can’t trust their users. Service providers do their best to ensure that the data they serve to you is clean but there can always be another backdoor or unpatched security hole, so validate and clean any data that comes from a RESTful API.

Subscribe to relevant security mailing lists as well as developer mailing lists for the APIs that you are using and be prepared to patch your application quickly. Get process in place to deal with security issues before it is needed.

Using REST Interfaces From Server-Side Code

So, you’ve chosen your API and your development language and are ready to start designing and coding your application. Where to start?

Re-Use An Existing Wrapper

The first stop I would recommend is to look for third party interface libraries in your language of choice:

• Perl coders should check CPAN
• PHP users should check PEAR
• Python wranglers look to PyPI
• Ruby hackers probably already know about RubyGems

Popular APIs often have third party wrappers available which will do a lot of the heavy lifting for you, like these libraries for the delicious.com API:

• Perl
• PHP
• Python
• Ruby

Almost every modern scripting language has a large repository of open-source libraries available, so if I didn’t mention your language of choice don’t despair, Google around and let us know about your findings in the comments.

Many interfaces are not 100% REST compliant or have other quirks so using a third-party wrapper will insulate you from having to know too much about the API. They usually have good community support and frequent updates too.

Roll Your Own

If you can’t find a wrapper library for the API you want, you’ll need to go in at a lower level, preparing the HTTP connection, sending the request and parsing the response. I’m not going to provide examples in every language because there are already some great articles explaining just how to do this:

• Dive Into Python has a great tutorial on calling calling web services over HTTP.
• Perl hackers just need to use LWP and then parse the response in XML or JSON.
• There’s a nice tutorial on using PHP to call REST resources by George Papayiannis.
• Similarly, for Ruby coders, here’s a lovely example of writing a a RESTful API wrapper by ActsAsFlinn.

A good place to find further examples and tutorials is the delicious.com search engine.

Server-Side Designs

Think carefully about the design of your server-side code, here are some ideas that you might want to take into consideration:

1. You might want to switch service providers (from del.icio.us to diigo, for example) so abstract your calls to third party services.
2. Handle failures gracefully. If a third-party service provider fails, it should not make your application look bad.
3. Think about timeouts, how long should you wait for a response? Can you send feedback to your users whilst they wait?
4. Record all success/failure rates. You’ll need to examine this data for trends.
5. Can you use multi-threading or asynchronous network calls to improve the response time of your application?
6. You can use XML schema definitions to validate XML responses, but beware that this gets pretty slow for complex interfaces.

Using REST Interfaces From Client-Side Code

Client-side coding generally means just one language - Javascript. As with server-side coding, you can find a lot of Javascript wrappers to add the output of a RESTful webservice to your web pages. One example of this is Google’s AJAX Search API.

Sending The Request

Using a RESTful API from Javascript is complicated by the same origin policy of modern web browsers which seeks to prevent XSRF by ensuring that Javascript from one domain cannot modify a document from another domain. This means that trying to POST to the del.icio.us API from your webpage (not hosted at del.icio.us) will always fail unless you do it with code loaded from del.icio.us via a <script> tag.

If, however, you have control of the domain which provides the service, then you can use an XMLHttpRequest object to send your HTTP request to the server. You can send GET, POST, PUT and DELETE requests using this object.

Note that XMLHttpRequests are asynchronous. That means that your code will not wait until it receives a response from the server, it just carries straight on. You need to provide a callback that will handle the response some time in the future. This is a common stumbling block for people starting out with AJAX.

Handling The Response

RESTful interfaces typically send their responses as either XML or JSON. Javscript can handle both although JSON is the easier of the two to work with.

In the case of an XML response, you can grab the document’s DOM object from the XMLHttpRequest’s responseXML member. From there you can use DOM navigation to access child or sibling nodes, jsut as you would with an HTML document, or you could even use XML Stylesheets to transform the document into the format of your choice.

JSON responses are easy to work with since they can be interpreted into a Javascript object but don’t be tempted to just eval() the response text - remember “security is job number one”.

Instead, ensure that invalid characters are not accepted by using a parsing function like this:

String.prototype.parseJSON = function () {
    try {
      return !(/[^,:{}\[\]0-9.\-+Eaeflnr-u \n\r\t]/.test(
        this.replace(/"(\\.|[^"\\])*"/g, ''))) &&
        eval('(' + this + ')');
    } catch (e) {
      return false;
    }
};

End Of Part Four

Phew, we covered quite a lot during this post but still there was hardly any code in there. I recommend that you have a good look through the links for code and examples that apply to your project, then sit down and do some design before you start coding your own projects.

You’ll probably have noticed that being a client of a RESTful API is very much like being a client of any published API - don’t trust the input, handle errors gracefully, etc - so there wasn’t too much influence from the RESTful side of things. That will all change in the next post, when we come to designing RESTful interfaces.

As always, feel free to ask questions below and thanks for reading.

---

Creative Commons licensed photos by Nat W and davestfu respectively.

Continuing our look at Representational State Transfer interfaces, part three lets us get our hands dirty with some real-life RESTful interfaces from BrightKite and del.icio.us.

But first...

What Is A RESTful Interface?

Three articles into the series and I finally get to explain this, thanks for sticking around!

REST stands for Representational State Transfer, a term first coined in Roy Fielding’s PHd Thesis. It involves using HTTP to manipulate resources identified by URLs via a completely stateless protocol.

The URLs used are generally quite pretty. For example, an e-commerce store might sell a product identified by this URL:

http://mystore.com/products/1234

The store might have an API that supplied users with a list of products by performing a HTTP GET request on this URL:

http://mystore.com/products

The response might be supplied in an XML document, like this:

 <?xml version="1.0"?>
<products>
    <product id="1">
         <url>http://mystore.com/products/1<url>
         <name>Red widget<name>
         <description>A large, left-handed widget<description>
         <stock-level>406<stock-level>
    <product>
    <product id="2">
         <url>http://mystore.com/products/2<url>
         <name>Blue widget<name>
         <description>A small, left-handed widget<description>
         <stock-level>12<stock-level>
    <product>
<products>

Stateless Protocol

I mentioned that REST is a stateless protocol, so how can that work in an e-commerce context? People need to browse around the site and fill up their shopping basket!

Simple, a shopping basket is a resource too. Create a basket with a POST request to this URL:

http://mystore.com/baskets

And add items with POSTs like this:

http://mystore.com/baskets/1/products/2

You can use HTTP authentication to ensure that only the basket owner can view or manipulate the contents of the basket.

Enough of fabricated example shops, let’s look at some RESTful APIs that are being used on the internet today...

Delicious

One of the first mainstream web applications to offer a RESTful interface was del.cio.us with its version 2 API. We’ll be using the del.icio.us API for this example, so if you’re following along be sure to get a del.icio.us account and download CURL, the tool that we’ll be using.

We’ll begin by using the API to get a list of tags that we’ve applied to our bookmarks. Open a command line terminal and type the following (changing username & password to your values):

curl --user username:password https://api.del.icio.us/v1/tags/get

This executes an authenticated GET request for your set of tags and the output will look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<tags>
  <tag count="1" tag="Action"/>
  <tag count="1" tag="Bookmarklet"/>
  <tag count="1" tag="Day"/>
  <tag count="8" tag="EC2"/>
  <tag count="1" tag="GTD"/>
  <tag count="14" tag="IE"/>
  <tag count="14" tag="Maps"/>
  <tag count="1" tag="On"/>
  <tag count="3" tag="yahoo"/>
  <tag count="1" tag="youtube"/>
<tags>

Pretty cool, no?

Now, let’s examine the REST API to add a new bookmark:

curl --user username:password -d "?url=http%3A%2F%2Fwww.mmmeeja.com%2Fblog%2F&tags=webdev%20web%20design" https://api.del.icio.us/v1/tags/add

The command above will perform a HTTP POST to add this blog :) to your del.icio.us account. Note that the url parameter is URL encoded and the tag parameter has spaces replaced by %20.

The observant reader will note that there’s something fishy about the two del.icio.us URLs we just used. The first ended in /get and the second in /add - this goes against the whole idea of using HTTP methods for REST! Yes, delicious have compromised by allowing people to use HTTP GET to create resources and the only way to do that is to break the HTTP model.

This is very common and I understand why they did it, but it’s a shame, especially when I’m using their API to teach about RESTful interfaces. Roy Fielding successfully argues that a resource should only ever have one URL no matter whether the API changes.

So, how about another API that’s a bit more compliant?

BrightKite

I had to search for a long time to find a well-known service that implemented to truly RESTful API, so hats off to BrightKite.

If you’ve not got a BrightKite account already, you might be out of luck because its invite only at the moment, but there are plenty of invites around if you know where to look.

The BrightKite API documentation explains that people, places, notes, photos etc are all resources available via their API. Placenames are rarely unique and unambiguous so places are assigned UUIDs, so the URL for City Square, in Leeds is:

http://brightkite.com/places/b1adc0a0b65e11dd8a90003048c0801e

That URL takes you to the web page, but you can get the information in XML format by performing a HTTP GET of the URL with “.xml” on the end:

curl http://brightkite.com/places/b1adc0a0b65e11dd8a90003048c0801e.xml

You should be able to access that part of the BrightKite API without an account, so try it and check out the results. You can also get the results in JSON format by substituting “.json” for “.xml”.

Posting Notes

Places can have notes associated with them, get a list of notes attached to City Square like this:

curl http://brightkite.com/places/b1adc0a0b65e11dd8a90003048c0801e/notes.xml

In correct RESTful fashion, we can create a note with an HTTP POST. We must be an authenticated user to be allowed to do this, so replace the username and password fields in the command below:

curl -u username:password -X POST http://brightkite.com/places/b1adc0a0b65e11dd8a90003048c0801e/notes -dnote[body]=My%20lovely%20note

Finally, We Get Some REST!

That was part three of the series, only two more to go. I hope that it’s been worth getting up to speed on URLs and HTTP before we got into the REST examples.

In the next installment, I’ll cover using RESTful interfaces from software. Both client-side javascript and server-side scripts will be featured, so don’t forget to subscribe.

---

Creative Commons licensed photo by Bekathwia.

HTTP (HyperText Transfer Protocol) defines how resources get transfered across the web. In the previous article on URLs, I explained that there are other protocols but HTTP is the most important when it comes to RESTful interfaces.

HTTP Methods

When a client (like your web browser) asks for a resource from a web server, it uses one of:

• GET
• PUT
• POST
• DELETE
• HEAD

HTTP GET

This method means “I’d like the resource, please” The client or the server can override the request and grab a copy from a cache.

I’m going to say that again the client can override the request and grab a copy from its cache. That means no network activity, the server doesn’t even know that the resource was requested.

HTTP PUT & POST

Both these can mean “here is a resource for you” They could be an update to an existing resource or create a new one, the HTTP standard does not specify which. We will see later on that in RESTful interfaces a PUT usually means update, whilst POST means create.

POST is commonly used in web forms, so when you buy an item from Amazon or bid on an Ebay auction, you are sending a POST request. They should never be cached.

HTTP DELETE

I bet you can guess what this does!

HTTP HEAD

This is a more interesting method, it means “tell me about the resource, but don’t bother sending the resource” It can be used to check whether a resource exists (by checking the response code), the type of the resource or how old it is (by check the headers).

HTTP Response Codes

The server will answer a HTTP request with a code that indicates whether the request could be processed. I’m sure you know some of these code already, like the dreaded 404!

There’s quite a bunch of them, so here are a select few that are applicable to RESTful APIs:

• 200 OK
• 201 Created
• 202 Accepted
• 304 Not Modified
• 400 Bad Request
• 401 Unauthorized
• 403 Forbidden
• 404 Not Found
• 405 Method Not Allowed
• 406 Not Acceptable
• 409 Conflict
• 410 Gone
• 411 Length Required
• 412 Precondition Failed
• 413 Request Entity Too Large
• 414 Request-URI Too Large
• 415 Unsupported Media Type
• 500 Internal Server Error
• 501 Not Implemented
• 503 Service Unavailable

I’ll explain how these might apply to a web service in a later article, when we cover REST in more detail.

HTTP Headers

Both requests and responses can have extra headers attached. If you’ve done any web development you’re bound to know a few common headers (like UserAgent and Referer) already but there are many more that are useful when we’re dealing with APIs.

Request Headers

A classic use of a request header comes about when a resource can be represented in many forms, such as a product description available as a HTML page, XML document, JSON or a photo. In this case, a client might send its GET request with an Accept header indicating that it wants JSON, like this:

Accept: text/json

Another useful request header can be used to get resources that have been updated recently:

If-Modified-Since: Sat, 25 Oct 2008 19:43:31 GMT

Response Headers

Response headers contain meta-data about the requested resource. They can explain what format the data is (Content-Type), how long the data is valid for (the Expires header) or when it was last written (Last-Updated).

Don’t forget that you can use the HEADERS method to get just the response code and headers back from the server.

End Of Part Two

Hopefully you now have a decent grasp on the magic protocol behind the web. I find it really helps if you can think in terms of objects or resources instead of pages, then you’ll get a better feel for exactly what your browser is doing when you ask for a URL.

As always, feel free to ask questions via the comments, I’ll do my best to answer.

This post is part of a series on REST so if you’ve found it useful, subscribe to catch the other articles in the series.

I am writing a series of articles on RESTful web services but some research showed that a surprising number of people don’t understand the basics of what a URL is. So, in the interest of learning to walk before we can run, I’ll explain what a URL is and how the various components are used in modern web design.

URL stands for Uniform Resource Locator, as defined by this W3C standard and a URL identifies a unique item on the web. The item might be a web page, an image, a database item (like an ebay auction), an MP3 file or anything that can be represented on the web.

An item may have several different URLs, but each URL only points to one thing - although that thing may change over time.

A typical URL might look like this:

http://www.example.com/directory/page.html?param1=value1&param2=another%20param

That’s a pretty complex example, so let’s break it into its constituent components...

Protocol

The bit before the colon is the URL protocol, http in our case.

This tells the web browser how to talk to the server - how to ask for a resource, what will happen if the resource does not exist and so on.

There are several URL protocols available, but four are most common on the world wide web today:

• http- HyperText Transfer Protocol is the protocol used by web servers. The page you are reading now was delivered via HTTP.
• https - Secure HTTP is just the same as normal HTTP except that the transmissions are encrypted. If you enter passwords or your credit card details on a web site, you want to ensure that this protocol is used by checking for a padlock icon in your web browser.
• mailto - This protocol allows for clickable email addresses.
• ftp - File Transfer Protocol is used to manipulate files over the internet.

Double Slash

Our example has a double slash (//) after the protocol. This indicates that the URL is an absolute URL - that it does not need any context to resolve to a unique resource.

The opposite of an absolute URL is a relative URL. It makes no sense to type a relative URL into your browser’s address bar, but embedding one in a web page would indicate that the resource can be found relative to the URL of the page. If you want to know more about relative URLs, this article on UNIX relative paths should help, as relative URLs follow the same standards.

Domain Name

The next bit of an absolute URL is the domain name, www.example.com in the example. This identifies a computer (or cluster of computers) on the internet that stores the resource you want. Domain names are not case sensitive so www.example.com and WwW.eXAMplE.COM are equivalent.

Path

The URL path tells the server how to find the resource that you require. The path in our example URL is directory/page.html. Unlike domain names, paths are case sensitive.

You might think that the .html indicates that the resource is a HTML file but that is not necessarily true! Your web browser will check the contents of the file to determine what kind of resource it is and how it should handle it and if you are writing any code that downloads from the internet, you should do the same.

CGI Parameters

The end of our URL has two parameters, param1 and param2.

Parameters are optional for all URLs and their presence is indicated by the question mark (?). An ampersand (&) is used to separate multiple parameters.

Parameters can also be assigned a value using an equals sign (=), as is the case in our example.

param1’s value is “value1” but param2’s value is a bit more complex - it contains a space!

URL components can only contain certain characters: A-Z 0-9 underscores and dashes. All other characters must be “URL encoded” that is translated into a percent sign (%) followed by their ASCII hex value.

ASCII translation can also be used in other parts of the URL, but it’s not recommended - can you imagine reading out the URL over the phone and saying “H-T-T-P-colon-slash-slash-A-B-percent-twenty-C” Ridiculous!

Other URL Components

There are many other URL components that you might encounter that weren’t present in our example.

Port Numbers

Some URLs add a port number to the domain name, like this:

http://www.example.com:8080/page.html

Most web servers operate on default ports, but sometimes another port might be specified. The default ports are:

• Port 80 - HTTP
• Port 443 - HTTPS
• Port 21 - FTP

Specifying a different port does not mean you can supply an incorrect protocol in the URL - trying to talk HTTP with an FTP server will fail.

Usernames & Passwords In URLs

It is possible, although rare, to specify a username and password inside a URL. In this case, the username and/or password are supplied before the domain name, like this:

ftp://username:password@hostname/

I hope I don’t have to tell you just how insecure it would be to embed a URL like this in a web page.

You Should Now Know All About URLs

That pretty much covers the basics of URLs. Feel free to experiment with your own web spaces - there’s nothing that can go wrong with asking a webserver for a resource via a URL. Ask questions in the comments too, I’ll do my best to answer.

This post is part of a series on REST so if you’ve found it useful, subscribe to catch the other articles in the series.

---

Creative Commons licensed photos by Laughing Squid and dailyinvention.