gilt-tech:

We’re excited to introduce the latest open source project by Gilt Principal Systems Engineer (and Docker talker) Jonathan Leibiusky: Shmock, an HTTP mocking library based on Express. If you’re not familiar with Express, it’s an open-source, Sinatra-inspired web development framework for…

APIdays: The perspective that matters


This post is more or less the transcript of my presentation at the Mediterranean APIdays conference in Barcelona, May 2014 and the 1st API Meetup in Athens, June 2014.

The perspective that matters when designing an API is the perspective of the client.

Being in the position of the client, would make you recognize that there are two important rules to validate your API design:

  1. API design should honor our business. It is an interface that should be giving the same exact product “value” that a user interacting with a UI gets.
  2. API should be rich in hypermedia controls in the same sense our (web) UI is.

API

There are two cases that we can find ourselves building an API for:

  1. There is an existing product which is already functional and we need to build an API on top of this application.
  2. Starting a new product and being “API first” by having an API as the only way to interact with the application.

problem-solving mode

In the second case, when building an API in parallel with the application, we are in the “problem-solving” mode. We are focused on building the application, not concerned about the interface our clients will interact with.

What data structures and algorithms do I need for the job, and what input and output parameters are necessary to get it done?

thinking in db tables

In the 1st case, when all the application functionality exist, we are biased on how our API should look like. We would find ourselves stuck in “thinking in database tables”. Our perspective of what our product is, maps to database table rows:

+-----+----+--------+--------+-----+--------+
| ... | id | attr_1 | attr_2 | ... | attr_n |
+-----+----+--------+--------+-----+--------+

Adopting this perspective results in:

  • An API design poor in business value
  • Thinking all we have to do is mapping database CRUD operations to POST, GET, PUT, DELETE.
  • Us being happy with a result similar to this:
{
    "id": "...",
    "attr_1": "...",
    "attr_2": "...",
    //...
    "attr_n": "..."
}

Interface

API design, simply put, is all about designing an interface.

During an “API First” project (deskdonkie.com) I’ve worked for the last couple of months, we picked up a standard before a single line of our client was written. By the time we started working on an angular application that would use the API, we realized there is no Angular nor vanilla js client for this standard. We did not have the resources to build a client for this standard at the time and on the other side angular comes with simple tools to consume an API (see Angular $resource).

What we end up doing was altering our API design in order to fit the needs of the client. The client had, at the time, the power to dictate the interface.

So, who are the clients that would consume our API?

In the book RESTful Web APIs by Leonard Richardson, Mike Amundsen, Sam Ruby there is a chapter called “Kind of API clients”, which was mind opening for me. The authors divide the API clients in 2 groups.

Human-driven clients

They present representations to a human being, and convey the human’s decisions back to the server.

source: RESTful Web APIs (O’Reilly)

Meaning a human interacts with this client (through a UI) in order to solve the problem or achieve a goal. Mobile applications, our javascript rich web application etc. are human-driven clients.

Automated clients

As the book says:

Automated clients receive representations but don’t render them. There’s no human to see the rendering.

source: RESTful Web APIs (O’Reilly)

And continue by dividing the clients in 4 sub-categories.

monitor

It simulates a human who’s obsessed with one particular web page. Give it a URL to start with, and the monitor will fetch a representation of that URL and process it somehow. But it won’t follow any links.

source: RESTful Web APIs (O’Reilly)

Monitor would probably be a client if your product is an exchange rates service.

script

A script simulates a human with a set routine that never changes. A script happens when a human is tired of this routine and wants to automate it.

source: RESTful Web APIs (O’Reilly)

Predefined process to get a result. Consider creating your application’s cron jobs to use your API to achieve their scheduled tasks and you have script clients. Script clients that know “when” and “what” to do, without knowing “how” to do it.

crawler

The crawler simulates a very curious but not very picky human. Give it a URL to start with, and it will fetch a representation. Then it will follow all the links it can find to get more representations. It will do this recursively, until there are no more representations to be had.

source: RESTful Web APIs (O’Reilly)

Agent

The most interesting of the Automated clients that I would like to focus on is the Agent.

A software agent simulates a human being who is actively engaged with a problem. It’s not as smart as a human, and it has no ability to make subjective judgments, but it does what a human would do in the same situation. It looks at a representation, analyzes the situation, and decides which hypermedia control to activate to get closer to its final goal.

source: RESTful Web APIs (O’Reilly)

Think of a “smart” refrigerator that after recognizing you run out of milk, it will search for milk in some online stores through their API, pick up some choices and prompt you to select based on best price, quality etc. and finally order them for you.

We know those “smart” devices will eventually be available to almost every household because we see them in sci-fi movies.

Her

Since we talk about movies, recently I watched a movie called “her”, a love story between a human and an operating system. Samantha, the name of the operating system, is an automated-client, an Agent with superior artificial intelligence that use APIs to surf the web and find resources. Resources that would share with her loved one, like music.

So let see how some services like soundcloud and 8tracks looks like in Samantha’s world.

GET api.soundcloud.com

{
  "errors": [{
      "error_message": "404 - Not Found"
  }]
}

Simply put, soundcloud does not exist for an agent.

If there was any way to explain why an API would be designed like this, it would be the "thinking in database tables" mentality. No table that match for the resource path, no response.

GET api.soundcloud.com/tracks

Let’s say the Samantha somehow figured out that api.soundcloud.com/tracks will give her some music tracks back.

[
    {
        "kind": "track",
        "id": 150424356,
        "user_id": 92043892,
        "duration": 221851,
        "downloadable": false,
        "genre": "Twerk",
        "title": "Trey Songz - Na Na (Kush Electricity Remix)",
        //...
        "user": {
            "id": 92043892,
            "kind": "user",
            "username": "Kush Electricity",
            "uri": "https://api.soundcloud.com/users/92043892",
            //...
        }
    },
    { /* 19 more tracks... */ }
]

The response contains 20 tracks and no indication of pagination. Making an agent believe that this services has only 20 tracks to offer.

GET api.soundcloud.com/users/55884576

{
    "id": 55884576,
    "kind": "user",
    "permalink": "dmtrsflaco",
    "username": "dmtrsflaco",
    "uri": "https://api.soundcloud.com/users/55884576",
    "permalink_url": "http://soundcloud.com/dmtrsflaco",
    "description": "Alter ego of @dmtrsslvdr",
    //...
    "track_count": 4,
    "playlist_count": 0,
    "plan": "Free",
    "public_favorites_count": 186,
    "followers_count": 75,
    "followings_count": 321,
    "subscriptions": []
}

Following user.uri does not get us to any more tracks to discover. Neither controls to navigate more eg. users this user follows, tracks he liked etc.

This is the case where we do not deliver the same product value through our API as we do with our web users.

GET api.8tracks.com

Trying another service which is called 8tracks. It’s a service where you can curate music lists called “mix sets” and listen what other people have curated.


{
    "mix_cluster": {
        "pagination":{
            "current_page": 1,
            //…
            "total_pages": 1
        },
        "mixes_per_mix_set_page": 4,
        "mix_sets": [ {
            "pagination": {
                "current_page": 1,
                //…
                "total_pages": 2
            },
            "mixes": [ {
                    "user": {
                        "id": 11055544, //...
                        "path": "/users/11055544",
                    },
                    "id": 3969592,
                    "path": "/mixes/3969592",
                    "web_path": "/gang-related/gang-related",
                    "duration": 10178,
                    "tracks_count": 40
                } //...
            ],
            "path": "/explore/all/hot",
            "name": "Trending",
            "web_path": "/explore/all"
        }], //...
        "id": "home_first_time",
        "path": "/mix_clusters/home_first_time"
    },
    "status": "200 OK",
    "errors": null,
    "logged_in": false
}

8tracks exist for an agent. Although there is a sense of pagination (current/total page etc.), the “instructions” on how to navigate through pages is missing.

GET 8tracks.com/users/11055544

{
    "user": {
        "id": 11055544,
        "login": "Gang Related",
        "path": "/users/11055544",
        "web_path": "/gang-related",
        "avatar_urls": {
            //...
        }
    },
    "status": "200 OK",
    "errors": null,
    "notices": null,
    "logged_in": true,
    "api_version": 3
}

Navigating through user uri, again, unfortunately does not give any hypermedia controls to navigate further. Mix sets this user has curated or the sets this user has liked etc.

Conclusion

Next time you need to design an API, put yourself in Samantha’s shoes. That would help you get the perspective that matters the most, the perspective of the client.

Good Friday

Lately, I started reading the “Remote - Office not required” book by 37signals’ co founders. Similar to “Rework”; it’s a really interesting guide that you can always refer back to.

In the Forward motion chapter of the How to collaborate remotely section, if you hide the implementation details of remote working, you still end up with good advice:

To instill a sense of company cohesion and to share forward motion, everyone needs to feel that they’re in the loop.

In our team when we felt that the flow of information was running slowly through the office, we attempted to gave it a boost with 1~2 per 2~3 weeks team “gatherings”. Consider this event as the synch version of 37singals’ weekly discussion thread with topic “What have we been working on?”.

Everyone chimes in with a few lines about what they’ve done over the past week and what’s intended for the next week It’s not a precise rigorous estimation process, and it doesn’t attempt to deal with coordination. It simply aims to make everyone feel like they’re in the same galley and not their own little rowboat.

In terms of the authors, this is what our “Good Fridays” evening team gathering is all about. Product, customer support, developers have a discussion on the latest metrics, roadmap updates, what code was shipped and devop operations.

Simply put, progress is a joy best shared with coworkers.

PeoplePerHour team Good Friday

The Future of programming via @worrydream

It was really hard for me to realize where I could view the commit history of a git repository on github.com in 1 second, so I wasted more time. Finally, I got it, the number of commits link to the commit history, but it’s not that obvious if not obvious at all. I think what most of git users know as git log is more important than “eye candy” graphs/networks etc. What do you think?

When your API uses Oauth1.0

know no #fear #okeanos @grnet_gr

Thoughts on w̶̶e̶̶b̶ application development

“How you build your application, it is not the tools you use.”

This quote is result of discussions with other developers. Every time the “how you build your application?” question drops, I get answers like “we use $x framework”, “$y database” in some rare cases “we do tdd” is between the answers.

I will stand in the framework as it is considered to be main library to build your application. And since MVC is the monopoly out there

Model, View Controller.

Lets rethink about it for a minute, shall we?!

Model

There is a misconception that Active Record is the Model. So what is Active Record if not the model of the application:

The interface of an Active Record object would include functions such as Insert, Update, and Delete, plus properties that correspond more or less directly to the columns in the underlying database table.

Using an active record as you model, dominate the shape of your code. Now the persistence logic is all over your application code. In the case of an application a model would be a Beer object. If I asked you to create a Beer Class would you ever thought about a function name like save(), insert(). But your code is bloated and you end up having the database in the middle of your application, this make transition to other type of persistence more hard. There other patterns out there…

Before moving on a note on the Beer object we mentioned before. A Beer object has different values and functionality based on the place it belongs example: in the factory, during the transportation and finally at a bar.

Controller

Controller here is to define the context. Based of context he uses the appropriate models. Its use case, a scenario. Its not Create Read Update Delete. Who is talking to your application controller? A Mobile Client A Desktop Client A Web Client … a.k.a. a browser. Since we are talking about web let’s be clear about it. There is no such thing as a “user” in your application. Your web/mobile/desktop client may have a user but your application only accepts requests to do magic. Parameters of your request should state if the action should be done or not. /myBasket/view /basket/41491/view?customer_id=1943&secret=? The request and the result have 1-1 relation. The same url must always give the same result no matter what. Hints: Reverse caching. This same result should be predictable this mean your controller should be testable. It’s is an object in the of the day isn’t it ? And the http protocol it is just a detail.

View

Speaking of results, what a result could be except a representation of your application models in variety of states during their lifecircle. A view of your application should not contain html. Html is used by browser to render a web page. Your models data may appear on that page but they are not the page. You can always use XML if you like tags or if your web client needs to navigate with xpath. What is the tricky part in a view like more than encoding my objects to json? Your applications navigation. Your clients should be ignorant about your applications urls. Every response from your application should contain paths to related application calls the same way a web site provide links to related resources.

A really “interesting” finding from the youtube UI, is the “Show more” functionality for the video description. It is is triggered by clicking at any part of the “description box” ( blue box on the picture, div with id watch-descirption. Liked it, because since you realize it you don’t need to “target” that small part with your mouse cursor to trigger see the full video description. Hated it, because you could need 2 clicks if you need to access a link in the description.