Implementing API Gateway in F# Using Rx and Suave
One of the impressive aspects of functional programming is that it will enable you to write simple and expressive code to solve complex problems. If you are wondering, how is it possible? Well, It is because, Functional Programming provides a lot of higher level abstractions to solve the problem in hand and avoids the need of writing lot of boilerplate & plumbing code. As a developer, you will be more productive and deliver the solution faster.
In this blog post, we are going to see the application of the above statements by implementing an API Gateway Pattern in F# using Rx and Suave.
This blog post is my contribution towards fsharp advent calendar 2015. Do check out other great fsharp posts there
API Gateway Pattern
Let’s assume that you are asked to build the backend for the below GitHub user profile screen.
This profile screen has 3 components
-
User - Username and Avatar
-
Popular Repos - Top 3 Repos of the Given User (determined by the number of stars)
-
Languages being used in the corresponding repos
Now as a developer, you have two options
-
Build the Profile completely in client side by calling their corresponding GitHub APIs
This approach makes the client side complex as it involves five HTTP requests to create the complete profile screen. It has a lot of drawbacks as mentioned in this blog post by Netflix.
-
Create a Facade API which takes care of the heavy lifting and hides the complexity of calling multiple services by exposing a single endpoint which returns all the data required to create the profile screen in one API call
This approach is called API Gateway. It is one of the commonly used patterns in the microservices world.
Rx
API Gateway is normally implemented using Rx. Rx, an implementation of Functional Reactive Programming, makes it easy to implement this kind of problems by enable us to think each asynchronous operation as streams (aka Observables). These streams offer a declarative API to work with the asynchronous operations.
In the implementation that we are going to see, we will be using FSharp.Control.Reactive, an extension and wrapper for using the Reactive Extensions (Rx) with F#
Project Structure
Create a new F# Console Application Project with the name RxFsharp
and then create the files mentioned below in the given order
1 2 3 4 5 6 7 8 9 10 11 |
|
Then install the following NuGet packages
Modeling Http Request & Response as Stream
In Functional Reactive Programming(FRP) every action is treated as stream. So, first step in implementing an API Gateway is modeling each HTTP request&response as stream.
Open Http.fs
file and update it as below
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
|
The getResponseAsync
function makes use of Http.fs, an HTTP Client library for F#, to fire the HTTP GET request and returns the HTTP response asynchronously.
The HTTP response has been modeled as either Ok
with the response content for all the responses with the status code 200
and everything else is treated as Error
for simplicity.
In the final line, we create a stream (aka Observable) from asynchronous response using the function Observable.ofAsync
from FSharp.Control.Reactive
As all the GitHub API requests require user agent, we are adding one before firing the request.
Parsing and Transforming GitHub API Responses
Upon successful response from GitHub, the next step is parsing the JSON response and transforming the parsed response as per the Profile screen requirements.
To parse the response, we are going to leverage the powerful tool of FSharp called JsonTypeProvider
The JSON Type Provider provides statically typed access to JSON documents. It takes a sample document as an input. Here in our case, it is user.json and repos.json
Get the sample response from GitHub using its User API and Repos API and save the response in user.json and repos.json respectively.
With the help of JsonTypeProvider, we can now easily parse the raw JSON response to its equivalent types in just a few lines of code in GitHub.fs
!
1 2 3 4 5 6 7 8 9 10 |
|
As GitHub.fs
is an abstraction of GitHub let’s put their URLs also here in the same file
1 2 3 4 |
|
We have used partial application of function sprintf
in the userUrl
and reposUrl
function here, so both of these functions take username as its parameter implicitly
The JSON response of languages API is little tricky to parse as its keys are dynamic. The type provider will work only if the underlying response has a fixed schema. So, we can’t use the type provider directly to parse the response of languages API.
We are going to use the inbuilt JSON Parser available with JsonTypeProvider to parse the languages response
1 2 3 4 5 |
|
The JsonValue.Parse
parses the raw string JSON to a type called JsonValue
and the JsonExtensions.Properties
function takes a JsonValue
and returns the key and value pairs of all the properties in the JSON as tuples. As we are interested only in the Keys here, we just pluck that value alone using the function fst
Great! Now we are done with parsing the JSON response of all the three APIs and creating it’s equivalent types. The next step is doing some business transformation
One of the requirements of the Profile Screen is that we need to show only the top 3 popular repositories based on the stars received by the repository. Let’s implement it
1 2 3 4 5 6 7 8 9 |
|
In the Http.fs
file, we defined how to get the raw JSON response using Stream as HttpResponse
(Ok
& Error
) and in this GitHub.fs
, we have defined how to parse this raw JSON response and transform them its equivalent strongly types.
The final step is integrating both these logic and return the Output record type needed by the Profile Screen
let’s start by defining the Profile type
1 2 3 4 5 6 7 8 9 |
|
Then write functions to get the value from HttpResponse
and create this profile output.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
All the above functions except toProfile
take HttpResponse
as its last parameter implicitly.
In reposResponseToPopularRepos
function, if the repos API request is successful, we parse the response to its equivalent type and then pick only three of them based on star count and in the case of an error we just return an empty array.
The languageResponseToRepoWithLanguages
function handles the response from the Languages API request which takes its associated repository as its first parameter. If the response is successful, then it creates the Repository
record with the returned languages else it just the Repository
record with an empty array for languages.
The last function toProfile
is a merge function which takes a tuple of HttpResponse
(of User API request) and Repository []
and creates a Profile
record if the response is successful. In case of an error, it just returns None
Note: To keep this blog post simple, I am handling the errors using empty arrays and None. It can be extended using ROP
Implementing API Gateway
Let me quickly summarize what we have done so far. We have created two abstractions.
Http
- Responsible for firing the HTTP GET request with the given URL and give the response as a Rx Stream
1 2 3 |
|
GitHub
- Takes care of parsing the JSON response from GitHub API and does some business logic (finding top 3 popular repositories). Then returns the output in the format that the Client needs.
With the help of these abstractions now we are going to build the API Gateway to get the profile object.
Open Gateway.fs
and add the getProfile
function
1 2 3 4 5 6 |
|
It is just a skeleton which just describes the what function does. Let’s start its implementation.
In Rx world, everything is a stream. So, the first step is converting GitHub User API request to stream
1 2 3 4 |
|
We have created the userStream
using the userUrl
and asyncResponseToObservable
function defined earlier.
1 2 3 |
|
Then we need top three popular repositories as a stream
1 2 3 4 5 6 7 8 |
|
Except the last line, everything is same as that of creating userStream
. If you check GitHub repos API it just returns all the repos associated with the given username. But what we need is only top three of them based on the number of stars it has received.
We already have a function reposResponseToPopularRepos
in GitHub.fs
which does the job of picking the top three repos from the raw JSON. As the response is in the form Rx stream, we need to get the value from the stream and then we need to apply this function and that’s what the Observable.map
does. It is very similar to Array.map and LINQ Select
1 2 3 4 5 6 |
|
The next operation is a little complex. i.e. finding out the programming languages being used in these popular repositories. To get this GitHub API, we need to fire three separate requests to the Languages API for each repository and then merge the results back with the corresponding repository.
To implement this with the help of Rx streams, we need to understand the flatMap
function of Rx.
It is similar to the map
function that we have seen before with a difference that it takes a function that returns a new item stream instead of a new item.
Map
The map
function has the following signature
1
|
|
FlatMap
The flatMap
function has the following signature
1
|
|
This function is also called as bind
function in the functional programming world. If you would like to know further on this topic, I strongly recommend this blog series by the fsharp great, Scott Wlaschin
Back to the problem in hand, we need to fire three HTTP GET Requests to get back the languages associated with the each of the top three popular repos. In terms of the flatMap
function, it boils down to three functions of the following syntax
1 2 |
|
We can implement the solution using three flatMap
functions using the above syntax. But, we can make it more granular by creating a new variant of flatMap
function to achieve this in a more straightforward way.
The ideal function that we are looking for in this flatMap
variant holds the following signature
1
|
|
i.e three flatMap
can be rewritten as
1 2 |
|
Let’s name it as flatMap2
and add the implementation of it in the file ObservableExtensions.fs
1 2 3 4 5 6 7 |
|
Here is the representation of what this function does
1 2 3 4 5 6 7 8 9 10 11 |
|
It’s hard to get it right in a single shot. So, let’s see it in detail step by step by applying it to our use case here
- We have the stream of three popular repos (i.e array of repos)
1
|
|
- To get the languages associated with the each repo we need to call the languages API for every repo item in the above step
1 2 3 4 5 6 |
|
- After we received the response from each languages API call, we need to merge them into one stream
1 2 3 4 5 6 7 |
|
- To integrate this with the expected application response, we need all the responses in a single go
1 2 3 |
|
Great! You got it right!!
Let’s use this flatMap2
function and complete the implementation of profile API gateway.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
|
The flatmap2
function takes the function toRepoWithLanguagesStream
which converts the GitHubUserRepos.Root
type to IObservable<Repository>
to find out the languages associated with the given popular repositories.
The toRepoWithLanguagesStream
function does the following
1 2 3 4 5 |
|
The Observable.map
function takes only one input, but here we need to two inputs. So, With the help of partial application, we created an intermediate function by partially applying the first parameter alone
1
|
|
The languageResponseToRepoWithLanguages
function has been already defined in the GitHub.fs
file.
The last step of the getProfile
function is combining this popularReposStream
with the userStream
created earlier and return the Profile
type asynchronously.
1 2 3 4 5 6 7 8 9 10 |
|
The Observable.zip
function takes two streams as its input, then merges the output of each stream and return the output as a tuple. From this tuple, we have used Observable.map
function to map it a Profile
type using the toProfile
function created earlier in GitHub.fs
1 2 3 4 5 6 |
|
The last functions TaskObservableExtensions.ToTask
and Async.AwaitTask
does the conversion of IObservable
to async
by converting it to a Task
first and then the Task
to async
The final getProfile
function will be like
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
|
This function is a testimonial on How functional programming helps to write less, robust, and readable code to solve a complex problem
We have handled all the five HTTP requests asynchronously, did some error handling (by returning empty types), and finally efficiently combined outputs of these five HTTP requests and created a type to send back to the client. Everything is asynchronous!
Pretty awesome isn’t it?
Exposing the API
The final step is exposing what we have done so far as an API to the outside world. We are going to implement this using Suave
Open ApiGateway.fs
file and update it as below
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
|
The JSON
is a utility function (WebPart in the world of Suave) which takes any type, serialize it to JSON format and return it as JSON HTTP response.
The getProfile
function is the API WebPart which calls our backend API gateway implementation and pass the received response to the JSON
WebPart defined before.
In case if there is no profile available (Remember? we return empty types in case of errors), we just return 404
with the message that the given username is not found.
Then update the Program.fs
to write the web server code
1 2 3 4 5 |
|
Thanks to Suave
for it’s lightweight and low-ceremony offerings in creating an API. We just exposed it in two lines!
Hit F5
and access the API at http://localhost:8083/api/profile/{github-username}
Bingo!!
Summary
A language that doesn’t affect the way you think about programming is not worth knowing - Alan Perils
The above quote summarizes the gist of this blog post. Functional Programming will help you think better. You can get the source code associated with the blog post in my blog-samples GitHub repository
Wish you a happy and prosperous new year