Comments

In my previous blog post, we have seen how interfaces in golang can help us to come up with a cleaner design. In this blog post, we are going to see an another interesting use case of applying golang’s interfaces in creating adapters!

Some Context

In my current project, we are using Postgres for persisting the application data. To make our life easier, we are using gorm to talk to Postgres from our golang code. Things were going well and we started rolling out new features without any challenges. One beautiful day, we came across an interesting requirement which gave us a run for the money.

The requirement is to store and retrieve an array of strings from Postgres!

It sounds simple on paper but while implementing it we found that it is not straightforward. Let me explain what the challenge was and how we solved it through a Task list example

The Database Side

Let’s assume that we have database mytasks with a table tasks to keep track of the tasks.

The tasks table has the following schema

1
2
3
4
5
6
CREATE TABLE tasks (
  id SERIAL PRIMARY KEY,
  name TEXT NOT NULL,
  is_completed BOOL NOT NULL,
  tags VARCHAR(10)[]
)

An important thing to note over here is that each task has an array of tags of type varchar(10).

The Golang Side

The equivalent model definition of the tasks table would look like the following in Golang

1
2
3
4
5
6
type Task struct {
  Id          uint
  Name        string
  IsCompleted bool
  Tags        []string
}

The Challenge

Everything is set to test drive the task creation.

Let’s see what happens when we try to create a new task!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
package main
import (
  "fmt"

  "github.com/jinzhu/gorm"
  _ "github.com/jinzhu/gorm/dialects/postgres"
)

func panicOnError(err error) {
  if err != nil {
    panic(err)
  }
}

func CreateTask(db *gorm.DB, name string, tags []string) (uint, error) {
  newTask := &Task{Name: name, Tags: tags}
  result := db.Create(newTask)
  if result.Error != nil {
    return 0, result.Error
  }
  return newTask.Id, nil
}

func main() {
  db, err := gorm.Open("postgres",
    `host=localhost 
      user=postgres password=test
      dbname=mytasks 
      sslmode=disable`)
  panicOnError(err)
  defer db.Close()

  id, err := CreateTask(db, "test 123", []string{"personal", "test"})
  panicOnError(err)
  fmt.Printf("Task %d has been created\n", id)
}

When we run this program, it will panic with the following error message

1
panic: sql: converting Exec argument $3 type: unsupported type []string, a slice of string

As the error message says, the SQL driver doesn’t support []string. From the documentation, we can found that the SQL drivers only support the following values

1
2
3
4
5
6
int64
float64
bool
[]byte
string
time.Time

So, we can’t persist the task with the tags using this approach.

Golang’s interface in Action

As a first step towards the solution, let’s see how the plain SQL insert query provides the value for arrays in Postgres

1
INSERT INTO tasks(name,is_completed,tags) VALUES('buy milk',false,'{"home","delegate"}');

The clue here is the plain SQL expects the value for array as a string with the following format

1
'{ val1 delim val2 delim ... }'

double quotes around element values if they are empty strings, contain curly braces, delimiter characters, double quotes, backslashes, or white space, or match the word NULL. Double quotes and backslashes embedded in element values will be backslash-escaped. - Postgres Documentation

That’s great! All we need to do is convert the []string to string which follows the format specified above.

An easier approach would be changing the Tags field of the Task struct to string and do this conversion somewhere in the application code before persisting the task.

But it’s not a cleaner approach as the resulting code is not semantically correct!

Golang provides a neat solution to this problem through the Valuer interface

Types implementing Valuer interface are able to convert themselves to a driver Value.

That is we need to have a type representing the []string type and implement this interface to do the type conversion.

Like we did in the part-1 of this series, let’s make use of named types by creating a new type called StringSlice

1
type StringSlice []string

Then we need to do the type conversion in the Value method

1
2
3
4
5
6
7
8
func (stringSlice StringSlice) Value() (driver.Value, error) {
  var quotedStrings []string
  for _, str := range stringSlice {
    quotedStrings = append(quotedStrings, strconv.Quote(str))
  }
  value := fmt.Sprintf("{ %s }", strings.Join(quotedStrings, ","))
  return value, nil
}

Great!

With this new type in place, we can change the datatype of Tags field from []string to StringSlice in the Task struct.

If we rerun the program, it will work as expected!!

1
Task 1 has been created

Filter by tag

Let’s move to the query side of the problem.

We would like to get a list of tasks associated with a particular tag.

It’d be a straightforward function that uses the find method in gorm.

1
2
3
4
5
6
7
8
func GetTasksByTag(db *gorm.DB, tag string) ([]Task, error) {
  tasks := []Task{}
  result := db.Find(&tasks, "? = any(tags)", tag)
  if result.Error != nil {
    return nil, result.Error
  }
  return tasks, nil
}

Then we need to call it from our main function

1
2
3
4
5
6
7
// ...
func main() {
  // ...
  tasks, err := GetTasksByTag(db, "project-x")
  panicOnError(err)
  fmt.Println(tasks)
}

Unfortunately, if we run the program, it will panic with the following error message

1
2
3
panic: sql: Scan error on column index 3: unsupported Scan, storing driver.Value type []uint8 into
type *main.StringSlice; sql: Scan error on column index 3: unsupported Scan,
storing driver.Value type []uint8 into type *main.StringSlice

As the error message says, the SQL driver unable to scan (unmarshal) the data type byte slice ([]uint8) into our custom type StringSlice.

To fix this, we need to provide a mechanism to convert []uint8 to StringSlice which in turn will be used by the SQL driver while scanning.

Like the Valuer interface, Golang provides Scanner interface to do the data type conversion while scanning.

The signature of the Scanner interface returns an error and not the converted value.

1
2
3
type Scanner interface {
  Scan(src interface{}) error
}

So, it implies the implementor of this interface should have a pointer receiver (*StringSlice) which will mutate its value upon successful conversion.

1
2
3
func (stringSlice *StringSlice) Scan(src interface{}) error {
  // ...
}

In the implementation of this interface, we just need to convert the byte slice into a string slice by converting it to a string (Postgres representation of array value) first, and then to StringSlice

1
[]uint8 --> {home,delegate} --> []string{"home", "delegate"}

After successful conversion, we need to assign the converted value to the receiver (*stringSlice)

1
2
3
4
5
6
7
8
9
10
11
12
func (stringSlice *StringSlice) Scan(src interface{}) error {
  val, ok := src.([]byte)
  if !ok {
    return fmt.Errorf("unable to scan")
  }
  value := strings.TrimPrefix(string(val), "{")
  value = strings.TrimSuffix(value, "}")

  *stringSlice = strings.Split(value, ",")

  return nil
}

That’s it. If we run the program now, we can see the output as expected.

1
2
[{2 schedule meeting with the team false [project-x]}
  {3 prepare for client demo false [slides project-x]}]

Summary

In this blog post, we have seen how we can make use of Valuer and Scanner interfaces in golang to marshal and unmarshal our custom data type from the database.

The source code can be found in my GitHub repository

Comments

In my previous blog post on using golang in production, I have mentioned that interfaces are my favorite feature in golang.

As a follow-up of this comment, I would like to share how we are using (my current project is also in golang!) the interfaces to keep our code clean and consistent through a series of three blog posts

This blog post series assumes that you are familiar with the basics of interfaces in golang. If would like to know what it brings to the table, I strongly recommend to check out this well-written article by Yan Cui.

Let me start the series with a solution that we have implemented a few days back.

Some Context

The product that we are building consists of a suite of web applications. To authenticate the users, these web applications talk to a centralized web application called “Identity Server”.

Upon receiving valid login credentials, the Identity Server generates a JSON Web Token(JWT) with the corresponding user claims and signs it using a public/private key pair using RSA.

The downstream web applications will then use this JWT to grant the access to their corresponding protected resources.

Let’s assume a simple JWT claims payload

1
2
3
4
5
{
  "sub": "1234567890",
  "name": "John D",
  "admin": true
}

The above payload uses one of the JWT standard claims, sub, to communicate the unique identifier of the user in the product for all the web applications. The name and admin are the custom claims.

As per the JWT spec, the sub claim is a case-sensitive string containing a StringOrURI value which is locally unique in the context of the issuer or be globally unique, but in our system, the unique identifier of a user is an unsigned integer(uint) type.

As we will be sharing the JWT with other third party applications, we made a call to stick to the JWT spec and converted the uint to string type while generating the token and did the reverse while authenticating the user using this token.

Update - After writing this blog post, I came to know from the comment that the use case of this blog post, unmarshalling a JSON string to uint type can be done by adding string to the json tag. Being said that, if you’d like to know about how to use an interface to solve it, the rest of the post would help.

Unmarshalling JWT - A Naive Approach

Before witnessing the golang interface in action, let’s see a naive implementation how we can unmarshal the claim and use.

The straightforward thing would be creating a struct matching the properties of the claim

1
2
3
4
5
type UserJwt struct {
  Sub    string
  Name   string
  Admin  bool
}

and unmarshalling using the json package in golang

1
2
3
4
5
6
7
8
9
claims := `
{
  "sub": "1234567890",
  "name": "John D",
  "admin": true
}`

var userJwt *UserJwt
err := json.Unmarshal([]byte(claims), &userJwt)

To convert the sub from string to uint, we can have a method Id on the UserJwt type.

1
2
3
4
5
6
7
func (u *UserJwt) Id() (uint, error) {
  v, err := strconv.ParseUint(u.Sub, 10, strconv.IntSize)
  if err != nil {
    return 0, err
  }
  return uint(v), nil
}

and use it after successful unmarshalling of the JWT claims.

1
2
3
4
5
6
7
8
9
10
11
// ...
err := json.Unmarshal([]byte(claims), &userJwt)
if err != nil {
  return err
}
id, err := userJwt.Id()
if err != nil {
  return err
}
// Do something with the id of type uint
fmt.Println(id)

That’s great. But did you see any code smell here?

Let me share what’s wrong with this approach,

Let’s say that we have the following JSON claim with sub has the value user/john instead of a string representing the unsigned integer identifier of the user.

1
2
3
4
5
6
claims := `
{
  "sub": "user/john",
  "name": "John D",
  "admin": true
}`

Unmarshalling this claim will work, and it won’t return any error

1
2
3
4
5
// ...
err := json.Unmarshal([]byte(claims), &userJwt)
if err != nil {
  return err
}

We can share the unmarshalled userJwt with the rest of the code to carry the business logic.

We will come to know that the claim has an invalid sub value only when we try to get the id of the user by calling the Id method

1
2
3
4
// ...
id, err := userJwt.Id()
// This will return the following error
// strconv.ParseUint: parsing "name/john": invalid syntax

If we didn’t call the Id method, this subtle bug slip silently into the product and someday will end up as a production issue!

In a nutshell, this approach is not robust, and the resulting code is not clean.

Using Interfaces - A better approach

Ideally, we want the json.Unmarshal function should return an error if the sub doesn’t contain a uint value as string.

To make it happen, we need to inform the json.Unmarshal function somehow to do the type conversion while unmarshalling and return an error if the conversion fails.

How to make it happen?

We can do this by using the Unmarshaler interface.

In our case, we can declare the UnmarshalJSON method with the UserJwt type and in the definition, we can do the type conversion. But that’d be an overkill as we need to do the unmarshalling of the other fields, Name, and Admin, which is already working well without any custom logic.

In other words, the effective way would be overriding the JSON unmarshalling behavior of Sub field alone by having the UnmarshalJSON method with uint type. But according to golang’s spec we can’t do it

You can only declare a method with a receiver whose type is defined in the same package as the method. You cannot declare a method with a receiver whose type is defined in another package (which includes the built-in types such as int).

To handle this kind of scenario, we can make use of the named types in golang and define a new type called Sub with an underlying type uint

1
type Sub uint

Then we can declare the UnmarshalJSON method with this Sub type.

1
2
3
4
5
6
7
8
9
func (s *Sub) UnmarshalJSON(b []byte) error {
  sub := strings.Replace(string(b), `"`, "", 2)
  v, err := strconv.ParseUint(sub, 10, strconv.IntSize)
  if err != nil {
    return err
  }
  *s = Sub(uint(v))
  return nil
}

Using the Replace function, we are getting rid of the double quotes in the actual JSON encoded value

With this new Sub type in place, We can rewrite the UserJwt by replacing the Sub field with the Id field of type Sub.

1
2
3
4
5
type UserJwt struct {
  Id    Sub `json:"sub"`
  Name  string
  Admin bool
}

The json tag with the value "sub" is required to map the sub key in the JSON claim with the Id.

Now if we try to unmarshal the invalid claim, the json.Unmarshal function will return an error

1
2
3
4
5
6
7
8
9
10
claims := `
{
  "sub": "user/john",
  "name": "John D",
  "admin": true
}`
var userJwt *UserJwt
err := json.Unmarshal([]byte(claims), &userJwt)
// This will return the following error
// strconv.ParseUint: parsing "name/john": invalid syntax

For a valid claim, we can now get the Id directly from the UserJwt Id field.

1
2
3
4
5
6
7
8
9
10
claims := `
{
  "sub": "1234567890",
  "name": "John D",
  "admin": true
}`
var userJwt *UserJwt
err := json.Unmarshal([]byte(claims), &userJwt)
// ...
fmt.Println(userJwt.Id)

That’s it! The code is in better shape now :)

Summary

In this blog post, we have seen how we can write cleaner code by leveraging the interfaces. The source code is available in my GitHub repository.

Comments

For the last one year, I was working with the development team of a leading media entertainment company in India and developed a cloud platform to distribute the DCPs across the globe in a uniform, scalable and cost-effective manner.

We built the entire platform using golang and event-driven microservices architecture. It was an exciting journey, and this blog post summarizes my experiences of using golang in this project.

The Learning Curve

Golang is a simple language to learn. Any developer who has some good experience in any programming language can pick it in a month. It took me two weeks to understand the language.

I’ve also learned a lot from the code review comments from the other developers of my team who are already familiar with the language.

The official golang site has done an excellent job in coming up two important learning resources for the beginners.

  • A Tour of Go - Through a series of in-browser hands-on tutorials you can get a good feel of the language in few hours.
  • Effective Go - It is one the best introduction as well as best practices documentation of a programming language that I have read. Right from package naming convention to how to do state sharing, this documentation will provide all it required to write an idiomatic golang code.

In addition to these resources, I recommend the Golang FAQs and the GoByExample for the beginners.

Go Routines, Channels and Select

In my perspective, go routines, channels and select are the sweet spots of golang. These abstractions are elegant and enable you to write concurrent programs without any hassles.

We have made substantial use of golang concurrency whenever it made sense, and the benefits were incredible. The blog post on Go Concurrency Patterns is an excellent read to get a feel of it.

Interfaces in golang

My favorite feature in golang is its support for interfaces.

Unlike C# or Java, you don’t need to specify the name of the interface when a type implements an interface. Because of this, the package in which the type present, doesn’t need to refer the package in which the interface has been declared.

Yan Cui has written a great blog post about its elegance.

The design for Go’s interface stems from the observation that patterns and abstractions only become apparent after we’ve seen it a few times. So rather than locking us in with abstractions at the start of a project when we’re at the point of our greatest ignorance, we can define these abstractions as and when they become apparent to us. - Yan Cui

The Go Proverbs

The Go Proverbs provides a set of principles and guidelines while developing software in golang.

Though certain principles (A little copying is better than a little dependency.) look weird, they have profound meaning behind it.

Like the saying, “You need to spend time crawling alone through shadows to truly appreciate what it is to stand in the sun.”, to appreciate these proverbs you need to spend a decent amount time in developing software in go.

Golang Standard Library

The golang standard library is futuristic. It has pretty much all that is required for solving the general problems efficiently.

The surprising factor for me is treating JSON as a first class citizen in the standard library.

Most of the languages rely on an open source library to deal with JSON. Having JSON handling as part of the standard library shows how much thought and care has been put in.

Declared and Not used Compiler Error

Another good thing that golang got it right is showing compiler errors for unused variables and packages.

The software which we write evolve along with its business requirements. Continuous refactoring, feature addition or deletion, changing the architecture to meet the new demands are some the stuff that we do during the evolution of the software. During these activities often we miss removing the unused code and the packages that we were using before.

Through this compiler error, golang forces the developer to write better code. The biggest advantage is while reading the code written by someone, you can be sure that the variables and the packages are always being used.

Also removing unused packages results in smaller binary size.

go Command (Command Line Interface) and go tools

The go command is an another well thought out feature in golang which comes by default with the installation.

Using go command, you can compile & build the code, download and install external packages, run the tests and lot more without relying on any other external tools. The go test command is incredibly fast and it made our job easier while doing Test-Driven development.

The gofmt command enabled our team to follow a consistent formatting of golang source files. Thanks to this awesome tool we hardly had code review comments on the format of the code.

The golang tools, especially goimports and gorename has been extremely useful

go install and Single executable binary

The command which I loved the most in golang is the go install command. It takes care of compiling the code and produces a stand alone executable binary. Due to caching of intermediary packages, it builds the executable binary incredibly fast.

To run this resulted binary file, we don’t have to install anything on the target machine (Like JVM, .NET).

It enabled us to share the microservice as an executable file with the other team members to create spikes, develop and locally test other dependent microservices.

Just like bundling the front-end assets into single javascript file and serving them via CDNs, we can even do continuous deployment of golang codebase by serving the binaries from a content provider like Amazon S3, if it makes sense!

We can also do cross-compilation of your golang code for different OSes and Architectures using go build command.

Open Source Libraries in golang

Golang has a thriving open source community, and it houses a vast collection of open source libraries and frameworks. Here is the list of some of the things that we have used heavily

  • AMQP - To interact with RabbitMQ
  • GORM - To perform database operations in PostgreSQL
  • go-uuid - To generate and use UUIDs
  • migrate - For database migrations
  • envconfig - For managing configuration data from environment variables
  • negroni - To write HTTP middlewares
  • httprouter - For handling HTTP routes. We have used this along with golang’s standard HTTP handlers
  • logrus - Simple and powerful library for logging
  • testify - For unit test assertions

For most of the common things that you’d like to do in your project, you can always find an off the shelf open source library

Private methods, functions and struct fields

I was admired when I came to know that just by using a naming convention, I can specify public and private methods, functions and struct fields.

It is a smart feature in the language. If you want to expose them as public, start with an upper case letter and start with a lower case letter if you want to them to be used only inside a package or a struct.

Not So Good Features in Golang

So far in this blog post, I’ve shared the features that I liked very much in golang. In the rest of the post, I am about to share the not so good features in golang. The heading was intentionally started with “Not So Good” as the things that I am about to share are not bad ones according to the golang language design principles and the go proverbs that we have seen earlier.

IMHO there is no perfect programming language, and there is no silver bullet. All languages shine in certain areas and not so good in some other areas. At the end of the day, a programming language is just a tool which we use to solve the business problems. We need to pick an appropriate language that solves the problem in hand.

Disclaimer: The intention of this blog post is to share my experiences and provide honest feedback to the people who is evaluating golang.

The Golang way of handling errors

The imperative way of handling errors is one of the common complaint raised by many people.

1
2
3
4
5
6
7
8
9
10
11
12
13
_, err = foobar1()
if err != nil {
 return err
}
_, err = foobar2()
if err != nil {
 return err
}
_, err = foobar3()
if err != nil {
 return err
}
// and so on

Rob Pike addresses this concern in a detailed blog post, Errors are values and quotes a pattern using errWriter to avoid the repetition. It is a good design and elegantly wraps the nil check on the error. The downside of this approach is we need to write an extra wrapper for every new type to get rid of the repetitiveness. The wrapper approach may get complex if the foobar1, foobar2, and foobar3 are addressing different concerns.

For people who is coming from typed functional programming languages (F#, Haskell, Scala) background, like me, this may look little awkward to write it in this way. There is a proposal to include Result type in golang but I feel it may not be incorporated.

Treating nil as value

While other languages are moving away from dealing with null values and replacing it with option types, golang is taking a backward step by treating nil (equivalent for null in Go) as a valid value. So we need to be cautious while using the nil value. Lack of discipline may result in runtime exceptions (panic in go vocabulary)

This problem can be avoided to some degree by having a function/method to return two values, the actual value to be returned and the error, and ensuring that the actual value will never be nil, if the error is nil and vice versa. Unfortunately, this approach leads to the repetition concern that we just saw.

Absence of generics

This also an another typical critic in golang. There is an answer of for this question in golang FAQ saying, it is in the pipeline but my opinion is it may not be in the near future.

There are some workarounds like using an empty interface and casting it to appropriate types using type assertion.

The real concern is trying to do map, filter, reduce operations over collections. Again there are workarounds by using templating and reflection, but it has its own cost (performance and boilerplate code). After fiddling around with different options, We went back to the classic way of iterating through for loop.

Mutable variables and imperative coding

For developers who is coming from the functional programming background, the mutable variables and imperative way of writing code are speed breakers in the golang journey.

Though golang has some good support for doing functional programming, the presence of mutable variables and imperative way doing certain things, hinder the progress.

Summary

Apart from these few concerns, I’ve liked golang and enjoyed using it.

The language design principles of golang are my important takeaways. I strongly recommend watching the talks of Rob Pike on golang design; even you are not going to use golang.

I’d consider using golang in an another project in future for sure if it is an appropriate language for the problem at hand.