While working on a Go-based service, I realized I had written a few hundred lines of code to load configuration settings from environment variables. I decided that it made sense to create a better way that worked like the native encoding/json library.

This article is a detailed breakdown starting with how you can use it (so you can use it if you want, it's MIT licensed), how it works and going into the weeds about Go features it uses (struct tags), and then finally a short digression on refactoring and how a little bit of refactoring made the code much easier to understand. There will be a follow-up article at some point about wrapping up the TODO items for the library.

You can see the library code here: https://github.com/everettcaleb/envconfig.

How You Can Use It

To get started using the library, you need to go get it. Here's the command:

go get -u github.com/everettcaleb/envconfig

After that, you can define a struct that will contain your configuration settings. This will be used (using reflect) to look up environment variables and map them to the native Go types in your struct. Here's the example one from the README:

type appSettings struct {
    Port         int    `env:"PORT"`
    DBConnection string `env:"DB_CONNECTION" required:"true"`
    RedisPort    int    `env:"REDIS_PORT"`
    RedisHost    string `env:"REDIS_HOST" required:"true"`
}

All the fields need to be exported if you want them to be used by reflection. Currently, all built-in types except arrays, slices, and structs work. I'll add those in later (and write about how they work). Struct tags are used to say which environment variables to look for and whether they are required or not. If the "env" struct tag is omitted on a field, that particular field will be ignored. The ideal pattern for loading configuration is something like this:

  • Create an instance of the config struct populated with defaults
  • Load environment variables into it with envconfig.Unmarshal
  • Validate options and start any connection pools, etc.

Here's what this might look like:

settings := appSettings{
    Port:      8080,
    RedisPort: 6379,
}

err := envconfig.Unmarshal(&settings)
if err != nil {
    panic(err)
}

// set up the DB pool and Redis pool...

Currently, there's a restriction on the kind of fields that can be mapped. This will be fixed in a later version but currently (as of version v1.0.0) arrays, pointers, slices, and structs will not work.

How It Works (an exploration of struct tags)

In this section, I'll refer to specific code snippets and show them here. If you want to see the full source code for version v1.0.0 you can find it here on GitHub. First off, let's walk through what envconfig.Unmarshal does. Source code is in "unmarshal.go":

The definition of Unmarshal:

func Unmarshal(i interface{}) error

It checks for nil

if i == nil {
    return fmt.Errorf("expected Unmarshal parameter to be a pointer to struct, received nil")
}

It looks up the type of the parameter (should be a pointer to a struct)

it := reflect.TypeOf(i)
if it.Kind() != reflect.Ptr {
    return fmt.Errorf("expected Unmarshal parameter to be a pointer to struct")
}

It gets the type of the pointer's target (should be a struct)

et := it.Elem()
if et.Kind() != reflect.Struct {
    return fmt.Errorf("expected Unmarshal parameter to be a pointer to struct")
}

And then it gets the value of the struct (using reflection)

v := reflect.ValueOf(i).Elem()

That's a lot to digest all at once but basically, here's what we've got now:

  1. We've checked that the parameter (an "interface{}") is definitely a pointer to a struct
  2. We have a way of exploring the struct's definition via reflect.Type ("et" above)
  3. We have a way of manipulating the struct's value via reflect.Value ("v" above)

What the code does next is it iterates over all the fields in the struct. This is done like this:

for fi := 0; fi < et.NumField(); fi++ {
    f := et.Field(fi)
    
    // ... other code ...
}

As you can see, we now have "f" which is a reflect.StructField and gives us access to information like the field's name, its type, its tags, etc. What's done next is we lookup to see if the field has the "env" tag:

envName, ok := f.Tag.Lookup("env")
if !ok {
    continue
}

If it doesn't have that tag ("!ok" would be true) then we just want to continue looping. We also need to check for the "required" tag:

required, err := getFieldTagBool(&f, "required", false)
if err != nil {
    return err
}

The function getFieldTagBool (link to code) is just a convenience helper that looks up the specified struct tag and otherwise uses the provided default. After that, we run the big helper function that looks up the environment variable and attempts to use reflection to set the applicable struct field's value. The function is called getEnvAndSet (link to code) and we pass in the reflected struct value "v", the reflected field "f", the field index "fi", and the name of the environment variable that we got from the "env" struct tag.

func getEnvAndSet(obj *reflect.Value, f *reflect.StructField, fieldIndex int, envName string) (bool, error)

It returns whether the field was set (the environment variable exists) or an error. It works by using a big switch statement over the field "Kind"

k := f.Type.Kind()
switch k {
    // ... cases here ...
}

The cases looks like this:

case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
    v, isSet, err := getEnvInt(envName, 10, bitsOf(k))
    if isSet {
        obj.Field(fieldIndex).SetInt(v)
    }
    return isSet, err

In plain English, each case uses a helper that calls os.Getenv(name) and converts it to it's native type using a function like strconv.ParseInt(s, base, bits). If the environment variable was set, it uses reflection to set the struct field to that value. This is the same for all the other kinds as well, and integers were a particularly easy to read example. You can see the full code here.

Now, going back up to Unmarshal, we can see that if a field was required to be set and was not, we'll fail with an error.

if required && !isSet {
    return fmt.Errorf("the environment variable %s is required but was not specified", envName)
}

That's all there is to it.

A Little Bit of Refactoring

The original version of the getEnvAndSet function (link to code) was very bloated because it treated each kind as its own case. This lead to code that looked like this:

case reflect.Float32:
    v, isSet, err := getEnvFloat(envName, 32)
    if isSet {
        obj.Field(fieldIndex).SetFloat(v)
    }
    return isSet, err

case reflect.Float64:
    v, isSet, err := getEnvFloat(envName, 64)
    if isSet {
        obj.Field(fieldIndex).SetFloat(v)
    }
    return isSet, err

This was not ideal as there were very similar cases that seemed like an ideal target for refactoring. I created a quick function called "bitsOf" (link to code) that would allow me to use the same case and provide a bit value for the "getEnvFloat" function. Here's what it looks like:

func bitsOf(k reflect.Kind) int {
    switch k {
    case reflect.Int8, reflect.Uint8:
        return 8
    case reflect.Int16, reflect.Uint16:
        return 16
    case reflect.Float32, reflect.Int, reflect.Int32, reflect.Uint, reflect.Uint32:
        return 32
    case reflect.Float64, reflect.Int64, reflect.Uint64:
        return 64
    default:
        return 0
    }
}

This allowed me to combine cases which made the code much easier to read. Here's what it made the float case look like:

case reflect.Float32, reflect.Float64:
    v, isSet, err := getEnvFloat(envName, bitsOf(k))
    if isSet {
        obj.Field(fieldIndex).SetFloat(v)
    }
    return isSet, err

So whenever you're looking at your code, think for a second, "Does this look similar to other code I've written? How could I combine similar cases?" Always strive to make your code as easy to read and maintain as possible. Even if you understand what it does now, the easier you can make it for you-in-six-months to understand, the happier you'll be down the road when you have to maintain it.

Conclusion

Thank you for reading my article. I will try to publish an article about once every week or two about some coding topic that's on my mind. The next article will be about adding support for arrays, pointers, slices, and structs to this library and this whole sentence will become a link to it once I've written it. You can follow me on Twitter for more coding stuff: @everettcaleb.