Structuring Go Packages

Making reusable packages that don't suck

Packages should solve one problem domain, be reusable by default, and individually testable.

Two Types of Packages

In-app packages are a small part of a broader application. These packages typically serve a specific purpose and isolate functionality rather than address reuse.

Stand-alone packages are created to share functionality across many applications. They should be independent of application implementation. Allowing users to call the package in many different ways.

StanD-alone

In-App

Directory Structure

There are many proposed structures for In-app packages. But there is no standard everyone follows. 

In-App

$ find . -type d
./internal/app/
./internal/pkg/
./internal/pkg/nonreusablepkg2/
./internal/pkg/nonreusablepkg1/
./pkg/reusablepkg1/
./pkg/reusablepkg2/

There are recommendations, such as creating a /pkg directory for public sharable packages. At the same time, creating an /internal/pkg directory for internal-only packages.

$ find . -type d
./app/
./reusablepkg1/
./nonreusablepkg1/
./reusablepkg2/

I put all packages in the repository top level under their own package names. I like this method as it keeps all packages in the same place, a sane easy to find location.

Directory Structure

For stand-alone packages, there is a common pattern followed by most.

Stand-Alone

Stand-alone packages should be in their own repository.

The code files should be located in the top-level directory. Any sub-packages should be directories at the top level.

$ find . -type f
./go.mod
./drivers/redis/redis_test.go
./drivers/redis/redis.go
./drivers/cassandra/cassandra.go
./drivers/cassandra/cassandra_test.go
./LICENSE
./go.sum
./README.md
./CONTRIBUTING.md
./hord_test.go
./hord.go

Naming and Organization

Naming files within a package is a key part of keeping a package organized. The guidelines below apply to both in-app and stand-alone packages.

 

  • The primary file, the one that should have the package documentation, should have the same name as the package. For example, a package by the name of config should start with a config/config.go file.
  • It's good to break out functionality into multiple files. Make sure that similar functionality is grouped. I.E., yaml.go, env.go, json.go, or consul.go.
  • When breaking out functionality, keep types, constants, or errors where they are referenced or logically sit. Many times, this may be the base file, i.e., config.go. Avoid creating a constants.go or types.go.
  • Apply the same logic to utility functions, put them where they are most used, or default to the base file. Avoid creating a util.go.

Structuring the Package Interface

As important as naming and directory structure. How we structure the interface is how users work with our packages. The functions, methods, and types we export from our packages dictate how users use them.

 

Some packages export simple functions that hold no long term state. Other packages will export structures to hold data and provide methods of access. Many packages will do both.

Just Functions

Many packages provide users with simple helper functions. These functions take data as input and provide something new or modified as output.

Package Interface

  • These packages should never be used to keep data as this prevents users from creating multiple instances to work with different sets of data.
  • If you find yourself using init() to create a sync.Map or initialize other things. Ask yourself, if I run multiple tests in parallel, will it break?
  • Watch out for turning these packages into a generic utils package. It's easy to group helper functions into one package, but a package should solve one problem. Utilities are not one problem.
package config

// Marshal will take in an interface and format it
// into config data.
func Marshal(in interface{}) ([]byte, error) {
	// do stuff
}

// Unmarshal will take in raw config data and parse
// it into the provided interface.
func Unmarshal(in []byte, out interface{}) error {
	// do stuff
}

Structs with Methods

Some packages store context, data, or control services. These packages usually export a pointer to a custom type with methods. Each pointer is an individual instance a user can interact with.

Package Interface

  • By using the custom types with our getter and setter methods. Users can create many instances of Config, each with different data.

  • When writing tests, each test should have its own instances of Config. Then, they can run in parallel.

  • Stick to using New() or Dial() to create new instances. Users would be calling config.New().

package config

import "sync"

// Config is a custom type for holding and changing configuration
// data.
type Config struct {
	sync.RWMutex
	item string
}

// New will create a new instance of configuration.
// Each instance is unique and can hold different information.
func New() *Config {
	// setup the config object
	return &Config{}
}

// Item is used to access Config.item
func (c *Config) Item() string {
	c.RLock()
	defer c.RUnlock()
	return c.item
}

// SetItem is used to change Config.item
func (c *Config) SetItem(s string) {
	c.Lock()
	defer c.Unlock()
	c.item = s
}

Push Dependencies, Don't initialize them again

Sometimes, packages must load outside dependencies. This is common with Config, Logging, etc. It's tempting to re-initialize these on package load, but don't do that. Let the calling application push those dependencies to the package.

package database

import "github.com/some/logger"

// don't do this... just don't
func init() {
	log = logger.New()
	cfg = config.New()
}

func MyFunc() {
	// do work
	log.Printf("Tell someone about it")
}
package database

// imagine some imports here

type DB struct {
	cfg *config.Config
	log *logger.Logger
}

func Dial(log *logger.Logger, cfg *config.Config) (*DB, error) {
	// create a new DB control object
}

func (db *DB) MyFunc() {
	// do work
	db.log.Printf("Tell someone about it")
}

Do not even...

Better, but not perfect

You don't need those dependencies, not like that atleast

In the example before, we pushed a config object into our database package. But is that what we want? Now our package is locked into using this specific config package. What did it buy us? Dynamically update our DB config? It sounds nice but it doesn't work like you think it will.

 

In this case, the implementing app should create a new DB instance and shut the old one down. That's the whole point of this package structure.

package database

type DB struct {
	server   string
	password string
}

func Dial(server, password string) (*DB, error) {
	// create a new DB control object
}

NOw We are Talking

When pushing dependencies, ask yourself. Is there a better way of doing this? A way that doesn't kill reusability?

But What about that logger? Just... Don't

Logging has no place in stand-alone or in-app packages. It should only be executed within the implementing application.

 

While logging within packages is a pattern that is popular in other languages. That doesn't mean it is right for Go.

  • Is it the right log level? An error to you might not be an error to me.
  • It removes the user's ability to act on error. Maybe I need to do more than log.

Handle Errors by Passing them up the stack

But if you can't log, how do you get errors to the user? That's where multi-value returns and custom error types come into the picture.

 

As a package, it's preferable to pass any errors to the calling application. This allows the calling application to check the error and perform a specific action.

myVal, err := db.MyFunc()
if err == db.ErrMyCustomError {
	// do something fancy
}
if err != db.ErrMyCustomError && err != nil {
	// ok maybe log and do something less fancy
}

Handling Errors Asynchronously

Sometimes a package must perform tasks in the background. When writing these types of packages, it is still better to provide the user with an error. But rather than returning the error on a function, call back to the user with an error.

 

I like two ways of accomplishing this. Either through an error channel or an error function. Error functions are user-defined functions that the package can callback with errors.

type Setup struct {
	// user defined error function
	errFunc func(error)
}

func main() {
	// on setup
	err := New(Setup{
		errFunc: func(err error) {
			if err == db.ErrMyCustomError {
				// do something fancy
			}
			if err != db.ErrMyCustomError && err != nil {
				// ok maybe log and do something less fancy
			}
		},
	})
}

Packages should be made asynchronous only when necessary. It is often easier to implement asynchronous execution within the implementing application itself. This includes error handling.

If you can't convert an In-App package to STandAlone it's too tightly coupled

Earlier, we discussed the differences between in-app and stand-alone packages. At the same time, there is often a difference in the locations of these packages. Functionally and structurally, there should be no difference between the two.

 

A well structured in-app package should be able to be moved to a stand-alone package with minimal changes.

Summary

  • Naming and location of packages are important. But there is no official standard, try to keep it sane.
  • In-app and stand-alone packages should functionally and structurally work the same.
  • When storing data, use structs with pointers and methods. Not only is this better for users, but its also easier to keep things goroutine safe.
  • When not storing data, keep it simple.
  • Don't initialize dependencies; push them from the implementing application. Or better yet, structure the package not to need them.
  • Always pass errors to users for error handling. Don't assume a package knows better than the implementing application. 

EOF

Benjamin Cane

Love this deck? Want a PDF version to keep for offline use? Buy it now.