If you are developing a distributed application that consists of multiple services, you might be thinking about how to manage the ever growing application configuration data. Instead of maintaining individual configuration files for each service, you can store all your configuration data in a key-value store. In this blog post we’ll check out the key-value store in Consul.
Consul is an open-source product developed by HashiCorp and licensed under the MPL 2.0. While Consul uses an open core business model, it comes with a great deal of functionality in its free edition. The top two features of Consul would be the service discovery combined with health checking and the key-value store functionality that we are going to review in this article. They both come handy when building distributed applications.
HashiCorp products are known for its thorough documentation and the Consul’s documenation is not an exception.
What I like about Consul is its installation. Written in the Go language, Consul is distributed as a single statically linked binary. Download
links for various platforms are provided. After unzipping the distribution archive you can directly run the
Let’s put togher a command-line to start the Consul cluster. Our test cluster consists of a single node (
-bootstrap-expect 1). For a production deployment, you should be looking at a cluster of three or five Consul nodes that is able to survive node failures. We will make the Consul Web UI available at
http://localhost:8500 by appending the
-ui parameter. Consul needs a location where it will persist its data. In our example, we instruct Consul to create a directory
mydata and store all its data in this directory. After a bit of typing, the complete commmand-line to start the Consul cluster looks as follows:
In several seconds the one-node Consul cluster is up and running:
In the log output, Consul informs us that the Consul API is available at 127.0.0.1:8500. That’s where the Consul client will connect to by default. In the following, you want to make sure that you’re running the Consul commands on the same box as you started your Consul cluster.
consul binary provides the server as well as the client functionality. Let’s list our current cluster members to verify that the client can connect to the cluster:
Basic CRUD with Consul
In this section, we’re going to exercise the basic Create, Read, Update and Delete functionality of the Consul key-store. First, let’s store the value
12345 under the key
Great, the value is saved in the store. To retrieve the value under the key
foo from Consul we can type:
By the way, Consul doesn’t impose any restrictions on what kind of data you may store. Only the size of the data is limited to 512KB of data per key. It’s up to your application, what data format you choose to use. For example, you can decide to store numbers, strings, JSON-formatted data or arbitrary binary data. For instance, when designing a centralized configuration management solution for your application, you have the flexibility of storing individual configuration options as key-value pairs or decide to save entire configuration files as values in Consul.
To replace the value, simply put a new value in Consul under the existing key:
The value has been successfully updated as we can see:
To remove the value from the key-value store you can use the
To verify that the value is really gone, try to retrieve it:
Hierarchical keys and prefix matching
Keys in Consul can be organized in a hierarchy where different levels of the hierarchy are separated by the slash character (
/). For example, you can create a database that holds the population numbers in different continents and countries (in millions of inhabitants) like this:
Now that you organized your keys hierarchically, you can use the Consul’s prefix matching to discover the keys on the single level of hierarchy. For example, to retrive the keys with the prefix
Prefix matching can be used to retrieve the values, too. For example, to retrieve the population numbers in Europe, you can type:
Note that when retrieving the keys recursively, only the keys on the single level of hierarchy were returned whereas when retrieving the values recursively, values on all the nested levels of hierarchy were returned.
To obtain the population numbers for the European countries, you can append a slash to the keys name (
And if you are interested only in the European countries that start with letter
g, you can use the prefix
Export/import of key-value pairs
Another useful feaure of the Consul’s key-value store is the bulk export and import of key-value pairs. To export the entire key-value store database, you can type:
Consul exports the key-value pairs into the JSON format which is currently the only supported format. In the sample output, you can see that all the values are base64 encoded. The base64 encoding is commonly used in the text-based formats like JSON and XML to allow embedding of binary data.
You can export a subset of the key-value pairs by specifying the prefix. For instance, to export the data pertaining Europe, you can speficy the
To import the JSON-formatted data back to the Consul key-value store, you can use the command
./consul kv import.
Besides the commmand-line client, you can access Consul through its beautiful Web interface. Point your web browser to http://localhost:8500.
In this blog post, we reviewed the basics of the key-value store in Consul. There are many other cool features of the key-value store that we didn’t cover like atomic key updates using Check-and-Set operations, transactions, locks or watches. Also, I recommend to you to take a look at the Consul’s great RESTful API that allows you to interact with Consul programatically.
If you’re looking for a key-value store that would enhance your distributed application, Consul is definitely a candidate to consider. Besides that, Consul will be ready when you later on realize that service discovery is what you need to address next.
Are you considering or already using Consul at your company? I would like to hear your experiences, please leave your comments below.