Salt - Beginners Tutorial

  • Jul

At first learning Salt can seem like a daunting task, which can be if a holistic view of the system is not clear during the process. Here we will list and describe the basic components of Salt and how they interact with each other. This is not a step-by-step guide, the folks at SaltStack (the company that makes Salt) have provided us with great documentation and good step-by-step walkthroughs to get you through your first steps. I recommend you go there after getting a sense of the big picture here.


What is Salt?

Salt is a multi OS provisioning and manteinance system. What it does is get all the software and configuration into brand new servers automatically. This allows administrators and IT personnel to spend the least amount of time setting new servers, liberating them for more important tasks like developing new systems, investigating new tools, fixing bugs or getting the HP printer to work (good luck with that). It also standarizes what gets installed on each server, so bugs are easier to diagnose and fix. Salt also allows you to automatize and paralellize other common tasks like updating the server software or running backup scripts.



The installation process is very well covered in the Salt docs, you can find them here. As of Salt 2014.1.5 the following OS are supported: Archlinux, Gentoo, Debian, Ubuntu, Fedora, Windows, OS X, Solaris, SUSE, RHEL, CentOS, Scientific Linux, Amazon Linux and Oracle Linux; so you're probably covered.


Salt basic components

  • Master

  • Minions

  • Keys

  • States

  • Grains

  • Pillar


Master and minions

The master is the Salt server, the minions are the target machines that will be provisioned. Each minion has an unique ID based on its FQDN. One nice thing about Salt is that the firewall configuration of the minions rarely has to be changed, as the master doesn't connect to its minions, but is the minions that open and maintain a TCP connection to the master. The master then uses these open TCP connections to send commands back.



Keys are used for security. When a minion initializes it generates an asymmetric cryptographic key that must be approved by the master before it can send commands to that minion.



The states are the heart and soul of Salt, they define just that, a state, a specific set of things that are to be a certain way. States are stored on the master as Jinja2 templates and passed to the minions where they are rendered and parsed as YAML. They can be parameterized by grains and Pillar (more on that in a minute). States can manage a myriad of things including, but certainly not limited to, files, MySQL databases, PostgreSQL databases, system packages, system services, ssh keys, users, python virtual environments, rvm environments, pip packages, timezones, mail aliases; heck, they can even run arbitrary commands on the console.

States are stored in sls files, each of these files can store several states and may reference other sls files. An sls file may look something like this:

### ssh.sls ###
ssh-server: # State ID.
pkg.installed: # Make sure that ssh is installed.
- name: openssh-server

sshd-config: # State ID.
file.managed: # Make sure that the ssh config on the minion
- name: /etc/ssh/sshd_config # is the same as the one stored in the master.
- source: salt://ssh_state/files/sshd_config
- require: # Make sure that ssh is installed before
- pkg: ssh-server # managing its config file.

ssh-service: # State ID.
service.running: # Make sure the ssh service is running and
- enable: True # that is configured to start at boot.
- watch: # Restart the service if there are
- file: sshd-config # changes in the ssh config file.

Don't fret if it looks daunting or there are things that are not clear, the idea is to get a basic understanding of the abstractions that states provide.



The grains are stored on each minion in YAML and hold OS and hardware specific information to that minion. Things like the FQDN of the minion, its minion ID, the CPU flags, IPs of the different interfaces, kernel information, total memory, OS family, the salt version of the minion among other pieces of information; all of this is automatically gathered by Salt. You can add grains to a minion by placing them in the file /etc/salt/grains in YAML format, or in the minion configuration file also in YAML format under the id 'grains' . Here is just a fraction the grains on my computer:

kernel: Linux
kernelrelease: 3.14.7-100.fc19.x86_64
localhost: jonhdoe
lsb_distrib_id: Fedora
manufacturer: TOSHIBA
master: salt
mem_total: 5442
nodename: jonhdoe
num_cpus: 4
num_gpus: 1
os: Fedora
os_family: RedHat
osarch: x86_64
oscodename: Schrödinger’s Cat
osfinger: Fedora-19
osfullname: Fedora



Pillar is completely awesome and totally optional, but throughout this tutorial we'll assume that you are/will use it. Pillar is stored in the master as a Jinja2 template; and is passed to minions, rendered and parsed as YAML when its information is needed, thus, Pillar has access to the minion grains, which includes the minion ID.

Pillar is used to store data, and it can store any kind of data; it is a very versatile system. It usually holds:

  • Sensitive data: Passwords, usernames, etc.

  • Minion configuration: ie. The name of the Apache package (RedHat based is httpd, Debian based is Apache2)

  • Variables: ie. The list of the company DNS servers, or a variable that indicates to which deparment/subnet/category the minion belongs.

Configurations files are usually not stored in Pillar as they're generally specific to certain states and are thus stored next to their sls files.

One of the neat things about Pillar is that the transfer is cryptographically secured, and that it is stored in the master but rendered in the minion. This way you can pass common, static data to the minions from the master (ie. The list of users to create on the minions for the admins to use), as well as generate dynamic information regarding one minion in particular (ie. A list of name packages to install based on the minion OS and IP).

Pillar files are also stored with an sls extension. A Pillar file, lets call it common.sls, may look something like this:

### common.sls ###

{% if grains['os'] == 'Debian' %}
apache: apache2
{% elif grains['os'] == 'RedHat' %}
apache: httpd
{% endif %}

username: gjaber
password: <cryptographic hash or a extract of a perl script here>
moba: LoL
username: mrondon
password: <cryptographic hash or a extract of a perl script here>
moba: Dota2


You may not need Pillar at first, but is a powerful and essential tool for more advanced setups.


How is it put together?

Basically, the states define how some things should be. Pillar and the grains are used to determine which minions are subjected to which states and to parameterize the states themselves. Here is a diagram:

I omitted the details of the YAML parser as they're not essential to get the point across. Also, in the image, it should be YAML not 'yalm'.

Note that the Pillar templates are rendered first and then fed into the states templates together with the grains. After the rendering is done, Pillar and the grains are still used to determine what states are to be applied to the minion.

In simpler cases, there won't be a complicated process to decide which states get applied because we'll directly specify the sls file. On the simplest case the state and Pillar files won't even be necessary because we'll just target a set of minions and specify one function we want to run on them.

But how exactly does Pillar and the grains target minions and parameterize states? The section "Targeting minions and specifying states/Pillar data" tackles these issues, but first we have to learn a bit about how Salt is organized and the different options we have to send commands to the minions.


Directory structure

Before diving onto how exactly we target minions, we must learn how Salt normally stores its states and Pillar information on the master.

The configuration files for the master are in /etc/salt/, the main one being /etc/salt/master. However the state files (sls files) are usually stored in /srv/salt. Pillar files are usually stored in /srv/pillar. Each of these directories has a file called top.sls that indicates which minions are subject to which states/Pillar files. The configuration files for the minion are in /etc/salt also, the main one being /etc/salt/minion.

Salt has support for a multienvironment setup that allows for a minion to use a different set of aggregated directories from which to pull states/Pillar based on this grains and Pillar data, but for the purposes of this tutorial and the sake of simplicity, we'll assume that there are no shenanigans, so all sls files are stored directly under /srv/salt and all Pillar files are stored under /srv/pillar, following a plain directory structure.


Applying states

There are two ways to apply states to minions.

One is specifying a given set of minions and an sls file. Salt allows you to group minions by a name beforehand, or use regular expressions, grains and pillar to define a set of minions. In this case, the top.sls files are irrelevant.

The other, more interesting one, is to specify a set of minions (may be all minions) and saying to Salt: “You find out what needs to be done to these guys, I'll drink cocoa while I wait”. Though this sounds nice, what comes before is having configured the top.sls files on /srv/salt and /srv/pillar, and all of the pertinent sls files, but to be fair that hard part you only have to do "once".

You can also specify a set of minions and a specific command to run on them (ie. pkg.install apache2). This is useful when we don't need/want to create an sls file for a minor change.


Targeting minions and specifying states/Pillar data

Targeting what minions get hit is done from the command line. Specifying what is done to the those minions is done either by writing a Salt function as part of the command (ie. pkg.install apache), specifying an sls file, or asking Salt to use the top.sls files to determine the pertinent states.


Specifying minions

Let go over how to say who gets hit. This is done from the command line.

Salt uses ZeroMQ for the communication with its minions. As of Salt 2014.1.5, minions can be targeted by: minion ID, grains, pillar data, IPv4 address, FQDN or predefined minion groups (node groups), or by any combination of the above; all of these support regular expressions too. Lets see some examples:

# is a special Salt
# function that test the connectivity
# of the minion's Salt daemons.
# It is not an ICMP ping.

# The special * wild card matches all minions
salt '*'

# By minion ID, target all minions that end in *
salt '*'

# By minion ID, and using regexes instead of shell-like globbing,
# target web-prod and web-dev
salt -E 'web-(prod|dev)'

# By minion ID, a list of minions
salt -L 'web-jonh, dborac-ether'

# By grains, target the RedHat and Debian systems
salt -G 'os:(RedHat|Debian)'

# By Pillar data, target human resources
salt -I 'deparment:HR'

# By IP/subnet, target the local network
salt -S ''

# By minion ID and grains
salt -C 'web-* and G@os:Ubuntu'

# By grains and Pillar data
salt -C 'G@cpuarch:x86_64 and I@office:32D'


Calling a Salt function

Now onto saying what gets done. The simplest way is to specify a Salt function. Salt offers a variety of functions. There are functions to manage the package system, run tests on the minion, manage files, manage web servers among other things. These functions are called Execution Modules; you can write your own execution modules by the way. Onto some examples:

# Ping all minions. Not an ICMP ping
salt '*'

# Emacs for everybody
salt '*' pkg.install emacs

# Run ls /etc
salt '*' 'ls -l /etc'


Specify an sls file

We can also specify an sls file. Sls files don't normally use execution modules, and instead use state modules that are called automagically by Salt when it processes the state, although there is a special state module to call execution modules from within sls files. More on writing states in a minute:

# Apply the states from the ssh.sls file on all minions.
# Notice how we omit the .sls extension in the command line.
salt '*' state.sls ssh

Highstate and the top.sls files

This is the most powerful way of saying what gets done, although is the most difficult to set up. It says to Salt to figure out what its to be done to each targeted minion. The command is simple enough:

salt '*' state.highstate # Change '*' for the target of your preference.

The parameters that determine which of the selected minions get what states and pillar information is in /srv/salt/top.sls and /srv/pillar/top.sls, directories may vary depending on your setup, but these are the most common ones.

We'll explain only how to setup /srv/salt/top.sls, as one is analogous to the other:

### top.sls ###
base: # Mandatory name of the base env, ignore it for now
'*': # All minions targeted get the following sls files
- common # Name of the sls file minus the extension

### common.sls ###
common-pkgs: # State ID
pkg.installed: # Make sure all of these are installed
- names:
- emacs
- openssh-server
- nginx

admin-user: # Set up a user for administration purposes
- name: {{ pillar['admin-name'] }}
- password: {{ pillar['admin-pass'] }}

# You could also store in Pillar a dictionary of users
# and iterate over them here, creating them all.

This top file is telling us: “all minions are to run the states on 'common'.sls". To clarify, this target is not telling what minions get hit, that was specified on the command line, this target is acting on the set of minions that were targeted, and is telling that all of those minions are to apply the states on common.sls:

Now, onto something more complicated:

### /srv/salt/ ###
. .. common.sls nginx.sls postgres.sls top.sls

### top.sls ###
- common

- nginx

- postgres

This uses the minion ID to run different sls on different minions, particularly, is telling the targeted web server minions to install nginx, and the database ones to install postgresql, and all of them to run the states in common.sls. So for example, a minion with ID 'web-skynet' would apply the state files 'common.sls' and 'nginx.sls', 'dbGlaDos' would apply 'common.sls' and 'postgres.sls', and 'theCakeIsALie' would only apply 'common.sls'. Lets see what else can we do:

### /srv/salt/ ###
. .. apache.sls ati_fglrx.sls common.sls nginx.sls top.sls

### top.sls ###
- common

'web* and G@wserv:nginx':
- match: compound # We need to specify the kind of
- nginx # match when not matching against IDs

'web* and G@wserv:apache':
- match: compound
- apache

'gpus:model:*Radeon*': # Two colons means that the grain
- match: grain_pcre # 'gpus' is a dictionary and we want
- ati_fglrx # the key 'model'

Here we have set grains in our servers in advance that tells us whether it should run Apache or Nginx, we also use a grain to install the ATI proprietary driver FGLRX on the targeted machines that have Radeon graphic cards.


The top.sls on /srv/salt/pillar (directory may change depending on your implementation) functions exactly the same way, only that instead of running states, it tells what minions receive which Pillar files, and hence what data.


Writing states

Now onto the heart and soul of Salt, writing state files. Lets begin simple:

### sl.sls ###
sl-pkg: # State ID
pkg.latest: # State module
name: sl

Short, concise. It installs the package sl if it not installed already, updates it if it is outdated. Salt makes sure that the package database is updated prior to doing package operations. What about files?

### unattended-upgrades.sls ###
unattended-upgrades-on: # State ID
file.uncomment: # State module
- name: /etc/apt/apt.conf.d/50unattended-upgrades
- regex: 'Unattended-Upgrade::Mail .*;'
- char: '//'

### collectd-global.sls ###
- name: /etc/collectd/collectd.conf
- source: salt://collectd/files/collectd.conf

The first one uncomments a line that matches 'regex', turning on Ubuntu's unattended upgrades. The second one makes sure that the file /etc/collectd/collectd.conf in the minion is exactly the same as the one in /srv/salt/collectd/files/collectd.conf in the master; the Salt file state module is very flexible. Now, what about services?

docker-serv: # Make sure the service 'docker' is
service.running: # running and enabled to start at boot.
- name: docker
- enable: True

This is simple, but how about a more complex configuration? With files, services, packages, repositories, and custom commands?

docker-kernel-pkgs: # Install these kernel packages
- pkgs:
- linux-image-generic-lts-raring
- linux-headers-generic-lts-raring

docker-apt-https-transport-method: # Run this command...
- name: apt-get update & apt-get install -y apt-transport-https
- unless: [ ! -e /usr/lib/apt/methods/https ] # ... unless this is true
- require:
- pkg: docker-kernel-pkgs # This state has to run successfully first

docker-repo: # Install the Ubuntu PPA for Docker...
- name: deb docker main
- file: /etc/apt/sources.list.d/docker.list
- keyserver: hkp://
- keyid: 36A1D7869245C8950F966E92D8576A8BA88D21E9
- require:
- cmd: docker-apt-https-transport-method # ...only if this state ran

docker-pkg: # Install the package lxc-docker...
- name: lxc-docker
- require:
- pkgrepo: docker-repo # ...only if you ran the state docker-repo already

docker-serv: # Make sure that the 'docker' is up and
service.running: # running...
- name: docker
- enable: True # ... also set it to start at boot.
- watch:
- pkg: docker-pkg # If this package changes, restart this service.

Well, that was a handful. The most confusing bits might be the require and watch clauses, these are requisites, which are pretty important in Salt.

You see, Salt ensures that states run in the order they are written, and if one fails, it will continue forward; but when you add requirements to the mix, you can do stuff like restarting a service when a file or package changes, turn off load balancing servers when a deployment is about to run, stop running a state if a previous one fails among other things.

Most likely you'll be using 'require' the majority of the time, with a couple of 'watch' into the mix. If you feel like you are writing too many boiler-plate 'require' clauses, you can turn on the fail hard global option.


Some caveats and a LIE (as of Salt 2014.1.5)

Caveat 1: The state ID namespace is flat

The next two states are exactly the same:

nginx-pkg: # State ID
pkg.installed: # State module
name: nginx

nginx: # State ID
pkg.installed # State module

You see, EVERY state receives a 'name' argument, and when that argument is not provided in the sls file, Salt uses the state ID and plugs it in the argument. This form is very common in examples around the internet.

I don't like this form, the reason is that *** the state ID namespace in salt is flat ***. That means that if there are any two states with the same ID, no matter in which folders, sls files or environment, when highstate gets run you'll get something along the lines of:

Detected conflicting IDs, SLS IDs need to be globally unique.

So lets say for example that you have one state for collectd for your web servers, and another for your dns servers:

### web-collectd-global.sls ###
/etc/collectd/collectd.conf: # State ID
- source: salt://web-collectd/files/collectd.conf

### dns-collectd-global.sls ###
/etc/collectd/collectd.conf: # State ID
- source: salt://dns-collectd/files/collectd.conf

It will explode when you try to run highstate. There are two ways to solve this, number one:

### web-collectd-global.sls ###
web-collectd-global: # State ID
- name: /etc/collectd/collectd.conf
- source: salt://web-collectd/files/collectd.conf

### dns-collectd-global.sls ###
dns-collectd-global: # State ID
- name: /etc/collectd/collectd.conf
- source: salt://dns-collectd/files/collectd.conf

Problem solved. Number two, use the stateconf renderer, problem solved.


Caveat 2: The state module function file.recurse doesn't recurse file permissions

There is this neat state module function called file.recurse, it functions kind like rsync, you pass a directory on the master and one on the minion, and it will make the one on the minion like the one on the master, except that it will not recurse file permissions, so any executable files you will have to make executable again with file.managed.


Caveat 3: The is a MySQL grants state module, but no PostgreSQL grants state module, nor a general database state module for that matter.

Well just that, if you want to manage PostgreSQL grants, you'll have to use the cmd state module, and your sls files will have to cater every kind of DB you're running separately.


Caveat 4: Minions might sometimes fail to return data to the master

I've encountered this problem several times, it is recorded as an issue. Don't fret if this happens, the minions are still working and each one has its own internal log, but it can be disorienting at first.


There are surely other caveats, but these are the ones I remember for now that have affected me. Onto the LIE.


Lie: States have to be in files

I lied...yes...I know...I'm a terrible person, but hear me out, I did it for your own good. The matter of fact is that states do have to be in files, but you can place them in files called 'init.sls' and place those in directories with the name of the state. For example, if I want to make a SSH state, I wouldn't make a file called 'ssh.sls', but I would create a 'ssh/' directory inside '/srv/salt/':

### /srv/salt/ssh/ ###
. .. init.sls files/

### /srv/salt/ssh/files/ ###
. .. sshd.conf

That way your sls files and all of its config files are kept inside a nice little box. This is particularly useful to contain in an orderly manner several sls files that call each other.

Also it was a white lie, and a lie of omission, so we're cool...right??? Here, I'll give you some gifts to compensate.


Extra features not covered here

  1. Orchestration: this is for more complex setups that need to be done in a certain way and order. It was previously handled by the Overstate system, but that is being replaced by the Orchestrate runner.

  2. salt-cloud: you give salt the credentials of your cloud computing services account (AWS, Linode, DigitalOcean, etc), and it can provision the creation of instances on these platforms with the salt minion package already installed. This makes a breeze to provision Salt with Salt for the parts of your platform that are in the cloud.

  3. salt-ssh: this is a functionality still in alpha. It is a command utility that allows to send orders to other machines using only SSH (more akin to what Ansible does). It could simplify your simpler use cases.

  4. State and execution modules for managing Docker: So if you're part of the Docker hype train, you might be interested in that execution and state modules for managing docker containers and images are already in beta.

  5. External authentication: if you want to run on a minion as somebody other than root (Salt runs as root by default), you should take a look at this feature. It allows to use alternate authentication methods.

  6. Syndic minions: Allows to build topologies of Salt servers for scalability.

  7. Proxy minions; This feature is still under development. It aims to tackle the situation of having some devices that can't, for whatever reason, run a Salt minion.

  8. salt-call: When we send commands to a minions, said minion does not return all of the output, moreover it sends it back only once it finished applying all of the required states. The command salt-call allows you to call a salt command from within the minion and see the live output. It is very useful for debugging purposes, specially when you are taking your first steps or testing a brand new state. To be clear, salt-call is called from the minion you're trying to debug, not from the master.

These are the ones I remember/know for now. Here are some other useful links I provided throughout the tutorial:

  1. Documentation main page

  2. Official walkthrough

  3. Requisites

  4. Stateconf renderer

  5. Fail hard global option

  6. Special state module to call execution modules

  7. Environments

  8. Nodegroups

  9. Targeting minions

If you find an error of have a suggestion, hop on the comments.

Posted on July 25, 2014, 7:05 p.m.