Eric Radman : a Journal

Concepts from Salt Worth Keeping

In early 2016 I discovered Salt, and I can honestly say that it has changed the way I view and manage systems. For the duration of this year I systematically encoded procedural knowledge of systems I maintain into SLS files, and then have become a contributor to saltstack/salt. This is not the first time a paradigm shift has occurred in my work:

The following are some of the things I think are brilliant about Salt, and what I learned before launching scriptedconfiguration.org.

Minimal Hierarchy

Rather than navigating a hierarchy that is assembled inside of the computer a strait-forward reading can tell you what is included


# top.sls
base:
  '*':
    - common
  db1:
    - postgres

And the import pattern is simple: common will match common/init.sls or common.sls. sls files may include other files, but the control flow is still easy to read. Easy to read, however this simple concept is compounded by Salt does confuse this simple concept by offering a dizzying array of merging strategies that require settings on both the master and the minions. See Repeatable State with Salt for the settings I use.

Managing Files is the Major Task

This lesson may have dawned on my from using bcfg2, and one might notice this while working with any configuration management: the major task of any system is installing files. Salt and Ansible bundle a massive number of modules, but this is one of the few that take you a long way


/pg_data/10/pg_hba.conf:
  file.managed:
    - source: salt://postgres/pg_hba.conf
    - mode: 640

File source paths are using salt:// pull from the local repository (file root) and http:// will allow you to pull from a web server. I find http very useful for binary files that I build and install on a mirror.

Similarly modifying existing files is very useful


/pg_data/10/postgresql.conf:
  file.replace:
    - backup: False
    - pattern: "^max_connections = 100"
    - repl: "max_connections = 200"

Executing Arbitrary Scripts

Even with a massive set of bundled modules complex tasks require scripting, and Salt lets you add do this in a way that feels natural


postgresql-init:
  cmd.run
    - creates: /pg_data/10/PG_VERSION
    - runas: postgres
    - name: |
        /usr/pgsql-10/bin/initdb -D /pg_data/10

This is an overly simple example, but using a script give you extreme flexibility. postgresql-declaration for does much more here, including logic to auto-upgrade to a newer release.

Custom Library Functions

Dropping files in a directory called _modules gives you functions that can be called from the command line or from templates


import hashlib

def hostname_minute_hash(hostname):
    """
    Generate a deterministic number from 0-59 to be used for randomizing
    cron tasks by host
    """
    m = hashlib.md5()
    m.update(hostname)
    return int(m.hexdigest()[:2], 16) % 59

Like any other module in Salt this can be called directly from the command line


$ salt-call pg_utils.hostname_minute_hash $hostname

But more likely from a template


/etc/cron.d/pg_verify:
  file.managed:
    - mode: 644
    - contents: |
        MAILTO=postgresql-notifications
        {{salt['pg_utils.hostname_minute_hash'](grains.host)}}  *  *  *  *  postgres bin/pg_verify.sh

These module functions can in turn process data defined in a dictionary named __pillar__ which is user or dynamically generated data that can be used for anything; the relationship between a master database and its replicas for example.

Terms such as pillars, grains, and mines refer to capable facilities, but the terminology is unfortunate. These special purpose terms put up a wall to adoption by other team members, and an accurate explanation is not trivial to comprehend.

Round Trip Time

Using the master-minion model Salt can typically run a simple state in 3-5 seconds. In unit testing the speed at which you can obtain a result determines the development model you are able to use, and the rate at which you can make progress. This is also true in configuration management. The ability to iterate on a change makes configuration management nearly interactive.

salt-ssh or ansible-playbook has enormous overhead while trying to stage modules and verify cached data, resulting in a round-trip-time easily reading 15-20 seconds.

The trouble area applies to any large framework: when performance is slow it is difficult to profile and even hard to fix. By eliminating layers of abstraction, rset(1) tries to provide a transparency that enables administrators to get rapid feedback on changes.

Last updated on October 13, 2018