May 8, 2016

Managing systemd units with Salt

In many cases, you will want to manage your own systemd service definitions. Here's how.

example_systemd_unit:
  file.managed:
    - name: /etc/systemd/system/example.service
    - source: salt://example/systemd_unit.jinja
    - template: jinja
  module.run:
    - name: service.systemctl_reload
    - onchanges:
      - file: example_systemd_unit

example_running:
  service.running:
    - name: example
    - watch:
      - module: example_systemd_unit

Let's walk through what this does. First we manage the systemd unit, which is just a file ending with .service in the correct directory. You may need to change the path to example.service based on your Linux distribution.

Second we have a module.run state calling service.systemctl_reload, but only when the service file changes. Systemd documentation will tell you that you need to run systemctl reload to apply changes made to service files, this is simply the Salt way of doing that.

Finally, we have a regular service.running. You just need to make sure the name of the service matches the name of your .service file, and also make sure that the every time the service definition changes and service.systemctl_reload gets called, the service also gets restarted. A watch is an implicit require, so we don't need to specify that the service state requires the service file to be present.

March 29, 2016

Dynamic includes in Salt

Writing Salt state files can be somewhat deceptive. They have a concept of includes, which allows you to split up state files and define dependencies, which can give you reduced duplication, a cleaner top.sls and a way to run state files individually without dropping all your requirements. However, unlike Python and other programming languages, the includes don't need (it's not even considered best practice) to be defined at the top of the file. Realizing this opens some opportunities.

For example, consider a state file uwsgi/apps.sls that sets up various uWSGI applications:

include:
  - uwsgi.install

{% for name, app in pillar.get('uwsgi_apps', {}).items() %}
/etc/uwsgi/{{ name }}.ini:
  file.managed:
    - source: salt://uwsgi/files/uwsgi.ini.jinja
    - template: jinja
    - context: { app: {{ app | json }} }
{% endfor %}

Obviously missing from this example is how to get the source code for the uWSGI applications, and setting up a systemd/supervisord service that keeps the app running. Ignore that.

uWSGI apps can be of many types: Ruby, Python (both v2 and v3), Perl, you name it. How do we deal with this? We could just include all the plugin types at the top of the SLS:

include:
  - uwsgi.install
  - uwsgi.plugins.psgi # perl
  - uwsgi.plugins.python2
  - uwsgi.plugins.python3
  - uwsgi.plugins.rack # ruby

{% for name, app in pillar.get('uwsgi_apps', {}).items() %}
...

But it'd be nicer if we could include the plugins dynamically, based on whether any apps use them:

{% set plugins = [] %}
{% for name, app in pillar.get('uwsgi_apps', {}).items() %}
  {% for plugin in app.get('plugins', []) if plugin not in plugins %}
      {% do plugins.append(plugin) %}
  {% endfor %}
/etc/uwsgi/{{ name }}.ini:
  file.managed:
    - source: salt://uwsgi/files/uwsgi.ini.jinja
    - template: jinja
    - context: { app: {{ app | json }} }
{% endfor %}

include:
  - uwsgi.install
{% for plugin in plugins %}
  - uwsgi.plugin.{{ plugin }}
{% endfor %}
March 23, 2016

Wait for a port to be listening in Salt

Sometimes, you want to wait for a service to be running before running other states. Usually this can be done with a service.running state, which is then required by other states. For example, a mysql_database.present state can require the mysql service state, and it won't be ran before the mysql service has been started.

However, sometimes the service can start up but still not be ready to serve requests. I faced this problem with InfluxDB - there would be up to a 1 second delay between service influxdb start and InfluxDB actually listening on all the ports. Because of this, influxdb_user.present states would fail if the service had been restarted due to configuration changes, because the connection to port 8086 would fail.

The solution: Using until and nc/netcat.

influxdb:
  pkg.installed: []
  service.running:
    - name: influxdb
  cmd.run:
    - name: until nc -z localhost 8086; do sleep 1; done
    - timeout: 10
    - onchanges:
      - service: influxdb

influxdb-user-example:
  influxdb_user.present:
    - name: example
    - passwd: example
    - require:
      - cmd: influxdb

What's happening here is that whenever the service gets restarted, that counts as a change in the service.running state. That triggers the cmd.run state, which will be executed synchronosusly - in other words, it'll block other states from being executed until it completes. Then, our states that require the port to be listening simply add a requirement for the cmd.run state.

In Salt 2016.3, you don't even need the cmd: in front of the requirement.

Even if your service doesn't listen on a port, this approach can still be used. All you need is to find some sort of shell comand that either blocks or exits with 1 (or higher) if your service is down, but exits with 0 when your service is up and running and fully operational. Simply replace the nc command in the example above with your command.