March 11, 2022

Complex groups in Ansible using the constructed inventory plugin

If you ever start working with large, semi-heterogenous groups of VMs and want to manage them with Ansible, you quickly start running into problems trying to make it work with complex groups.

As an example, let's assume we have webservers and jobservers spread out across 2 cloud providers, for 3 different applications, in 3 environments (production, staging, and test). How would you set variables for all webservers for application A? Or all production servers across all providers and applications? Or all webservers, but not jobservers?

The typical solution is to have a lot of duplication in your inventory files. For example:

all:
  children:
    env_prod:
      children:
        provider_gcp:
          children:
            type_webserver:
              children:
                app_a:
                  children:
                    app_a_webservers:
                      hosts:
                        web[01:03].app-a.prod.gcp.mycorp.net:
                app_b:
                  children:
                    app_b_webservers:
                      hosts:
                        web[01:03].app-b.prod.gcp.mycorp.net:
            type_jobserver:
              children:
                app_a:
                  children:
                    app_a_jobservers:
                      hosts:
                        job[01:03].app-a.prod.gcp.mycorp.net:
                app_b:
                  children:
                    app_b_jobservers:
                      hosts:
                        job[01:03].app-b.prod.gcp.mycorp.net:

Imagine many, many more lines like this, for other environments and providers. The only thing we're not repeating in this example is the top-level group (which in this case is env_prod, but it's up to you how to structure your hierarchy) and the lowest level group.

The alternative: Using constructed

# inventory/main.yml
all:
  children:
    app_a_webservers:
      hosts:
        web[01:03].app-a.prod.gcp.mycorp.net:
        web[01:05].app-a.prod.aws.mycorp.net:
        web[01:02].app-a.stag.gcp.mycorp.net:
        web[01:02].app-a.test.gcp.mycorp.net:

The constructed plugin lets you define jinja statements for groups. Basically, if the Jinja statement evaluates to true, the server will be added to this group. Any built-in Ansible variables (not group_vars, host_vars) are available here, so you can check things like the hostname, which user you're using to connect with, which non-constructed groups the server is a part of, and more. inventory_hostname is a useful one, and because it's a string you can do a lot of useful Python operations on it, such as .split and .startswith. You can also use Ansible filters.

# inventory/zz-constructed.yml
plugin: constructed
strict: true

groups:
  provider_gcp: inventory_hostname.endswith('.gcp.mycorp.net')
  provider_aws: inventory_hostname.split('.')[-3] == 'aws'
  env_prod: inventory_hostname.split('.')[-4] == 'prod'
  env_staging: '.stag.' in inventory_hostname
  env_test: inventory_hostname.endswith(('.stag.gcp.mycorp.net', '.stag.aws.mycorp.net'))
  type_webserver: inventory_hostname.startswith('web')
  type_jobserver: inventory_hostname | regex_search('^job\d+\.')
  app_a: inventory_hostname.split('.')[1] == 'app-a'
  app_b: inventory_hostname | regex_search('[a-z0-9]+\.')
June 4, 2020

venv.sh - sensible virtualenv workflow

Virtual environments can be kind of cryptic to people who haven't worked with Python for a while. I'd say that even for people that do work with Python, it can take a long time for it to "click". The short version is, you want one virtual environment for every Python project you work on, which, if you work on a lot of smaller projects, can get annoying.

The standard method of creating and using a virtualenv looks something like this:

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip setuptools

Now you're ready to install the project itself and its dependencies. Kind of verbose, though you do only have to run source .venv/bin/activate in the future - but if you forget whether you've created a virtual environment for a project (directory), it gets more tedious.

Wanting a more ergonomic solution to this, I've worked on some shell functions which I've now put in a Github repository: venv.sh.

It allows me to just run the command venv activate (I've actually aliased it to av so even less typing), and it will activate a virtual environment if one is found in the current working directory, otherwise it will create it for you using the newest version of Python found on your system (only by default, of course).

If you find it annoying to work with virtual environments and other solutions like virtualenvwrapper or direnv don't feel right, check it out.

May 20, 2020

PSM - Python Script Manager

It's fairly common for generally useful CLI tools to be written in Python. Some examples off the top of my head are ranger, streamlink, youtube-dl and glances.

Some of these are available as part of your operating system's package management, but may not be up to date. You can install them using pip, but installing things with pip is a beginner's trap.

If you install using sudo pip, you might overwrite packages on the system and break tools that are required for the operating system to work. This is avoidable by using pip install --user, which installs packages into $HOME/.local instead of globally, but does mean you have to modify your $PATH to include $HOME/.local/bin.

However, if you try to install two CLI tools which require two different versions of a dependency, you're likely to break one of them. The solution to this is virtual environments, which are fairly straight forward to use when working on a specific project, but not so much for installing things like CLI programs.

There are tools like pipsi and pipx which attempt to solve this: simply run pipx install ... and your thing will be installed. There's a chicken-and-egg problem, though: How do you install pipx? It's a python application with dependencies, afte rall. You end up having to make an exception for this specific tool and use pip install --user.

All of this annoyed me, and encouraged me to write PSM - Python Script Manager. It's a stand-alone shell script, so it can just be downloaded and put in a directory in your $PATH.

To install it:

curl https://raw.githubusercontent.com/anlutro/psm/master/install-psm.bash | bash

Some usage examples:

psm install glances
psm upgrade-all

More information can be found on Github: https://github.com/anlutro/psm

May 10, 2020

Asking for help in public or private chat

Here's a pattern I've seen in multiple organizations: If someone is stuck with a problem, they guess who might know the solution to that problem and ask them privately, either in person or over chat (e.g. Slack). I always preferred to be asked over chat rather than in-person because it means I can postpone reading it for 5 minutes if I'm in the middle of something, and now with remote working being pretty much mandatory, the in-person option isn't even there any more.

I think this is a bad habit, and something that should be discouraged especially in engineering departments. First, let's outline the advantages of asking in private, usually the reasons why people default to doing it:

  • You don't feel like you're bothering everyone in the channel who might not care about your problem.
  • You're guaranteed to get someone's attention, your question is very unlikely to be ignored.
  • It feels similar to what you would do in real life: Walk up to a colleague and ask them if they have time to help.

While these points are technically correct, I also think they're based on flawed premises, which we will get back to. For now, let's list the drawbacks of asking for help in private:

  • You have to know who to ask.
  • If you ask the wrong person, you've potentially interrupted them without really achieving anything.
  • If you ask the wrong person and you are referred to someone else, or you just guess someone else who might know, you're asking your question N times which wastes your time. This gets especially annoying if your question isn't a simple copy-and-paste one, for example if there are follow-up Q&A or additional context added after the fact.
  • If the person you ask gives you an answer, that answer may be incorrect or inefficient, and another colleague might be able to correct them - but because your messages are private, they can't.
  • Other colleagues cannot read your question and answers given to your question, and hence cannot learn from it.
  • It gives the impression that no one is asking for help, which can discourage others (especially new joiners) from asking for help.

It's worth noting that this is slightly different when not working remotely - if I walk over to a colleague in the office and ask them for help and we discuss the problem in person, others can overhear our discussion if they're not too busy with other things, and we avoid a lot of the problems mentioned above.

So what should you do instead? Ask in a public chat channel. Even Slack itself claims it's the ideal:

Slack is designed to add transparency to an organization, so it’s best to default to communication in public channels whenever possible. Slack’s own team sends tens of thousands of messages each week—in a recent summary, 70% of those were posted in public channels, with 28% occurring in private channels and just 2% in direct messages. Posting messages in public channels means anyone in the organization can see what various teams are working on, see how much progress people are making on projects, and search the archive for context they need.

Here are some tips for successfully asking for help in public chat channels:

  • If you're not sure which channel is the best fit, just put it in the most generic one. Most engineering companies have channels like #dev-example-app or just #development.
  • If your question is long and you don't want to flood the channel with text, summarize your question in a short paragraph then elaborate on it in a thread. This has the added bonus of encouraging others to use threads as well.
  • Ask your question well: Include the necessary context, what you've already tried, predict what follow-up questions might come - but also don't make it too long or include too many unncecessary details, or people will get overwhelmed and you're less likely to get help.
  • If you're worried that putting your question in a public chat channel will create noise for others, actually verify if this is the case. Are you bothered any time a question is asked in a public channel? Ask some colleagues in one-on-one situations what they think as well. In my experience, no one enables sound or pop-up notifications for channel messages. But if this is actually a problem somehow, consider creating dedicated channel(s) (e.g. #dev-help) for asking for help that can more easily be muted.
  • If, after some time (let's say an hour), you haven't resolved your question, consider mentioning colleagues who you suspect might be able to help (the people you would consider asking for help in private). You can do this either by sending a link to your question in a private message, or mentioning them in the channel/thread itself.
  • If you haven't resolved your question after a long amount of time (let's say 24 hours), simply post the question again and specify that you didn't get an answer within 24 hours.
  • If the question remains unresolved for even longer, you'll either have to escalate to your manager, or accept that no one actually knows and you'll have to figure out for yourself.