lutro.me

Volume in Debian distros (Crunchbang, Ubuntu...)

Tue, 20 Aug 2013 12:00:00 +0100

I've had this problem with my old Lenovo ThinkPad with every Linux installation. When I installed CrunchBang (a great disto, by the way) I had it again and had forgotten how to fix it. Amazingly, by googling around I found my own old blog with a post on how to fix it from all the way back to Ubuntu 9.10. I decided to re-write it a bit and host it here.

With virtually every Debian distribution I've used, my ThinkPad R61 (6 years old and still going strong) has had sound issues. The volume controls have been extremely uneven, there's been lots of distorted sound with anything over 15% etc. Trying to find the solution to this, I opened the command-line sound mixer that Debian distributions use:

alsamixer

Here, it's possible to tell exactly what is happening to your system volumes and you can more easily identify what is causing the problems. You may have to select a different sound card (press F6) to see all the details. In my case, I've had several different problems and solutions over the years - in one of the early Ubuntu distros I had to lock PCM at 47% (0dB gain), and nowadays I just make sure to lock Master at 95%. This will vary from computer to computer, so mess around but try to figure out which one you need to lock.

By default, the ALSA mixer will try to "merge" a bunch of these channels when you try to control volume. This means it will first try to increase one channel, then another, then a third until everything is max. This is not always what you want. Edit the following file - make sure you have root permissions, sudoedit is a good way:

sudoedit /usr/share/pulseaudio/alsa-mixer/paths/analog-output.conf.common

In the file, look for something like this:

[Element Master]
switch = mute
volume = merge

[Element PCM]
switch = mute
volume = merge

The documentation in the same file tells you that "merge" means "merge it into the device volume slider". This is what you want to change. By now you should know which channel you want to lock - for this channel, change "merge" to "ignore". Now, open alsamixer again and adjust the ignored channels volume until you feel your normal volume controls work fine.

Done!

Optimizing for production with Laravel 4

Tue, 20 Aug 2013 12:00:00 +0100

Documentation on optimization and performance is somewhat lacking for Laravel 4 at the moment. In this post I'll give some quick pointers as to how Laravel 4 works and how you can improve its performance.

First of all, realize that the things that apply to normal PHP development also apply to Laravel 4. Built-in PHP functions will be more efficient than trying to do stuff manually, your database layout can and will affect performance and so on.

Laravel uses lazy autoloading for classes. This means that if it doesn't include EVERY file in the vendor directory, but rather waits for a call on a class that hasn't been loaded yet, and then makes some intelligent guesses as to where it might be located. If you're using PSR-0 autoloading, it checks for the file Namespace/Namespace/Class.php in the PSR-0 directories you have defined in your composer.json. If you're using classmap, it simply looks up the name of the class and loads the corresponding file. This is why, if you've added a new class to a directory only being autoloaded via classmap, you get "class not found" until you run composer dump-autoload.

Laravel 4 adds a few directories to the autoloader on run-time by default. Open app/start/global.php and you'll see a call to ClassLoader::addDirectories - classes inside these directories will also be autoloaded using the PSR-0 rule. For performance, you should remove this function call altogether and just rely on Composer's autoloading.

PSR-0 is nice for development because it lets you add a class and it will instantly be found by the autoloader - as long as you've namespaced it correctly and named the file correctly, of course. In production, you usually don't want to use PSR-0 autoloading as it has a bit of overhead. Use composer dump-autoload --optimize to re-compile all your PSR-0 autoloading rules into classmap rules, which are faster.

When debug is set to false in app/config/app.php, Laravel's Artisan function php artisan optimize will do two things: Run the composer dump-autoload --optimize command, as well as generate the file bootstrap/compiled.php. This file contains a lot of the common Laravel framework class files, and allows for the system to just require one file even though the framework is in reality split into many hundreds of files.

Running php artisan optimize every time you update your files is highly recommended as it can increase your performance by a lot. Consider having it done automatically every time you deploy (preferably after a temporary php artisan down to prevent issues).

You can add files to this compiled file by adding them to the array in app/config/compile.php. Files should be referenced relative to the project root - for example:

'vendor/laravel/framework/src/Illuminate/Support/Collection.php',
'vendor/cartalyst/sentry/src/Cartalyst/Sentry/Sentry.php',
'app/library/HelperClass.php',

Use a profiler like loic-sharma/profiler to find out which files are being loaded on your requests and add them to your compiled.php array. I've had my response time almost halved by doing this.

And as a last note - never have workbenches on your production server. There is a lot of overhead related to workbenches, and they should only be used for local development.

Assertions on method calls using Mockery

Tue, 20 Aug 2013 12:00:00 +0100

I had a problem recently where I wanted to use PHPUnit to test what was being passed to a function. We're talking about a several hundred line string with a current timestamp, so simply using $mock->shouldReceive('method')->with('parameter string') wouldn't work.

One option would be to write a mock implementation of the class that should receive the method call which stores the method parameter in a public variable on the class and then do the assertions on that, but I usually try to avoid solutions like that - especially when it's just for doing simple assertions on a string.

The first solution I found was using Mockery's andReturnUsing method, which passes the parameters passed to the method on to a closure. The following works in PHP 5.4 or later:

$this->logger->shouldReceive('error')->once()
    ->andReturnUsing(function($logged) {
        $this->assertContains('Route: action', $logged);
        $this->assertContains('URL: url', $logged);
    });

The problem with this is that you can't chain andReturnUsing with a normal andReturn method - meaning if the method call should return anything, it needs to do so inside the closure. This can be problematic if you want to check the method call parameters in the same way multiple places but return different values - with andReturnUsing you'd have to redefine the same closure over and over again, only changing the return statement.

A little research revealed that the Mockery::on function is what I'm looking for:

$this->logger->shouldReceive('error')->once()
    ->with(Mockery::on(function($logged) {
        $this->assertContains('Route: action', $logged);
        $this->assertContains('URL: url', $logged);
        return true;
    }));

If closure you pass to Mockery::on should returns false, you'll get a Mockery\\Exception\\NoMatchingExpectationException: No matching handler found for class::method exception. We don't really care about that, the assertions will do what we want and tell us if something is wrong.

This way, you can even call a closure on a specific parameter while simply leaving the other parameters as variables as you normally would - and you can add ->andReturn at the end of the chain.

However, using $this in a closure doesn't work in PHP 5.3 and older. For that, you need to use a reference to an outside variable which we will then later do assertions on.

$this->logger->shouldReceive('error')->once()
    ->with(Mockery::on(function($input) use(&$logged) {
        $logged = $input;
        return true;
    }));

$this->handler->handleException($exception);

$this->assertContains('Route: action', $logged);
$this->assertContains('URL: url', $logged);

Notice how we had to delay the assertions until after the event chain has been triggered (in this case, I was testing an exception handler) - you can't assert on $logged until the closure has actually been invoked.

An example I posted on StackExchange, testing mails in Laravel 4: http://stackoverflow.com/a/18431205/2490608

Testable, simple L4 code without repository patterns

Sat, 31 Aug 2013 12:00:00 +0100

There's a lot of people advocating the repository pattern for testability in your Laravel 4 projects. Fact is, it doesn't make your code that much more testable, and you can easily achieve the same level of testability by using your models as you would a repository.

First of all, let me say that repositories are very handy if you find yourself either chaining tons of query builder function calls in your controller, or defining a lot of static functions on your models. A repository can then be a handy layer between your controllers and models which gathers all that clutter into one class, where you can name your functions something like searchAndFilterWithRelatedModels.

I'm not a huge fan of the pattern and you can probably find others who can argue for it better than me...

Anyway, on to my main point. The point that many people have been making is that by injecting a repository into your controllers, you gain testability because you can swap out the repository with a mock. Unless you're using something extremely high-tech like AspectMock to intercept function calls, you won't be able to do the same if your controller passes Post::all() to your view. What most people don't seem to tell you is that you can do the exact same dependency injection with your models, and in most cases it's a simple search and replace to swap out your static class calls with the new injected method.

Here's a very simple controller example. I've commented out the "old" code.

class MyController extends Controller
{
    protected $model;

    public function __construct(MyModel $model)
    {
        $this->model = $model;
    }

    public function index()
    {
        // MyModel::all();
        return View::make('myindex', [
            'models' => $this->model->all()
        ]);
    }

    public function show($modelId)
    {
        // MyModel::find($modelId);
        return View::make('myshow', [
            'model' => $this->model->find($modelId)
        ]);
    }
}

The magic that happens here is that Laravel automatically hands the controller an instance of MyModel. You'll see how we can manipulate Laravel into passing it a mock object instead of a real model later on.

The reason we can simply swap MyModel:: with $this->model is because of the way Eloquent models handle static method calls. Every static method call on a model basically spins up a new instance of that model and then does a normal method call on that new instance. There are a few exceptions, but none that should matter in a controller context.

Here is the corresponding test. We'll define the mock in the setUp method as we'll be re-using it practically for every test in the file. Notice how we create a mock object where we can set expectations etc., and then tell Laravel ($this->app) to use that instance whenever it's asked for an instance of MyModel.

class MyControllerTest extends TestCase
{
    public function setUp()
    {
        parent::setUp();
        $this->mockModel = Mockery::mock('MyModel');
        $this->app->instance('MyModel', $this->mockModel);
    }

    public function tearDown()
    {
        Mockery::close(); // important when using mockery!
    }

    public function testIndex()
    {
        // return an empty array because our index view has a foreach
        // loop that would error if we returned something non-iterable
        $this->mockModel->shouldReceive('all')
            ->once()
            ->andReturn([]);

        $this->call('get', '/my-route');

        $this->assertResponseOk();
        $this->assertViewHas('models');
    }

    public function testShow()
    {
        // this is the best way to mock a real model to pass to a
        // view without having to add ->shouldReceive for every
        // single function and defining every single variable on it.
        $mock = Mockery::mock(new MyModel);

        $this->mockModel->shouldReceive('find')
            ->once()
            ->with(1)
            ->andReturn($mock);

        $this->call('get', '/my-route/1');

        $this->assertResponseOk();
        $this->assertViewHas('model');
    }
}

Maybe there are times when this isn't appropriate - if your controllers use a lot of different models that all need to be injected and there's no way to move those onto relationships, there might be a slight performance loss? I doubt it's big though - and the testability you gain from practically no effort is huge.

When to use repositories?

Mon, 30 Sep 2013 12:00:00 +0100

Repositories have their place in applications that deal with fetching stuff - either if it's from a database or an external source. In the Laravel world the repository pattern has been praised a bit too much for its advantages in terms of testability and architecture. I've written about how you can achieve the same level of testability without repositories, but sometimes repositories really are recommended.

A lot of articles about the repository pattern mix in stuff that doesn't have with repositories to do at all like PSR-0 autoloading and dependency injection, which I won't go into depth in this article.

Let's assume you have a controller that's being bogged down with logic. You have several query builders, some spanning as much as 10 lines. This sort of thing does not belong in the controller and is generally making it hard to navigate the source code. Let's take action by moving it all into a third class that lies between the models and the controller - a repository.

A repository class is just a pure class. It doesn't need to extend anything, it's just a collection of functions really. A lot of tutorials will have you creating an interface for the repository - this is absolute nonsense unless you're writing a package for distribution or are writing a database-agnostic app (99% chance you aren't) - so all we're going to do is create a new file named ThingRepository.php and stick it somewhere that's being autoloaded by composer.

If you don't know how to do this - make a new folder inside app, let's name it repositories. Put the repository PHP file there. Next, open composer.json and look for the list of classmap autoloaded directories like app/models and app/controllers. Add app/repositories to this list and run composer dump (short for dump-autoload) from the command line. You'll need to do this every time you add or rename a class to this directory.

class ThingRepository
{
    public function getAllThings()
    {
        return MyModel::all();
    }
}

Obviously this is a very simple example, but you can easily imagine how to create your own functions that "hide" your 10 line long query builders from the controller. Now, let's utilize this class in our controller.

class MyController extends Controller
{
    public function index()
    {
        $repository = new ThingRepository;
        $things = $repository->getAllThings();
        return View::make('myview', ['things' => $things]);
    }
}

And this will work! It's a little ugly though - we have to instantiate a new ThingRepository in every function. Let's store it on the class.

class MyController extends Controller
{
    protected $repo;

    public function __construct()
    {
        $this->repo = new ThingRepository;
    }

    public function index()
    {
        $things = $this->repo->getAllThings();
        return View::make('myview', ['things' => $things]);
    }
}

You could also utilize dependency injection if you want! This makes the code more testable and flexible for a variety of reasons I won't go into in depth because this post is about repositories and the benefit they give!

class ThingRepository
{
    protected $model;

    public function __construct(MyModel $model)
    {
        $this->model = $model;
    }

    // ...
}

class MyController extends Controller
{
    protected $repo;

    public function __construct(ThingRepository $repo)
    {
        $this->repo = $repo;
    }

    // ...
}

And there you have it. No need for PSR-0 autoloading, namespacing or dependency injection if you don't want. Repository classes are not magic and are fairly simple when you get down to the core of it.

Check out a neat base repository class you can use here!

Using the Form Builder as little as possible

Sat, 05 Oct 2013 12:00:00 +0100

I love the Laravel 4 FormBuilder (accessible through the Form:: facade) - automatic re-populating of input from the session and from a model is awesome. In the upcoming versions you will even be able to use accessors to populate form fields from the model even if they're not actually fields in the database.

However, sometimes (especially when using a stylesheet framework) it can be hard to get the markup to behave as you want when constructing a form. If you want to construct the HTML for an input yourself but still reap the benefits of automatic repopulation from session/model, there is the function getValueAttribute which lets you get the value of a certain input in your form either from the session or the model.

{{ Form::model($model, ['class' => 'form-horizontal', 'role' => 'form']) }}

<input type="text" name="my_field" value="{{ Form::getValueAttribute('my_field', 'Default') }}" />

{{ Form::close() }}

Make sure the input name attribute and the key you use for getValueAttribute match. We still need to use Form::model if we want population from a model, but a Form::open can easily be replaced with normal HTML as well. Keep in mind the priority of data when forms are populated - session comes before 'Default' which again comes before model data.

Mocking models using Mockery

Fri, 17 Jan 2014 12:00:00 +0100

Mocking classes and defining what methods they should receive is easy.

// MyRepository
public function getAll()
{
  return $this->model->get();
}

// MyRepositoryTest
public function testGetAll()
{
  $model = Mockery::mock('MyModel');
  $repo = new MyRepository($model);
  $model->shouldReceive('get')->once()->andReturn('foo');
  $this->assertEquals('foo', $repo->getAll());
}

However, what if you want to test a method/class that not only calls method on that class but relies on or sets variables on it? Often you'll find that Mockery throws errors like 'getAttribute does not exist on this mock object'. The answer is simple - use Mockery's partial mocks.

A partial mock is what it sounds like - parts of the class is mocked, but other parts of it is not and functions as the class normally would. In this case we're making a partial mock because we want the attribute setting to work as normal, but we also don't want the real save() method to be called as that would write to our database.

// MyRepository
public function updateStuff(MyModel $model)
{
  if (!$model->exists) throw new Exception;
  if ($this->model->where('foo', '=', $model->foo)->exists()) return false;
  $model->foo = 'foo';
  $model->bar = 'bar';
  return $model->save();
}

// MyRepositoryTest
public function testUpdateSomeStuff()
{
  $model = Mockery::mock('MyModel');
  $model->shouldReceive('where->exists')->andReturn(false);
  $repo = new MyRepository($model);
  $model = Mockery::mock('MyModel')->makePartial();
  $model->shouldReceive('save')->once()->andReturn(true);
  $model->exists = true;
  $this->assertTrue($repo->updateStuff($model));
  $this->assertEquals('foo', $model->foo);
  $this->assertEquals('bar', $model->bar);
}

It is worth noting that in clean OOP it is better to simply wrap the logic shown in updateStuff in a method on the model and make the repository use that, but I'll leave it as is for the sake of this example. Partial mocks are useful in other places than this - sometimes you may want to send a partial mock to a view to test the view, sometimes it may be more work to mock out a method of a class than let it do its normal thing.

Partial mocks should in general be avoided when possible. If you need a lot of partial mocks in your tests it might be an indication that your classes are doing too much, and every time you make a partial mock to inject into another class you can not easily be confident that the two classes are only loosely decoupled. That being said, they are an extremely powerful tool - but use them with caution.

Russell

Mon, 17 Mar 2014 12:00:00 +0100

Over the weekend I had a fun little project - writing a tiny static HTML blog generator.

Feeling sick of CMSes and the one I was using in particular not support Postgres, I decided I wanted to switch to a static file system - no backend to care about.

I looked at existing solutions but none of them really spoke to me. Most of the popular ones are written in Ruby, and I will not install a bloated gem (let alone Ruby itself) just to create some html files. I looked at python solutions like pelican but they looked bloated as well. Why would you need more than ~10 files of source code for a static site generator?

So I wrote my own. Named completely at random by something funny that happened the night I had the idea, Russell is a static blog HTML generator written in Python 3. It is roughly 200 lines of code, requires a few packages (jinja2, markdown, slugify and docopt) and can be installed from PIP:

pip install russell

The command russell is now available on your command line. russell -h will show the help screen.

My own blog is of course now generated using Russell, and I am very happy with it. Head over to the Github page for more information. I would love to hear your feedback if you decide to try it out!

Assertions on mails in Laravel 4

Sat, 05 Jul 2014 12:00:00 +0100

Testing mails in Laravel 4 is a bit of a weak spot. You can say Mail::shouldReceive('send')->once()... but specifying everything the method should receive in terms of arguments as well as asserting that the closure sets the recipient and subject correctly is tedious at best. This SO answer shows an example of how to unit test a mail being sent.

There is a better way, as long as you're doing functional testing - that is, extending the TestCase that comes with Laravel and doing $this->call(...) stuff. We mock one layer deeper, the SwiftMailer service, and gain access to more rich information.

First of all, we swap the swiftmailer instance on the IoC container.

$mock = Mockery::mock('Swift_Mailer');
$this->app->make('mailer')->setSwiftMailer($mock);

Second, we define its expectations, and use Mockery's andReturnUsing to run assertions on the method call.

$mock->shouldReceive('send')->once()
  ->andReturnUsing(function($msg) {
    // assert stuff here
  });

$msg above is an instance of Swift_Message, a class which API is buried deep within the SwiftMailer class hierarchy, but I'll show the most important ones here.

$this->assertEquals('My subject', $msg->getSubject());
$this->assertEquals('foo@bar.com', $msg->getTo());
$this->assertContains('Some string', $msg->getBody());

If you're sending an email that is both HTML and plain text, you may want to get both the regular HTML body as well as the plain text one.

$htmlBody = $msg->getBody();
$children = $msg->getChildren();
$plainBody = $children[0]->getBody();

This should allow you to write shorter, more accurate and realistic tests.

Laravel and SemVer

Mon, 01 Sep 2014 12:00:00 +0100

There's been some discussion around Laravel and SemVer (semantic versioning) recently, which I appreciate. SemVer is in my eyes a very important guideline for frameworks and widely adopted libraries.

A major concern when picking frameworks is, am I going to get security and feature upgrades without risking breaking my application? In laravel, you currently don't. Each minor version (4.1, 4.2, 4.3) has introduced breaking changes - there have even been breaking changes in "bugfix" versions because new features are introduced here.

In addition to this, critical bugfixes are not being backported to older minor versions of the framework, so if you want to stay secure you pretty much have to upgrade.

Maybe this isn't a problem for a lot of people, but my experience is that in Laravel, a lot of things are hard coded, so for large applications with specific needs you often need to extend core classes. This makes the application more likely to break when you upgrade, and requires more time to fix.

If you're building a large application you also likely have your own sort of framework built on top of Laravel again, maybe you're depending on packages specific to Laravel - these are a lot of points that are vulnerable to breaking.

In fact, as a package maintainer myself, I often have to consider if I need to ditch the generic "~4.1" version constraint and exclude the specific minor versions that I know don't work 100% with the package.

Following semver makes it easier to catch bugs and problems that aren't catchable with regular unit tests. If your new minor version requires an hour's work or so to upgrade, and potentially break existing functionality (sometimes on purpose), few people are going to want to beta test that. If instead you make the promise that "change the version constraint in your composer.json - you won't have to change anything in your code, and if you do, it's a bug" a lot more people would be open to beta testing, I think.

If Laravel were to use semver, it shouldn't do so "just because". You would have to either release a new major version every time you make a breaking change, exposing how volatile the framework really is, or make an effort to make less breaking changes, enforcing new features to only be introduced in minor versions and bugfixes in bugfix versions. While following semver 100% is practically impossible, making a dedicated effort in that direction would make a welcome change.

Shrinking controllers and abstracting with manager classes

Fri, 17 Oct 2014 12:00:00 +0100

There are a lot of articles revolving around Laravel 4 specifically that try and explain how to abstract logic away from your controller, but most of them either kinda miss the point or overcomplicate things (by using facades, repositories, interfaces...).

The only thing you need to know before following this guide is how to create your own custom classes and make composer autoload them. This is a fairly simple thing to do which I won't cover here.

The simplest way to abstract logic away from your controllers is to grab all the logic that isn't directly tied to any of these things:

Fetching input (Input::all())
Returning a response (usually a redirect or a view)
Putting stuff into the session (flash messages)

Everything else - validation, mail, queue and database stuff - you should copy into a class called a manager. If your controller is called ThingController, your manager should be called ThingManager. Do not worry about using static method calls and facades within the manager class - our main focus for now is to move business logic away from the controller.

Once the manager class has been made, inject it into the controller simply by typehinting the constructor argument. Laravel's IoC container takes care of the rest for us. Now, we can call our manager's methods within the controller - for example, $this->thingManager->myMethod(Input::all()) and do something with what is returned. For example, we may redirect to two different URLs depending on whether it returned true or false.

Here's an example of what a refactored controller might look like.

SASS tests with a simple Makefile

Sun, 14 Dec 2014 12:00:00 +0100

Working with increasingly complex SASS mixins and functions recently, I wanted to set up some sort of test suite to check the CSS output of various files. Rather than bother with some silly NPM package/Ruby gem, I figured I might as well just use some basic shell commands that can be placed in a Makefile (which I already had anyway).

Assume you have your SASS files placed in a directory named sass. Create the directory sass/tests - this is where we'll place our test files.

We'll create one .sass file for each feature we want to test, as well as a file with the same name, but with the extension .expected.css - which, as you may have guessed, will contain the expected CSS output of the SASS file.

We can leverage the diff program to check for differences in output. We'll pipe the SASS compiler's output to it, and make it compare to the .expected.css file, like this:

css.test:
    for f in sass/tests/*.sass; do \
        sassc $$f | diff - $${f%.sass}.expected.css \
        && echo OK: $$f; done

${f%.sass} is a neat bash trick to remove an extension from a path string. The output on success will look something like this:

$ make css.test 
for f in style/tests/*.sass; do \
    sassc $f | diff - ${f%.sass}.expected.css \
    && echo OK: $f; done
OK: style/tests/columns.sass

Output on failure will look something like this:

$ make css.test 
for f in style/tests/*.sass; do \
    sassc $f | diff - ${f%.sass}.expected.css \
    && echo OK: $f; done
4c4
<   width: 27.33333%;
---
>   width: 29.33333%;
Makefile:130: recipe for target 'css.test' failed
make: *** [css.test] Error 1

Git(hub) - Push via SSH, pull via HTTP(S)

Fri, 26 Jun 2015 12:00:00 +0100

I work with public Github repositories a lot, and get super annoyed because I want to push with my SSH key (because I'd rather put in my key's password than my Github username/password), but I want to pull with HTTPS (because then I don't have to put in a username or password). Normally, the way you do this is:

git clone https://github.com/foo/bar.git
cd bar
git set-url origin git@github.com/foo/bar.git --push

However, I found a really cool way of doing this in all your repositories, without having to do anything each time you clone a repository. Add the following to your ~/.gitconfig:

[url "git@github.com:"]
pushInsteadOf = https://github.com/

This will replace "https://github.com" with "git@github.com" in the remote URL, but only when pushing.

Hints towards learning SaltStack

Sun, 28 Jun 2015 00:19:38 +0100

SaltStack is an awesome provisioning tool I've been implementing in the past few months. I'd like to share a few pointers to other people working with it for the first time.

Know Python. A lot of Jinja logic will borrow heavily from Python, and to best understand a lot of Salt's inner workings you need to understand how Python works. You need to know the difference between a dict and a list, and you need to know how to iterate over a dict.

Understand the data flow. When you're doing foo: {{ foo }} in an SLS file, it's not like passing a variable to a function. Your variable gets cast to a string before it's parsed as YAML. A lot of weird stuff can happen in the process. For example, foo: {{ 'yes' }} will result in the python dict {'foo': True} because "yes" is a boolean constant in YAML.

Learn the difference between states and modules. State functions are the ones you put into your SLS files (pkg.installed, file.managed etc.), module functions are called via {{ salt['module.function'] }}. pillar.get is a very commonly used module function. When you run salt '*' state.highstate you're actually calling the module function state.highstate.

Requisites are for state IDs/names, not files, services, packages etc.. When you have a require: [ pkg: nginx ] in your state, you're not requiring the package nginx to be installed - you're requiring a state with the ID or name "nginx" and of the type "pkg" (pkg.installed, for example). The same applies to files and services.

Use formulas as examples, not plugins. Many of the SaltStack formulas are either broken, overcomplicated or just not suitable for your use case. Use them for inspiration and learning, feel free to copy-paste bits from them, but manage your own formulas.

Consider pillars arguments for your states. Your states should contain instructions on how things are to be done, pillars are what needs to be done. What needs to be done may vary from server to server, but how it's done probably won't. If it does, add pillar entries that specify this varying behaviour (for example, versions of sotftware, whether to compile from source etc.).

Multiple pillar files can add to the same pillar dict. You can have foo.sls and bar.sls both adding an element to a dictionary in the pillar, and they'll be merged on compilation. Even works on nested dicts - but does not work with lists!

SaltStack is a fast moving project with a lot of intertwined functionality. Things break quite often, and the code is often quite a mess of patches. Expect there to be bugs, report them on Github.

Installing a virtualenv bin globally

Sat, 10 Oct 2015 09:54:43 +0100

TLDR: Symlink /path/to/virtualenv/bin/my-script to a directory in your $PATH, such as ~/.local/bin/my-script

Sometimes you want to install a python script so you can run it anywhere, but it has dependencies, so you can't just download it and run it as-is. While pip installing technically does work, the script's dependencies might clash with other python script's dependencies, or you may just not want to clutter the system with packages installed with package managers that aren't your system's native one.

Virtualenvs are usually a good way to deal with this, but you have to manually activate the virtualenv before running your script. Or do you?

Interestingly, python seems to look for packages/includes/whatever relative to the path of the python binary. If we run a python script with /usr/bin/python, the virtualenv's packages won't be pulled in, but if we run it with /path/to/virtualenv/bin/python, they will.

$ /usr/bin/python ./bin/salt --version
[ ... ]
ValueError: Expected version spec in [ ... ]
$ ./bin/python ./bin/salt --version
salt 2015.8.0-154-g4a69db2 (Beryllium)

If we look at the shebang line of a script file in our virtualenv's bin directory, we can see that it specifies the absolute path to the python bin file in our virtualenv:

$ head -1 bin/salt
#!/home/andreas/code/python/salt/bin/python2

This means that this particular script will be able to run from anywhere. This also means we can copy or symlink it to any directory in $PATH, and we will be able to run it as if it was globally installed.

Personally, I have ~/.local/bin added to my $PATH, but you could also use ~/bin or /usr/local/bin.

$ ln -s ~/code/python/salt/bin/salt ~/.local/bin/
$ which salt
/home/andreas/.local/bin/salt
$ salt --version
salt 2015.8.0-154-g4a69db2 (Beryllium)

I'd also like to mention pipx and psm, both of which aim to make this process easier for end-users.

Cache busting URLs with Flask and Nginx

Sat, 16 Jan 2016 08:30:48 +0100

In this post, I'll show you how to effectively override Flask's url_for function in order to add a timestamp to static asset URLs, as well as setting up Nginx to serve cache busted URLs.

import flask
app = flask.Flask()

def cache_busting_url_for(static_dir, old_url_for, bust_extensions=True):
    def transform_filename(orig_filename):
        file_path = os.path.join(static_dir, orig_filename)
        if os.path.isfile(file_path):
            timestamp = int(os.stat(file_path).st_mtime)
            directory, filename = os.path.split(orig_filename)
            filename, extension = filename.split('.', 1)
            if bust_extensions is True or extension in bust_extensions or \
                    extension.split('.')[-1] in bust_extensions:
                filename = '{}.{}.{}'.format(filename, timestamp, extension)
                return os.path.join(directory, filename)
        return orig_filename

    url_cache = {}

    def url_for_wrapper(endpoint, **values):
        if endpoint == 'static':
            filename = values.get('filename')
            if filename:
                if filename not in url_cache:
                    url_cache[filename] = transform_filename(filename)
                values['filename'] = url_cache[filename]
        return old_url_for(endpoint, **values)

    return url_for_wrapper

flask.url_for = cache_busting_url_for(app.static_folder, flask.url_for)

cache_busting_url_for is a function that returns another function. In this example, we've omitted the bust_extensions argument, which means every single static URL will get timestamped. In reality, you'll probably want to pass a list or tuple of extensions that you want to cache bust.

Next, we need to set up our Nginx configuration to rewrite cache busted URLs to real URLs ("static/style.1452929749.css" to "static/style.css").

location /static {
    location ~* ^(.+)\.\d+\.((min\.)?js|css(\.map)?)$ {
        try_files $uri $1.$2;
    }

    # more nginx configuration for static files here
}

In this regex, you need to make sure that whatever file extensions you want to cache bust are matched correctly. In my case, I want both minified and not minified JS/CSS files to match, as well as their corresponding source maps.

\.\d+\. is the regex that matches the timestamp that we added to our asset URLs. Notice how it's not captured by any regex groups (denoted by parenthesis), so when we rewrite the url to $1.$2, the timestamp has been removed.

Dangers of targetting grains in Salt

Tue, 01 Mar 2016 23:32:09 +0100

Targetting grains is probably the most widespread bad practice in Salt. It helps reduce verbosity and duplication in your top files, but also opens up some serious security holes in the event that a minion should be compromised.

A typical grain-focused setup would be something like this:

# a minion's /etc/salt/grains
roles:
  - webserver
  - database

# top.sls for states and/or pillars
base:
  {{ grains.id }}:
  {% for role in grains.get('roles', []) %}
   - {{ role }}
  {% endfor %}

If a minion gets compromised (someone gets root access who shouldn't have), that means the grains file could be edited. Depending on your setup, this could have various security implications.

If your pillar top file targets grains, sensitive pillar data like usernames, passwords and private keys (even for roles unrelated to the host's real role) would be easily obtainable. If you also use grains to determine which customer or which cluster the host is part of, a server could gain access to other customers'/clusters' data.

If you use the mine for basic service discovery, a compromised server could also make other servers start trying to connect to it. For example, it could fake itself as a RabbitMQ server, or a database server, and receive a bunch of data that may contain sensitive information.

Worst case scenario: A compromised host is able to change itself into being a salt master, which all your minions connect to because you use the mine as service discovery, giving the attacker full access to all your minions.

What to do instead?

Match on the minion ID/hostname whenever practical. A database server will always start with "db", an application server will always start with "app", and so on. The customer/cluster name will be another part of the minion ID, as will the environment and datacenter of the host.

If you want to avoid duplication between state and pillar top.sls files, you can consider rendering your state top.sls based on pillar data, like this:

# pillar top.sls
'saltmaster*':
  - roles.salt_master
'*':
  - roles.salt_minion

# pillar roles/salt_master.sls
states:
  salt.master: true

# pillar roles/salt_minion.sls
states:
  salt.minon: true

# state top.sls
{{ grains.id }}:
  {% for state in pillar.get('states', []) %}
   - {{ state }}
  {% endfor %}

Note how we use hostname matching to apply roles (roles being simply a set of pillar data). We then define which states the role includes in these role pillar files - we use a dictionary because unlike lists (at least until very recently), dictionaries are merged instead of overwritten. It also enables us to write a pillar to disable a state set by another pillar, if necessary.

As opposed to grains, this is secure, because pillar data is rendered entirely on the master side and cannot be tampered with by the minion.

There will be some exceptions, situations where the hostname alone simply can't convey enough information, and you don't want to enter each individual host into your top file. Grains can be an option here, but you need to decide whether it can be a security risk if the grain gets changed. For example, your app servers may run various types of web applications:

'app* and G@app_type:php':
  - match: compound
  - roles.webserver_php
'app* and G@app_type:python':
  - match: compound
  - roles.webserver_python

This is relatively safe because installing PHP instead of/in addition to Python on an app server won't damage anything outside of the compromised host. Also, we explicitly state which pillar/state to include, instead of blindly accepting the grain value.

Finally, if you use an external pillar to store your data instead of the top file structure, you automatically solve a lot of the problems you were trying to solve by using grains in the first place. Using a relational database, for example, means you could write a nice admin interface for managing all the different servers, and duplication of data is no longer a big deal. You can also do custom data processing in the external pillar python module.

Wait for a port to be listening in Salt

Wed, 23 Mar 2016 19:22:20 +0100

Sometimes, you want to wait for a service to be running before running other states. Usually this can be done with a service.running state, which is then required by other states. For example, a mysql_database.present state can require the mysql service state, and it won't be ran before the mysql service has been started.

However, sometimes the service can start up but still not be ready to serve requests. I faced this problem with InfluxDB - there would be up to a 1 second delay between service influxdb start and InfluxDB actually listening on all the ports. Because of this, influxdb_user.present states would fail if the service had been restarted due to configuration changes, because the connection to port 8086 would fail.

The solution: Using until and nc/netcat.

influxdb:
  pkg.installed: []
  service.running:
    - name: influxdb
  cmd.run:
    - name: until nc -z localhost 8086; do sleep 1; done
    - timeout: 10
    - onchanges:
      - service: influxdb

influxdb-user-example:
  influxdb_user.present:
    - name: example
    - passwd: example
    - require:
      - cmd: influxdb

What's happening here is that whenever the service gets restarted, that counts as a change in the service.running state. That triggers the cmd.run state, which will be executed synchronosusly - in other words, it'll block other states from being executed until it completes. Then, our states that require the port to be listening simply add a requirement for the cmd.run state.

In Salt 2016.3, you don't even need the cmd: in front of the requirement.

Even if your service doesn't listen on a port, this approach can still be used. All you need is to find some sort of shell comand that either blocks or exits with 1 (or higher) if your service is down, but exits with 0 when your service is up and running and fully operational. Simply replace the nc command in the example above with your command.

Dynamic includes in Salt

Tue, 29 Mar 2016 20:28:06 +0200

Writing Salt state files can be somewhat deceptive. They have a concept of includes, which allows you to split up state files and define dependencies, which can give you reduced duplication, a cleaner top.sls and a way to run state files individually without dropping all your requirements. However, unlike Python and other programming languages, the includes don't need (it's not even considered best practice) to be defined at the top of the file. Realizing this opens some opportunities.

For example, consider a state file uwsgi/apps.sls that sets up various uWSGI applications:

include:
  - uwsgi.install

{% for name, app in pillar.get('uwsgi_apps', {}).items() %}
/etc/uwsgi/{{ name }}.ini:
  file.managed:
    - source: salt://uwsgi/files/uwsgi.ini.jinja
    - template: jinja
    - context: { app: {{ app | json }} }
{% endfor %}

Obviously missing from this example is how to get the source code for the uWSGI applications, and setting up a systemd/supervisord service that keeps the app running. Ignore that.

uWSGI apps can be of many types: Ruby, Python (both v2 and v3), Perl, you name it. How do we deal with this? We could just include all the plugin types at the top of the SLS:

include:
  - uwsgi.install
  - uwsgi.plugins.psgi # perl
  - uwsgi.plugins.python2
  - uwsgi.plugins.python3
  - uwsgi.plugins.rack # ruby

{% for name, app in pillar.get('uwsgi_apps', {}).items() %}
...

But it'd be nicer if we could include the plugins dynamically, based on whether any apps use them:

{% set plugins = [] %}
{% for name, app in pillar.get('uwsgi_apps', {}).items() %}
  {% for plugin in app.get('plugins', []) if plugin not in plugins %}
      {% do plugins.append(plugin) %}
  {% endfor %}
/etc/uwsgi/{{ name }}.ini:
  file.managed:
    - source: salt://uwsgi/files/uwsgi.ini.jinja
    - template: jinja
    - context: { app: {{ app | json }} }
{% endfor %}

include:
  - uwsgi.install
{% for plugin in plugins %}
  - uwsgi.plugin.{{ plugin }}
{% endfor %}

Managing systemd units with Salt

Sun, 08 May 2016 07:58:11 +0200

In many cases, you will want to manage your own systemd service definitions. Here's how.

example_systemd_unit:
  file.managed:
    - name: /etc/systemd/system/example.service
    - source: salt://example/systemd_unit.jinja
    - template: jinja
  module.run:
    - name: service.systemctl_reload
    - onchanges:
      - file: example_systemd_unit

example_running:
  service.running:
    - name: example
    - watch:
      - module: example_systemd_unit

Let's walk through what this does. First we manage the systemd unit, which is just a file ending with .service in the correct directory. You may need to change the path to example.service based on your Linux distribution.

Second we have a module.run state calling service.systemctl_reload, but only when the service file changes. Systemd documentation will tell you that you need to run systemctl reload to apply changes made to service files, this is simply the Salt way of doing that.

Finally, we have a regular service.running. You just need to make sure the name of the service matches the name of your .service file, and also make sure that the every time the service definition changes and service.systemctl_reload gets called, the service also gets restarted. A watch is an implicit require, so we don't need to specify that the service state requires the service file to be present.

Using a puppet-control repo in Vagrant

Wed, 12 Jul 2017 11:11:43 +0200

Whether you use Puppet Enterprise or r10k, using a "control repo" with a branch for every environment is the way you want to set up Puppet these days. Finding a way to make this work well with Vagrant for local development was surprisingly difficult - most guides out there focus on a very simple puppet setup with no modules, or maybe assuming that puppet is installed on the host operating system. I wanted to write a bit about the things I discovered while experimenting trying to get a proper setup up and running.

This is not meant as an introduction to puppet or vagrant - you might want to read up on how to use these tools before starting this article, as I won't go into detail on how puppet or vagrant configuration works..

I'll assume you already have a puppet-control repository. If you don't, have a look at this template repo.

Modifications to the control repo

First of all, you probably want to add .gitignore rules in your control repo for node-specific hieradata for vagrant files, so that you can modify these files as much as you want. If you use the default hierarchy and make sure that all your vagrant hostnames end with .vagrant it would look like this:

/hieradata/nodes/*.vagrant.yaml

Also make sure to add some sort of generic vagrant hiera file which applies to all vagrant machines. We set a provider custom fact which is set to "vagrant" for vagrant machines, and then load the hiera file providers/%{facts.provider}.yaml, but if you can think of another way of setting generic hiera data for vagrant machines, you can do it however you want.

How we'll run Puppet

By default, r10k creates one environments for every git branch. This is rather nice for deploying things remotely, but for developing locally, this means we'd have to commit and run a deploy command before any change we make becomes "public" to the Vagrant machines. This is too slow for us, so we will be using r10k sparingly - mostly just to install modules. We could actually use puppet- librarian instead and get module dependency management, but we'll stick with r10k to stay consistent with our production environment.

We'll create a "fake" environment called "vagrant", which all of our VMs will use (configured through puppet.conf). This environment will be a plain directory on the VM's filesystem, and we'll simply invoke puppet using puppet apply.

Creating the Vagrant repo

We'll create a new git repo which contains the Vagrant configuration:

A Vagrantfile
Provisioning scripts
Puppet configuration
r10k configuration

The control repo can exist inside of this vagrant repo (make sure to .gitignore it!) or outside. The important thing here is to share the correct directories in the Vagrantfile:

config.vm.share './control', '/etc/puppetlabs/code/environments/vagrant'
config.vm.share './puppet', '/etc/puppetlabs/puppet'
config.vm.share './r10k', '/etc/puppetlabs/r10k'

Configuration files

We do not need a lot of configuration to make this work. I'll refer to configuration file paths relative to the directory where your Vagrantfile is.

puppet/puppet.conf only needs to contain "environment = vagrant". You might want to add various configuration to stay consistent with your production environment, of course.

puppet/hiera.yaml does need to be present, but does not need any actual configuration. We need to put "version: 5" in there to prevent Puppet warnings.

r10k/r10k.yaml should contain "cachedir: /var/cache/r10k".

Provisioning

While Vagrant comes with a Puppet provisioner, it does not work that well with our workflow, so we just write a custom shell script that does the necessary things to get everything set up. Here's an example for CentOS/RHEL:

#!/bin/sh
rhv=$(cat /etc/redhat-release | grep -Po '\d' | head -1)
rpm -Uvh https://yum.puppetlabs.com/puppetlabs-release-pc1-el-${rhv}.noarch.rpm
yum -y install puppet-agent
/opt/puppetlabs/puppet/bin/gem install r10k

Add it to our Vagrantfile:

config.vm.provision 'install_puppet', type: 'shell', path: 'install_puppet.sh'

Let's make sure it works by running this command:

$ vagrant up && vagrant ssh

Our first puppet run

We're almost ready to run puppet - only one thing is missing: Installing modules and their dependencies. We'll do this manually with r10k, inside the virtual machine:

$ cd /etc/puppetlabs/code/environments/vagrant
$ sudo /opt/puppetlabs/puppet/bin/r10k puppetfile install

You may also want to check for missing dependencies which need to be added to your puppetfile:

$ sudo /opt/puppetlabs/bin/puppet module list --tree

Once this is done, we can try executing a class:

$ sudo /opt/puppetlabs/bin/puppet apply -e "include profile::base"

At this point, you can start editing and testing your code changes in puppet-control.

Different SSH keys per github organisation

Wed, 23 Aug 2017 19:13:44 +0200

If you're like me, you prefer seting up different SSH keys for personal and professional use. Maybe you even work for multiple organisations at the same time and don't want to risk 1 compromised private key to have a wide-spread effect.

First, we need to set up our SSH config. We'll use a fake hostname for our github organisation. Put this in your ~/.ssh/config:

host example.github.com
    hostname github.com
    identityfile ~/.ssh/id_example_rsa

This will make sure that when we do any SSH (and indirectly, git) operations against the domain "example.github.com", the correct SSH key will be used.

We could just remember to replace github.com with example.github.com every time we git clone or add a remote URL, but that's tedious. Instead, we can set up git in a way that does this automatically for us. This is what you want in your ~/.gitconfig:

[url "git@example.github.com:example"]
insteadOf = git@github.com:example
insteadOf = https://github.com/example

This will dynamically replace any URLs that start with "https://github.com/example" or "git@github.com:example" with "git@example.github.com:example". This has the added benefit of letting you git clone the URL you would put in your browser to visit a repository on github, but git will automatically use SSH instead of trying HTTP authentication.

Everything should work as before, except git will use the correct SSH key. The only caveat here is that if you have keys in your ssh-agent, but not the one needed to work with the github organisation, your SSH client may not be smart enough to figure that out.

Proper logging in Django

Sat, 02 Sep 2017 10:03:00 +0200

Setting up logging in a sane way in Django has been surprisingly difficult due to some confusing setting names and the annoying way Django's default logging setup looks like. Here I'll go through some simple steps you can take to gain full control of your logging setup, without too many changes to a standard Django setup.

First of all, set LOGGING_CONFIG = None to prevent Django from setting up logging for you at all. You want this because in addition to the LOGGING dict that you define, Django has some defaults settings it will use, which you may not want.

Because we've set this, we need to call logging.dictConfig(LOGGING) ourselves. This can happen at the end of your settings file.

Make sure that LOGGING['disable_existing_loggers'] = False. If this is set to true, any loggers defined or invoked before logging.dictConfig is called will silently discard all its messages. You definitely don't want that.

Finally, I like to define LOGGING['root'] to have one log instance that controls everything, but sometimes log messages don't get sent to it. I found that setting the "" (empty string) logger can fix this:

LOGGING['loggers'][''] = { 'propagate': True }

Russell, revisited

Tue, 12 Sep 2017 18:58:00 +0200

3 years ago I wrote about Russell, a static site/blog generator I wrote. Since then, I've had a major rewrite of the project to make it easier to extend and configure.

My sentiments towards other static site generators and CMSes are still the same, though at least by now the most popular ones aren't all written in Ruby.

I realized quickly though that I wanted more control over how my site was to be generated. I didn't want to be limited to what could be expressed in a YAML file - it basically meant that I would have to think ahead of anything that the user of Russell would want to do, and add support for that in the code that reads the YAML config and acts upon it.

The solution to this was simple: Use Python to run and configure Russell instead. When you run russell setup to create a new Russell site, the main entrypoint will be run.py.

Furthermore, I now recommend that you install Russell into a virtualenv which you can bring in other dependencies to as well. For example, in the source code for the website you're reading now, I bring in libsass to compile Sass files into CSS.

blog.write_file('assets/style.css', sass.compile(
    filename=os.path.join(ROOT_DIR, 'sass', 'main.sass')
))

If you're looking for a static site generator, especially for a blog or similar, and you like Python, I recommend now more than ever to check out Russell!

Better ways of managing pip dependencies

Mon, 06 Nov 2017 18:57:00 +0100

Of all the languages I've worked with, Python is one of the most annoying to work with when it comes to managing dependencies - only Go annoys me more. The industry standard is to keep a strict list of your dependencies (and their dependencies) in a requirements.txt file. Handily, this can be auto-generated with pip freeze > requirements.txt.

What's the problem with requirement files? It's not really a problem as long as you only have one requirements file, but if you want to start splitting up dev vs test/staging vs production dependencies, you'll immediately run into problems.

The most common solution is to have a requirements directory with base.txt, dev.txt, prod.txt and so on for whatever environments/contexts you need. The problem with this approach starts showing up when you want to add or upgrade a package and its dependencies - because you no longer have a single requirements file, you can't simply pip freeze > requirements.txt, so you end up carefully updating the file(s) by hand.

There are some existing third-party tools out there written to help with this problem. Two of them create entirely new formats of storing information on dependencies:

pipenv/pipfile uses a completely new file format for storing dependencies, inspired by other language's more modern dependency managers. In the future this may be part of pip core, but it is not currently. Until then I'm staying far away from the project, as trying to implement it in a real-world project revealed all sorts of bugs. The codebase itself looks super sketchy, as it's downloaded upstream libraries like pip, but then applied patches on top of them.
poetry is a far more promising project. Its goals are similar to that of pipenv but it just seems to have been developed in a more sane way. It uses the PEP518 pyproject.toml file to store information about which dependencies should be installed.

Some other tools aim to stick closer to the existing workflow of requirement files:

pipwrap scans your virtualenv for packages, compares them to what's in your requirements files, and interactively asks you where it should place packages that are in your environment, but not in any requirements file.
pip-compile (part of pip-tools) lets you write more minimal requirements.in files, and auto-generates strict version requirements.txt files based on them. As a bonus you get to see where your nested dependencies are coming from.

There is also an existing solution that works without introducing third-party tools. Since version 7.1, there is a --constraint flag to the pip install command which can be used to solve this problem.

A constraint file is an additional requirements file which won't be used to determine which packages to install, but will be used to lock down versions for any packages that do get installed. This means that you can put your base requirements (that is, you don't need to include dependencies of dependencies) in your requirements file, then store version locks for all environments in a separate constraint file.

First of all, we want to make sure we never forget to add --constraint constraint.txt by adding it to the top of our requirements/base.txt file (and any other requirements file that does not include -r base.txt). Next, generate the constraint file with pip freeze > requirements/constraint.txt. You can now modify all your requirements files, removing or loosening version constraint, and removing nested dependencies.

With that out of the way, let's look at some example workflows. Upgrade an existing package:

pip install 'django >= 2'
# no need to edit requirements/base.txt, "django" is already there
pip freeze > requirements/constraint.txt

Install a new package in dev:

echo 'pytest-cov' >> requirements/dev.txt
pip install -r requirements/dev.txt
pip freeze > requirements/constraint.txt

Install requirements in a fresh production or development environment works just like before:

pip install -r requirements/base.txt
pip install -r requirements/dev.txt

This isn't perfect. If you don't install every requirement file in development, your constraint file will be missing those files' requirements. A code review would catch accidentally removing a constraint, but how do you detect a package that is entirely missing from the constraint file? pip install doesn't even have a dry-run mode. Still, constraint files (or any of the third-party tools, really) are nice ways of improving and simplifying dependency managment with pip.

There's also a shell command you can use as a commit hook or part of your test/CI suite to check that you're not missing anything in your constraint.txt:

! pip freeze | grep -vxiF -f requirements/constraint.txt -

This will output all pip packages that are installed but not present in your constraint file. We use the ! to make sure that the command gives a non-zero exit code if there are any matches.

Streaming and saving subprocess output at the same time in Python

Fri, 13 Apr 2018 21:18:00 +0200

Sometimes, you want to run a subprocess with Python and stream/print its output live to the calling process' terminal, and at the same time save the output to a variable. Here's how:

proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in proc.stdout:
    sys.stdout.buffer.write(line)
    sys.stdout.buffer.flush()
    # do stuff with the line variable here
proc.wait()

Multi-stage Docker builds for Python projects

Mon, 06 Aug 2018 21:34:00 +0200

Multi-stage builds can help reduce your Docker image sizes in production. This has many benefits: Development dependencies may potentially expose extra security holes in your system (I've yet to see this happen, but why not be cautious if it's easy to be so?), but mostly by reducing image size you make it faster for others to docker pull it.

The concept of multi-stage builds is simple: Install development dependencies, build all the stuff you need, then copy over just the stuff you need to run in production in a brand new image without installing development dependencies not needed to run the application.

Here's an example Dockerfile using the official Python Docker images, which are based on Debian - but you can easily apply the same principle when building from Debian, Ubuntu, CentOS, or Alpine images: Have one stage where build/development dependencies are installed and the application is built, and another where runtime dependencies are installed and the application is ran.

FROM python:3.7-stretch AS build
RUN python3 -m venv /venv

# example of a development library package that needs to be installed
RUN apt-get -qy update && apt-get -qy install libldap2-dev && \
    rm -rf /var/cache/apt/* /var/lib/apt/lists/*

# install requirements separately to prevent pip from downloading and
# installing pypi dependencies every time a file in your project changes
ADD ./requirements /project/requirements
ARG REQS=base
RUN /venv/bin/pip install -r /project/requirements/$REQS.txt

# install the project, basically copying its code, into the virtualenv.
# this assumes the project has a functional setup.py
ADD . /project
RUN /venv/bin/pip install /project

# this won't have any effect on our production image, is only meant for
# if we want to run commands like pytest in the build image
WORKDIR /project


# the second, production stage can be much more lightweight:
FROM python:3.7-slim-stretch AS production
COPY --from=build /venv /venv

# install runtime libraries (different from development libraries!)
RUN apt-get -qy update && apt-get -qy install libldap-2.4-2 && \
    rm -rf /var/cache/apt/* /var/lib/apt/lists/*

# remember to run python from the virtualenv
CMD ["/venv/bin/python3", "-m", "myproject"]

Copying the virtual environment is by far the easiest approach to this problem. Python purists will say that virtual environments shouldn't be copied, but when the underlying system is the same and the path is the same, it makes literally no difference (plus virtual environments are a dirty hack to begin with, one more dirty hack doesn't make a difference).

There are a few alternate approaches, the most relevant of which is to build a wheel cache of your dependencies and mount that in as a volume. The problem with this is that Docker doesn't let you mount volumes in the build stage, so you have to make complex shell scripts and multiple Dockerfiles to make it work, and the only major advantage is that you don't always have to re-compile wheels (which should be on pypi anyway, and my dependencies don't change that often).

Another thing of note: In our example, we install both project dependencies and the project itself into the virtualenv. This means we don't even need the project root directory in the production image, which is also nice (no risk of leaking example configuration files, git history etc.).

To build the image and run our project, assuming it's a webserver listening on port 5000, these commands should let you visit http://localhost:5000 in your browser:

$ docker build --tag=myproject .
$ docker run --rm -it -p5000:5000 myproject

Running tests

What if we want to build an image for running tests, which require some extra development dependencies? That's where the purpose of our ARG REQS comes in. By setting this build argument when running docker build, we can control which requirements file is read. Combine that with the --target argument to docker run and this is how you build a development/testing image:

$ docker build --target=build --build-arg REQS=dev --tag=myproject-dev .

And let's say you want to run some commands using that image:

$ docker run --rm -it myproject-dev /venv/bin/pytest
$ docker run --rm -it myproject-dev bash

Development in Docker

Note that you'll have to re-build the image any time code changes. I don't care too much about this since I do all my development locally anyway, and only use Docker for production and continuous integration, but if it's important to you, you'll have to:

Change pip install /project to pip install -e /project
Copy the entire /project directory into the production image as well
Mount the project's root directory as /project with docker run --volume=$PWD:/project

Example project

If you want a functional example to play around with, I've got a git repository on Github with a sample Python project which has a docker-multistage branch: python-project-examples

Building deb/rpm packages with FPM in Docker

Tue, 17 Dec 2019 18:23:00 +0100

Whether you're building on-premise software or just want to use packages as your atomic deployment mechanism of choice in a traditional bare-metal/VM infrastructure, deb/rpm packages are a nice thing to provide.

Unfortunately, building them is super tedious. Try googling for official documentation on how to build Debian packages and you'll find at least 5 official wiki pages all with slight variations. Redhat packages are a bit better but still rather tedious. Luckily, there's a program which has our back: FPM - Effing Package Management.

Looking into how to run this on an engineer's laptop as easily as in a CI/CD pipeline, annoyances keep piling up.

Running FPM with the correct arguments is annoying. Let's put the invocations in a shell script or Makefile.
We don't want to have to install FPM on our main system - it requires ruby, gems and more. Let's run it in a Docker container. Luckily, there exists a Docker image which contains FPM and its dependencies, as well as optional dependencies for extra features: eclecticiq/package
The application runtime (binaries, libraries...) needs to be available in the same container, otherwise FPM doesn't know what to package. Let's use multi-stage Docker builds to build the application and make the resulting files available to the FPM container.
Running docker build, docker run, then copying files out of the docker container is annoying. Let's put the invocations in the Makefile or another shell script.

With all of the above fixed, you end up with a single Makefile target or shell script which would build the project's deb/rpm packages for me, either on my laptop or in the CI/CD system of my choice.

A complete example of how to build, run, and package an application using this system is available on Github: fpm-docker-example

It has a few branches which are worth checking out: The default branch is a simple Golang example, but there's also a branch for Python applications, and one showing how the process can be simplified using Earthly.

Asking for help in public or private chat

Sun, 10 May 2020 19:42:00 +0200

Here's a pattern I've seen in multiple organizations: If someone is stuck with a problem, they guess who might know the solution to that problem and ask them privately, either in person or over chat (e.g. Slack). I always preferred to be asked over chat rather than in-person because it means I can postpone reading it for 5 minutes if I'm in the middle of something, and now with remote working being pretty much mandatory, the in-person option isn't even there any more.

I think this is a bad habit, and something that should be discouraged especially in engineering departments. First, let's outline the advantages of asking in private, usually the reasons why people default to doing it:

You don't feel like you're bothering everyone in the channel who might not care about your problem.
You're guaranteed to get someone's attention, your question is very unlikely to be ignored.
It feels similar to what you would do in real life: Walk up to a colleague and ask them if they have time to help.

While these points are technically correct, I also think they're based on flawed premises, which we will get back to. For now, let's list the drawbacks of asking for help in private:

You have to know who to ask.
If you ask the wrong person, you've potentially interrupted them without really achieving anything.
If you ask the wrong person and you are referred to someone else, or you just guess someone else who might know, you're asking your question N times which wastes your time. This gets especially annoying if your question isn't a simple copy-and-paste one, for example if there are follow-up Q&A or additional context added after the fact.
If the person you ask gives you an answer, that answer may be incorrect or inefficient, and another colleague might be able to correct them - but because your messages are private, they can't.
Other colleagues cannot read your question and answers given to your question, and hence cannot learn from it.
It gives the impression that no one is asking for help, which can discourage others (especially new joiners) from asking for help.

It's worth noting that this is slightly different when not working remotely - if I walk over to a colleague in the office and ask them for help and we discuss the problem in person, others can overhear our discussion if they're not too busy with other things, and we avoid a lot of the problems mentioned above.

So what should you do instead? Ask in a public chat channel. Even Slack itself claims it's the ideal:

Slack is designed to add transparency to an organization, so it’s best to default to communication in public channels whenever possible. Slack’s own team sends tens of thousands of messages each week—in a recent summary, 70% of those were posted in public channels, with 28% occurring in private channels and just 2% in direct messages. Posting messages in public channels means anyone in the organization can see what various teams are working on, see how much progress people are making on projects, and search the archive for context they need.

Here are some tips for successfully asking for help in public chat channels:

If you're not sure which channel is the best fit, just put it in the most generic one. Most engineering companies have channels like #dev-example-app or just #development.
If your question is long and you don't want to flood the channel with text, summarize your question in a short paragraph then elaborate on it in a thread. This has the added bonus of encouraging others to use threads as well.
Ask your question well: Include the necessary context, what you've already tried, predict what follow-up questions might come - but also don't make it too long or include too many unncecessary details, or people will get overwhelmed and you're less likely to get help.
If you're worried that putting your question in a public chat channel will create noise for others, actually verify if this is the case. Are you bothered any time a question is asked in a public channel? Ask some colleagues in one-on-one situations what they think as well. In my experience, no one enables sound or pop-up notifications for channel messages. But if this is actually a problem somehow, consider creating dedicated channel(s) (e.g. #dev-help) for asking for help that can more easily be muted.
If, after some time (let's say an hour), you haven't resolved your question, consider mentioning colleagues who you suspect might be able to help (the people you would consider asking for help in private). You can do this either by sending a link to your question in a private message, or mentioning them in the channel/thread itself.
If you haven't resolved your question after a long amount of time (let's say 24 hours), simply post the question again and specify that you didn't get an answer within 24 hours.
If the question remains unresolved for even longer, you'll either have to escalate to your manager, or accept that no one actually knows and you'll have to figure out for yourself.

Complex groups in Ansible using the constructed inventory plugin

Fri, 11 Mar 2022 17:57:00 +0100

If you ever start working with large, semi-heterogenous groups of VMs and want to manage them with Ansible, you quickly start running into problems trying to make it work with complex groups.

As an example, let's assume we have webservers and jobservers spread out across 2 cloud providers, for 3 different applications, in 3 environments (production, staging, and test). How would you set variables for all webservers for application A? Or all production servers across all providers and applications? Or all webservers, but not jobservers?

The typical solution is to have a lot of duplication in your inventory files. For example:

all:
  children:
    env_prod:
      children:
        provider_gcp:
          children:
            type_webserver:
              children:
                app_a:
                  children:
                    app_a_webservers:
                      hosts:
                        web[01:03].app-a.prod.gcp.mycorp.net:
                app_b:
                  children:
                    app_b_webservers:
                      hosts:
                        web[01:03].app-b.prod.gcp.mycorp.net:
            type_jobserver:
              children:
                app_a:
                  children:
                    app_a_jobservers:
                      hosts:
                        job[01:03].app-a.prod.gcp.mycorp.net:
                app_b:
                  children:
                    app_b_jobservers:
                      hosts:
                        job[01:03].app-b.prod.gcp.mycorp.net:

Imagine many, many more lines like this, for other environments and providers. The only thing we're not repeating in this example is the top-level group (which in this case is env_prod, but it's up to you how to structure your hierarchy) and the lowest level group.

The alternative: Using `constructed`

# inventory/main.yml
all:
  children:
    app_a_webservers:
      hosts:
        web[01:03].app-a.prod.gcp.mycorp.net:
        web[01:05].app-a.prod.aws.mycorp.net:
        web[01:02].app-a.stag.gcp.mycorp.net:
        web[01:02].app-a.test.gcp.mycorp.net:

The constructed plugin lets you define jinja statements for groups. Basically, if the Jinja statement evaluates to true, the server will be added to this group. Any built-in Ansible variables (not group_vars, host_vars) are available here, so you can check things like the hostname, which user you're using to connect with, which non-constructed groups the server is a part of, and more. inventory_hostname is a useful one, and because it's a string you can do a lot of useful Python operations on it, such as .split and .startswith. You can also use Ansible filters.

# inventory/zz-constructed.yml
plugin: constructed
strict: true

groups:
  provider_gcp: inventory_hostname.endswith('.gcp.mycorp.net')
  provider_aws: inventory_hostname.split('.')[-3] == 'aws'
  env_prod: inventory_hostname.split('.')[-4] == 'prod'
  env_staging: '.stag.' in inventory_hostname
  env_test: inventory_hostname.endswith(('.stag.gcp.mycorp.net', '.stag.aws.mycorp.net'))
  type_webserver: inventory_hostname.startswith('web')
  type_jobserver: inventory_hostname | regex_search('^job\d+\.')
  app_a: inventory_hostname.split('.')[1] == 'app-a'
  app_b: inventory_hostname | regex_search('[a-z0-9]+\.')

lutro.me

Volume in Debian distros (Crunchbang, Ubuntu...)

Optimizing for production with Laravel 4

Assertions on method calls using Mockery

Testable, simple L4 code without repository patterns

When to use repositories?

Using the Form Builder as little as possible

Mocking models using Mockery

Russell

Assertions on mails in Laravel 4

Laravel and SemVer

Shrinking controllers and abstracting with manager classes

SASS tests with a simple Makefile

Git(hub) - Push via SSH, pull via HTTP(S)

Hints towards learning SaltStack

Installing a virtualenv bin globally

Cache busting URLs with Flask and Nginx

Dangers of targetting grains in Salt

What to do instead?

Wait for a port to be listening in Salt

Dynamic includes in Salt

Managing systemd units with Salt

Using a puppet-control repo in Vagrant

Modifications to the control repo

How we'll run Puppet

Creating the Vagrant repo

Configuration files

Provisioning

Our first puppet run

Different SSH keys per github organisation

Proper logging in Django

Russell, revisited

Better ways of managing pip dependencies

Streaming and saving subprocess output at the same time in Python

Multi-stage Docker builds for Python projects

Running tests

Development in Docker

Example project

Building deb/rpm packages with FPM in Docker

Asking for help in public or private chat

Complex groups in Ansible using the constructed inventory plugin

The alternative: Using constructed

The alternative: Using `constructed`