Automate Frequent Tasks

Overview

Teaching: 10 min
Exercises: 10 min

Questions

• How should I handle tasks I do repeatedly?

Objectives

• Explain what build managers were originally designed to do, and what else they are now used to do.

• Make a build file self-explaining.

• Explain when to use checklists rather than a build manager.

DRY: Don’t Repeat Yourself

• Usually applied to nouns (code)
• Just as true for verbs (actions)
• The only thing you can accomplish by typing something repeatedly is to get it wrong

Build Manager

• Use a build manager
  • GNU Make defined the category, but depends on native shell commands
  • CMake is a meta-tool that creates build files for multiple systems
  • SCons and similar tools define build rules in a full-blown programming language
  • Language-specific tools like rake, pydoit, and SnakeMake
• Originally created to compile multi-file programs efficiently, but can all be used for arbitrary tasks
  • Run tests, build packages for release, create reports, …
  • Common pattern: build shell script or utility program, then launch from build file
• Key feature: dependencies
  • “X depends on Y depends on Z”
  • Usually implemented using timestamps or hashes
  • Not well suited to verbs
• General workflow tools may be a better fit for actual scientific work
  • Galaxy is a high-end tool
  • doit is a lot easier to start with
  • Only as stable as the pieces they connect

Checklists

• A checklist is a build file meant to be executed by human beings
  • The Checklist Manifesto describes how use of checklists cuts fatalities in surgery significantly, along with many other examples
• Use them for anything that can’t be done automatically by a machine
• Keep in version control
  • Ask every new contributor/user to use and give feedback
• Include a contact email address in every checklist

How Do You Manage Your Repository?

Describe in 3-4 bullet points how you actually manage your version control repository. How often are you working on several things simultaneously?

Create a Task list

1. If your project already uses a build manager, what tasks are used most often?
2. If your project doesn’t use a build manager, what are the first few tasks you should automate?

Self-Documenting Build Files

The default target in a build file (e.g., make with no parameters) should print a list of available commands. Look at the Makefile in this repository to see how this works, then modify the build file for your project to do so as well.

Create a Setup Checklist

1. Write a short point-form checklist describing the things you do when setting up a new machine to do development on your project.

2. How many of the steps in your checklist can be automated using shell scripts or other small programs?

3. How will newcomers know if they have completed the steps in the checklist correctly?

Key Points

• Use a build manager to manage repetitive tasks.

• Make build files explain themselves.

• Use checklists for tasks that have to be done repeatedly, but can’t be done by a computer.

• Have new contributors go through checklists to look for omissions and inaccuracies.