database (dā’tə-bās’) noun: “A collection of data arranged for ease and speed of search and retrieval by a computer”
— The American Heritage® Science Dictionary
- Three common options for storing data
- Easy to create, work well with version control
- But then we have to build search and analysis tools ourselves
- Good for simple analyses
- But don’t handle large or complex data sets well
- Include powerful tools for search and analysis
- Can handle large, complex data sets.
Relational databases are not as widely used in science as in business, but they are still a common way to store large data sets with complex structure. Even when the data itself isn’t in a database, the metadata could be: for example, meteorological data might be stored in files on disk, but data about when and where observations were made, data ranges, and so on could be in a database to make it easier for scientists to find what they want to.
The first few sections (up to “Missing Data”) usually go very quickly. The pace usually slows down a bit when null values are discussed mostly because learners have a lot of details to keep straight by this point. Things really slow down during the discussion of joins, but this is the key idea in the whole lesson: important ideas like primary keys and referential integrity only make sense once learners have seen how they’re used in joins. It’s worth going over things a couple of times if necessary (with lots of examples).
The sections on creating and modifying data, and programming with databases, can be dropped if time is short. Of the two, people seem to care most about how to add data (which only takes a few minutes to demonstrate).
Simple calculations are actually easier to do in a spreadsheet; the advantages of using a database become clear as soon as filtering and joins are needed. Instructors may therefore want to show a spreadsheet with the information from the four database tables consolidated into a single sheet, and demonstrate what’s needed in both systems to answer questions like, “What was the average radiation reading in 1931?”
Some advanced learners may have heard that NoSQL databases (i.e., ones that don’t use the relational model) are the next big thing, and ask why we’re not teaching those. The answers are:
- Relational databases are far more widely used than NoSQL databases.
- We have far more experience with relational databases than with any other kind, so we have a better idea of what to teach and how to teach it.
- NoSQL databases are as different from each other as they are from relational databases. Until a leader emerges, it isn’t clear which NoSQL database we should teach.
data/*.csv: CSV versions of data in sample survey database.
bin/create-db.sql: generate survey database used in examples based on CSV.
In order to execute the following lessons interactively, please install SQLite as mentioned in the setup instructions for your workshop. Then:
$ git clone http://github.com/swcarpentry/sql-novice-survey.git $ cd sql-novice-survey
Next, create the database that will be used:
$ sqlite3 survey.sqlite '.read bin/create-db.sql'
This reads commands from
which sets up the tables and loads data from the CSV files in the
To run commands interactively,
run SQLite on
$ sqlite3 survey.sqlite SQLite version 3.8.5 2014-08-15 22:37:57 Enter ".help" for usage hints. sqlite>
The command history and line editing features provided by
invaluable with a command-line tool like
sqlite3. Participants should be
encouraged strongly to start with a simple SQL statement and then use the
up-arrow key to go back and add clauses one at a time, or fix problems, rather
than typing each command from scratch. Unfortunately on some Linux and Mac OS X
systems participants have found that the arrow keys do not scroll through the
command history as expected.
A workaround for this it to use the rlwrap (readline wrapper) command when starting SQLite:
$ rlwrap sqlite3 survey.sqlite