Data Carpentry for Oceanographers: Glossary

Good data organization is the foundation of any research project.

Never modify your raw data. Always make a copy before making any changes.

Keep track of all of the steps you take to clean your data in a plain text file.

Organize your data according to tidy data principles.

Avoid using multiple tables within one spreadsheet.

Avoid spreading data across multiple tabs.

Record zeros as zeros.

Use an appropriate null value to record missing data.

Don’t use formatting to convey information or to make your spreadsheet look pretty.

Place comments in a separate column.

Record units in column headers.

Include only one piece of information in a cell.

Avoid spaces, numbers and special characters in column headers.

Avoid special characters in your data.

Record metadata in a separate plain text file.

Treating dates as multiple pieces of data rather than one makes them easier to handle.

Always copy your original spreadsheet file and work with a copy so you don’t affect the raw data.

Use data validation to prevent accidentally entering invalid data.

Use sorting to check for invalid data.

Use conditional formatting (cautiously) to check for invalid data.

Data stored in common spreadsheet formats will often not be read correctly into data analysis software, introducing errors into your data.

Exporting data from spreadsheets to formats like CSV or TSV puts it in a format that can be used consistently by most programs.

Python is an open source and platform independent programming language.

Jupyter Notebook and the Spyder IDE are great tools to code in and interact with Python. With the large Python community it is easy to find help on the internet.

Python is an interpreted language which can be used interactively (executing one command at a time) or in scripting mode (executing a series of commands saved in file).

One can assign a value to a variable in Python. Those variables can be of several types, such as string, integer, floating point and complex numbers.

Lists and tuples are similar in that they are ordered lists of elements; they differ in that a tuple is immutable (cannot be changed).

Dictionaries are data structures that provide mappings between keys and values.

Libraries enable us to extend the functionality of Python.

Pandas is a popular library for working with data.

A Dataframe is a Pandas data structure that allows one to access data by column (name or index) or row.

Aggregating data using the groupby() function enables you to generate useful summaries of data quickly.

Plots can be created from DataFrames or subsets of data that have been generated with groupby().

Pandas uses other names for data types than Python, for example: object for textual data.

A column in a DataFrame can only have one data type.

The data type in a DataFrame’s single column can be checked using dtype.

Make conscious decisions about how to manage missing data.

A DataFrame can be saved to a CSV file using the to_csv function.

In Python, portions of data can be accessed using indices, slices, column headings, and condition-based subsetting.

Python uses 0-based indexing, in which the first element in a list, tuple or any other data structure has an index of 0.

Pandas enables common data exploration steps such as data indexing, slicing and conditional subsetting.

Pandas’ merge and concat can be used to combine subsets of a DataFrame, or even data from different files.

join function combines DataFrames based on index or column.

Joining two DataFrames can be done in multiple ways (left, right, and inner) depending on what data must be in the final DataFrame.

to_csv can be used to write out DataFrames in CSV format.

install plotnine, a required pakcage for our lessons.

xarray and iris are the core Python libraries used in the atmosphere and ocean sciences.

Use conda to install and manage your Python environments.

Matplotlib is the engine behind plotnine and Pandas plots.

The object-based nature of matplotlib plots enables their detailed customization after they have been created.

Export plots to a file using the savefig method.

The data, aes variables and a geometry are the main elements of a plotnine graph

With the + operator, additional scale_*, theme_*, xlab/ylab and facet_* elements are added

Loops help automate repetitive tasks over sets of items.

Loops combined with functions provide a way to process data more efficiently than we could by hand.

Conditional statements enable execution of different operations on different data.

Functions enable code reuse.

NetCDF is a format to store gridded data and widely use in climate science.

A netCDF file contains dimensions, variables, and attributes.

Xarray is a library to work with NetCDF data in Python.

CMIP data is used in climate modelling.

Libraries such as xarray can make loading, processing and visualising netCDF data much easier.

The cmocean library contains colormaps custom made for the ocean sciences.

Define a function using def name(...params...).

The body of a function must be indented.

Call a function using name(...values...).

Use help(thing) to view help for something.

Put docstrings in functions to provide help for that function.

Specify default values for parameters when defining a function using name=value in the parameter list.

The readability of your code can be greatly enhanced by using numerous short functions.

Write (and import) modules to avoid code duplication.

For large arrays, looping over each element can be slow in high-level languages like Python.

Vectorised operations can be used to avoid looping over array elements.

Libraries such as argparse can be used the efficiently handle command line arguments.

Most Python scripts have a similar structure that can be used as a template.

Use git config to configure a user name, email address, editor, and other preferences once per machine.

git init initializes a repository.

git status shows the status of a repository.

Files can be stored in a project’s working directory (which users see), the staging area (where the next commit is being built up) and the local repository (where commits are permanently recorded).

git add puts files in the staging area.

git commit saves the staged content as a new commit in the local repository.

Always write a log message when committing changes.

git diff displays differences between commits.

git checkout recovers old versions of files.

A local Git repository can be connected to one or more remote repositories.

Use the HTTPS protocol to connect to remote repositories until you have learned how to set up SSH.

git push copies changes from a local repository to a remote repository.

git pull copies changes from a remote repository to a local repository.

Program defensively, i.e., assume that errors are going to arise, and write code to detect them when they do.

Put assertions in programs to check their state as they run, and to help readers understand how those programs are supposed to work.

The pdb library can be used to debug a Python script by stepping through line-by-line.

Software Carpentry has more advanced lessons on code testing.

It is possible (in only a few lines of code) to record the provenance of a data file or image.

sqlite3 provides a SQL-like interface to read, query, and write SQL databases from Python.

sqlite3 can be used with Pandas to read SQL data to the familiar Pandas DataFrame.

Pandas and sqlite3 can also be used to transfer between the CSV and SQL formats.