Introduction#

Version control, also known as revision control or source control, is the management and tracking of changes to computer code and other certain other types of data in an automated way.

Any project (collections of files in directories) under version control has changes and additions/deletions to its files and directories recorded and archived over time so that you can recall specific versions later.

Version control is in fact the technology embedded in the versioning of various word processor and spreadsheet applications (e.g., Google Docs, Overleaf).

Why Version Control?#

With version control of biological computing projects, you can:

• record all changes made to a set of files and directories, including text (usually ASCII) data files, so that you can access any previous version of the files

• “roll back” data, code, documents that are in plain text format (other file formats can also be versioned; see section on binary files below).

• collaborate more easily with others on developing new code or writing documents – branch (and merge) projects

• back up your project (but git is not a backup software - see sections on binary and large files below).

Why git ?#

We will use git, developed by Linus Torvalds, the “Linu” in Linux. This is currently the most popular tool for version control. It is a distributed version control system designed to handle everything from small to very large projects with speed and efficiency. It allows multiple developers to work on a project simultaneously without overwriting each other’s changes, facilitating collaboration and maintaining a history of all modifications.

The man advantages of Git are:

• Distributed System: Every developer has a complete copy of the project on their computer, including its history.

• Efficiency: Fast performance for both local and remote operations.

• Data Integrity: Every file and project snapshot is checked for data integrity.

• Flexibility: Supports “non-linear” development workflows through branching and merging.

• Collaboration: Facilitates teamwork by allowing multiple people to work on different parts (typically on separate branches) simultaneously.

Let’s get started!#

In git, each user stores a complete local copy of the project, including the history and all versions (the “repository”). So you do not rely as much on a centralized (“remote” - somewhere else, accessible through the internet) server.

The Git repository is a directory that contains your project’s files and the entire history of changes made to those files. Here is a graphical outline of the git repository, workflow and command structure:

There are several core git concepts and commands that you need to understand. But first things first - let’s first install get and set up your first git repository!

Installing git#

Let’s install and configure git. On Ubuntu:

sudo apt-get install git

Then, some configurations:

git config --global user.name "Your Name"
git config --global user.email "your.login@imperial.ac.uk"
git config --global init.defaultBranch main

Now check your git configurations:

git config --list

This output displays the configuration variables and their corresponding values that are currently active in your Git environment. As you are just getting started, the output will consist of just a few lines.

Your first repository#

Time to bring your TMQB coursework under version control. First, let’s navigate to it:

Then initialize your local git repository:

git init

Initialised empty Git repository in /home/mhasoba/Documents/CMEECourseWork/.git/

Create your README file (as a markdown document, so with the .md extension):

echo "My CMEE Coursework Repository" > README.md

Check again git config --list to see if anything has changed.

Check the files and directories that have been created:

ll #same as "ls -al"

Add the README (AKA “stage” it for for a git commit):

git add README.md

Check the current status of your repository:

git status

On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
	new file:   README.md

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	week1/
	week2/

This tells you that there are local changes that need to be committed.

git commit -m "Added README file."

[main (root-commit) b2a73ef] Added README file.
 1 file changed, 1 insertion(+)
 create mode 100644 README.md

Check status again:

git status

On branch main
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	week1/
	week2/

nothing added to commit but untracked files present (use "git add" to track)

What does it say now? - Compare with the above output of git status.

You can also add changes matching a particular pattern.

For example

git add *.txt

will add all text files that were changed since last commit.

Note that this adds ALL changes that match the pattern, recursively (through all sub-directories).

Next you can commit the rest of these files inside your current coursework (e.g., week 1) with another message, just like you did for readme above:

git commit -m "Full first commit of new project"

On branch main
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	week1/
	week2/

nothing added to commit but untracked files present (use "git add" to track)

git status

On branch main
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	week1/
	week2/

nothing added to commit but untracked files present (use "git add" to track)

How Git Stores Data#

Git doesn’t store data as a series of changes or differences, but as a series of snapshots of the entire repository. When you make a commit, Git stores a reference to the staged snapshot and a pointer to the commit that came immediately before it.

Git uses four types of Objects to store data:

• Blobs: Stores file data.

• Trees: Represents directories and contains pointers to blobs and other trees.

• Commits: Points to a tree object (snapshot of the working directory), parent commits, and includes metadata.

• Tags: Points to another object (usually a commit) and includes metadata.

These objects are stored in the .git/objects directory and are identified by their SHA-1 hash.

Basic git commands#

Here are some fundamentally important git commands (please make sure you read more about them):

There are more git commands which you will also learn, but it is important that you familiarize yourself with these ones in particular first.

The most common sequence of your git work and commands will be:

• Edit Files: Modify files in your working directory.

• Stage Changes: Use git add <file> to add changes to the staging area.

• Committ Changes: Use git commit -m "commit message" to save changes to the repository.

But before you do that, you will always want to

• Use git status to see the status of your working directory and staging area.

• Use git log to view the commit history.

git remote add origin git@github.com:YourGithubUsername/CMEECourseWork.git

git remote -v

origin  git@github.com:YourGithubUsername/CMEECourseWork.git (fetch)
origin  git@github.com:YourGithubUsername/CMEECourseWork.git (push)

git push origin master

Branching#

Imagine you want to try something out, but you are not sure it will work well.

For example, say you want to rewrite the Introduction of your $LATEX$ paper using a different angle, or you want to see whether switching to a new library or package for a piece of code improves speed. What you then need is branching, which creates a project copy in which you can experiment.

git branch anexperiment

git branch

  anexperiment
* main

git tells you that you have two branches.

git checkout anexperiment

Switched to branch 'anexperiment'

git branch

* anexperiment
  main

git tells you that you have successfully switched to the new branch

Or if you want to create and switch to a new branch immediately, you can use the -b flag:

git checkout -b anexperiment

Now make a change (update the Readme to record the new naughty thing you have started doing!):

echo "I am going to try this new naughty thing on this experimental branch!" >> README.md

cat README.md

My CMEE Coursework Repository
I am going to try this new naughty thing on this experimental branch!

git commit -am “Testing experimental branch”

The -am flag performs the commit with a message while also automatically staging certain changes before committing.

Specifically, the a part of the flag stages all changes to tracked files (files that have already been committed at least once). This flag does not stage new, untracked files; you must use git add to stage those separately.

Now if you decide to merge the new branch after modifying it:

git checkout main

M	README.md
Switched to branch 'main'

git merge anexperiment

Already up-to-date.

cat README.md

My CMEE Coursework Repository
I am going to try this new naughty thing on this experimental branch!

Unless there are conflicts, i.e., some other files that you changed locally had diverged from those files in the main branch in the meantime (say, due to new changes pushed by another collaborator), you are done, and you can delete the branch:

git branch -d anexperiment

Deleted branch anexperiment (was 3fbab16).

If instead you are not satisfied with the result, and you want to abandon the branch:

git branch -D anexperiment

Obvously, we have been just messing around, so you don’t really want the Readme of your main branch to say what it is currently saying!

To fix this, you now have two options: edit the Readme back to the state you wanted (and then add, commit and push), or use the git reset --hard command. However, use the latter with caution. Before you do anything to reverse the merge, read about resetting and revertinf below.

Teamwork using git and common branching mistakes#

As you can see, branchhing is the cornerstone of collaborative coding and software development using git.

Here are common mistakes to avoid with git branching and merging in team projects:

• Working Directly on the Main Branch

  • Issue: Making changes directly to the main or master branch without using feature branches.

  • Why It’s a Problem: This practice can lead to a cluttered commit history and increases the risk of introducing unstable code into the main codebase, affecting all team members.

• Not Keeping Branches Up-to-Date

  • Issue: Failing to regularly merge or rebase the main branch into feature branches.

  • Why It’s a Problem: Leads to large, complex merge conflicts when it’s time to integrate changes back into the main branch.

• Not Pulling Latest Changes Before Starting Work

  • Issue: Forgetting to fetch and merge the latest changes from the remote repository before starting new work.

  • Why It’s a Problem: Leads to working on outdated code and causes significant merge conflicts when pushing changes later.

• Improper Branching Practices

  • Issue: Not creating separate branches for new features or bug fixes, or using inconsistent branch naming conventions.

  • Why It’s a Problem: Makes it difficult to manage code changes, track progress, and collaborate effectively.

• Poor Commit Hygiene

  • Issue: Making large, infrequent commits with vague messages like “Fixed stuff” or “Changes.”

  • Why It’s a Problem: Hinders the ability to trace specific changes, debug issues, and understand the project’s evolution.

• Not Handling Merge Conflicts Properly

  • Issue: Overwriting code, deleting others’ work, or incorrectly resolving conflicts without understanding the implications.

  • Why It’s a Problem: Can introduce bugs, cause loss of important code, and disrupt the functionality of the application.

• Overwriting Team Members’ Changes

  • Issue: Using git push --force or not merging the latest remote changes before pushing local commits.

  • Why It’s a Problem: Leads to loss of others’ work and creates confusion in the team’s codebase.

• Ignoring the .gitignore File

  • Issue: Committing unnecessary files like binaries, temporary files, or sensitive information.

  • Why It’s a Problem: Clutters the repository, increases clone/download times, and poses security risks.

• Misusing Git Commands

  • Issue: Incorrectly using commands like git reset, git revert, or rebasing public branches.

  • Why It’s a Problem: Can rewrite commit history, cause loss of work, and confuse team members due to unexpected repository states.

• Lack of Communication and Coordination

  • Issue: Not informing team members about new branches, changes, or issues encountered.

  • Why It’s a Problem: Leads to duplicated work, merge conflicts, and a lack of cohesion in development efforts.

• Pushing Broken or Untested Code

  • Issue: Committing untested code.

  • Why It’s a Problem* Disrupts the workflow for others, especially if continuous integration pipelines fail due to the broken code.

• Confusion Between Local and Remote Branches

  • Issue: Forgetting to push local branches to the remote repository or misunderstanding the state of branches.

  • Why It’s a Problem: Causes discrepancies between local and remote repositories, leading to confusion and potential loss of work.

• Conflict Between Personal and Team Workflows

  • Issue: Using personal Git configurations or workflows that don’t align with the team’s agreed-upon practices.

  • Why It’s a Problem: Creates inconsistency and can cause issues with code integration and collaboration.

• Over-Reliance on GUI Tools Without Understanding Git Basics

  • Issue: Using graphical interfaces without grasping the underlying Git commands.

  • Why It’s a Problem: Limits the ability to troubleshoot issues and understand what’s happening behind the scenes, leading to mistakes.

• Ignoring Merge Warnings and Errors

  • Issue: Proceeding with merges without addressing warnings or errors.

  • Why It’s a Problem: Can result in incomplete merges, broken code, and unresolved conflicts that affect the entire team.

• Mismanaging Merge Strategies

  • Issue: Inappropriately using git merge vs. git rebase without understanding the consequences.

  • Why It’s a Problem: Can create a confusing commit history and complicate collaboration efforts.

• Deleting Branches Prematurely

  • Issue: Removing branches that others are still using or that contain unmerged changes.

  • Why It’s a Problem: Causes loss of work and disrupts team members who are relying on those branches.

Tips to Avoid These Mistakes:

• Educate Yourself: Take time to learn Git fundamentals, including branching, merging, and resolving conflicts.

• Use Feature Branches: Create a new branch for each feature or bug fix to isolate your work.

• Commit Often with Clear Messages: Make small, incremental commits with descriptive messages.

• Pull Frequently: Regularly fetch and merge changes from the remote repository to stay up-to-date.

• Communicate: Keep open lines of communication with your team about what you’re working on.

• Test Before Committing: Ensure your code compiles and passes tests before pushing.

• Understand Git Commands: Before using advanced commands like git reset or git rebase, make sure you understand their effects.

• Use .gitignore Wisely: Configure your .gitignore file to exclude unnecessary or sensitive files.

• Review Pull Requests: Participate in code reviews to catch issues early and share knowledge.

• Follow Team Conventions: Adhere to agreed-upon workflows, branch naming conventions, and commit message formats.

The README file#

A README (like the README.md that you created in your git repo above) is a text file that introduces and explains a project. It contains information that is required to understand what the project is about and how to use or run it.

While READMEs can be written in any text file format, Markdown (saved as an .md file) is most commonly used, as it allows you to add simple text formatting easily. Two other formats that you might most often see are plain text and reStructuredText (saved as an .rst file, common in Python projects).

You can find many README file suggestions (and templates) online. Essentially, it should ideally have the following features/content:

• Project name / title

• Brief description: what your project does and/or is for. Provide context and add links to any references to help new visitors.

• Languages: List language(s) and their versions used in the project

• Dependencies: What special packages (which are not part of standard libraries of the language(s) used) might be needed for a new user to run your project

• Installation: Guidelines for installing the project (if applicable), including dependencies.

• Project structure and Usage: How the project is structured and how to run/use it. Explain, if relevant, what specific files do. No need to list every file, such as data or experimental ones (like the ones in sandbox).

• Author name and contact

In addition, you may want to include(but not necessary for your current coursework), License, Acknowledgments, and instructions for Contributing.

Ignoring Files#

You will have some files you don’t want to track (log files, temporary files, executables, etc). You can ignore entire classes of files with .gitignore.

$\star$ Let’s try it (make sure you are in your coursework directory (e.g., CMEECourseWork)):

echo -e "*~ \n*.tmp" > .gitignore

cat .gitignore

*~ 
*.tmp

git add .gitignore

touch temporary.tmp

Then,

git add *

You can also create a global gitignore file that lists rules for files to be ignored in every Git repository on your computer.

git rm --cached <file>

git rm -r --cached <folder>

*.csv
!pushabledata.csv

*.*
!.gitignore
!*.R
!*.py
!*.sh
!*.tex
!*.bib
!Week1/results/myfirstlatexdoc.tex

Dealing with binary files#

A binary file is computer-readable but not human-readable, that is, it cannot be read by opening them in a text viewer. Examples of binary files include compiled executables, zip files, images, word documents and videos. In contrast, text files are stored in a form (usually ASCII) that is human-readable by opening in a text reader (e.g., gedit). Without some git extensions and configurations (coming up next), binary files cannot be properly version-controlled because each version of the entire file is saved as is in a hidden directory in the repository (.git).

However, with some more effort, git can be made to work for binary formats like *.docx or image formats such as *.jpeg, but it is harder to compare versions; have a look at this and this¹, and also, this.

Dealing with large files#

As such, git was designed for version control of workflows and software projects, not large files (say, >100mb) (which may be plain-text or binary). Binary files are particularly problematic because each version of the file is saved as is in .git, when you have a large number of versions it means that there are the same number of binary files in the hidden directory (for example 100 $\times$ >100mb files!).

So please do not keep large files (especially binary files) under version control². For example, if you are doing GIS work, you may have to handle large raster image files. Do not bring such files under version control. We suggest that you include files larger than some size in your .gitignore. For example, you can use the following bash command:

find . -size +100M | cat >> .gitignore

The 100M means 100 mb – you can reset it to whatever you want.

Then what about code that needs large files? For this, the best approach is write code that scales up with data size. If it works on a 1 mb file, it should also work on a 1000 mb file! If you have written such code, then you can include a smaller file as a MWE (minimum working example).

You may also explore alternatives such as git-annex (e.g., see this), and git-lfs (e.g., see this).

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v5.0.0  # or use the latest version
    hooks:
      - id: check-added-large-files
        args: ['--maxkb=500']  # optional: set a custom size limit in KB

pre-commit install

Removing files#

To remove a file (i.e., stop version controlling it) use git rm:

echo "Text in a file to remove" > FileToRem.txt

git add FileToRem.txt

git commit -am "added a new file that we'll remove later"

[master 706f40d] added a new file that we'll remove later
 2 files changed, 3 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 FileToRem.txt

git rm FileToRem.txt

rm 'FileToRem.txt'

git commit -am "removed the file"

[master 284c62e] removed the file
 1 file changed, 1 deletion(-)
 delete mode 100644 FileToRem.txt

I typically just make all my changes and then just use git add -A for the whole directory (and it’s subdirectories; -A is recursive).

git rm --cached <file>

Accessing history of the repository#

To see particular changes introduced, read the repo’s log :

git log

For a more detailed version, add -p at the end.

git reset --hard
git commit -am "returned to previous state" #Note I used -am here

git status

git log

git checkout *version number*

git checkout master

Running git commands on a different directory#

Since git version 1.8.5, you can run git directly on a different directory than the current one using absolute or relative paths. For example, using a relative path, you can do:

git -C ../SomeDir/ status

Using git through a GUI#

There are many nice git GUI’s (Graphical User Interfaces) out there, such as gitKraken. Or if you are using a code editor like Visual Studio Code, there are nice extensions that will give you considerable GUI functionality.

A friendly concluding note on git#

Git can feel hard and unintuitive. It is likely that at some point you will make a mistake and lose some work as a consequence.

THIS IS NORMAL

Don’t worry about it, this has happened to pretty much everyone who has ever used git.

When things like this do happen though, it is imperative that you work out what went wrong so you can make sure it doesn’t happen again.

Questions to ask yourself include:

• Did I use git status to check what I was adding?

• Did I commit regularly enough, whenever I had completed a logical section of work?

• Did I make sure to only commit code rather than committing data files and outputs too?

• Did I make sure to gitignore the things I needed to?

When you get used to working with git, it becomes as natural as juggling chainsaws: i.e. it will always feel a bit odd and strange but it won’t be as scary as it once was.

Next Steps:

• Learn Commands: Familiarize yourself with common Git commands.

• Explore Branching: Practice creating, switching, and merging branches.

• Use Hosting Services: Try using platforms like GitHub to collaborate with others.

• Advanced Features: Delve into rebasing, cherry-picking, and submodules for more complex workflows.

Practicals#

Readings & Resources#