Gitting Good - Using Submodules to Manage Dependencies - Keyva

Have you ever worked on a project that required some sort of third-party dependency, and didn’t have the luxury of a mature package manager such as Python’s pip, JavaScript’s npm or Rust’s cargo?

Whatever the reason, be it use of a language that doesn’t have a standard package manager such as C++, or a specific internal repo not available as a package, Git has a feature to ease the pain of managing these external dependencies, known as submodules.

At the core, the submodule system is a means of importing a specific commit into a project. These submodules are managed separately of the rest of the repository, and the only data pushed to your project is the information on how to fetch the submodule, rather than pushing the entire contents of a needed dependency.

Adding a submodule

Before adding a submodule to a project, consider mirroring the target repository to your own Git instance. If a target repository server goes down, then a fresh pull of your project will not be able to receive this code without some effort (see Gitting Good – How to Collaborate Without a Central Service for sample workarounds), so it’s better to be prepared ahead of time by having a regularly syncing mirror repository.

Once this has been done, you can add your dependency as its only directory by using the git submodule feature:

# Adding a submodule at the default (master/main) branch’s HEAD

git submodule add https://git.keyva.internal/externals/openssl.git contribs/openssl

# Or, using '-b' to declare a specific branch to use

git submodule add -b openssl-3.6 https://git.keyva.internal/externals/openssl.git contribs/openssl

A new file named .gitmodules will be added to your project root the first time a submodule is added:

[submodule "contribs/openssl"]

path = contribs/openssl

url = gitea@https://git.keyva.internal/externals/openssl.git

branch = openssl-3.6

An entry in .git/config will also be created:

[submodule "contribs/openssl"]

active = true

url = gitea@https://git.keyva.internal/externals/openssl.git

Whenever a new submodule is added, the .gitmodules file should be committed to track that change. No other objects need to be added to the index.

Pulling submodules within a project

Cloning a project does not automatically also clone submodules within that project. Cloning submodules can be done in two main ways:

# At the time of cloning:
git clone --recurse-submodules https://git.keyva.internal/projects/test-app.git

# Or, if the project is already cloned:
git submodule update --init --recursive

The –recursive flag is used to handle nested submodules; that is to say: submodules of submodules.

Updating submodules

It is important to note that submodules track a specific commit, rather than a branch, though defining a branch as above will tell Git what branch the desired commit is in.

For this reason, new updates to a submodule’s project are not automatically pulled in. The purpose of this is in part to ensure that a project’s state, that is to say each commit, remains consistent. Should it be needed to roll back to an earlier commit to check on prior behavior, that commit will point to the same commit of the submodules as when it was originally authored.

Because of this, submodule updates are done with a different step from pulling the main repository:

# Update everything recursively
git submodule update --remote –recursive


# Commit the new updates after testing
git add contribs/*
Git commit –m "Update submodules"

Removing submodules

For the same reasons that updating submodules is a seperate process from the main repository, removing submodules also requires a different process to be “clean”:

# Deinitialize
git submodule deinit -f -- contribs/openssl

# Remove from index and working tree
git rm -f contribs/openssl

# Remove metadata
rm -rf .git/modules/contribs/openssl

# Commit
git add .gitmodules
git commit -m "Removing openssl submodule"

Common issues

When within a submodule directory, you exist in what is effectively that submodule’s project, and within a detached HEAD state of that project. If changes need to be made, a branch reference must be defined first:

# Get to the project directoy
cd contrib/openssl


# Checkout an existing branch
git checkout main 
# Or create a new one
git checkout –b keyva-dev


# Do whatever changes are needed, and then:

git push origin keyva-dev # Still in the submodule's directory


# Move back to main project directory
cd ../..

# Update the pointer for the submodule
git add contrib/openssl

# And then finally commit
git commit -m "Update submodule to latest"

Another somewhat common issue is changing the repository URL. This can be because a project owner changed where their repository is hosted, or maybe because you started with an external reository and now what to move to an internal, mirrored one.

URLs can be updated first by editing the relevant area in .gitmodules, followed by running git submodule sync, which will update .git/config. Add .gitmodules to the index and commit once tested.

Common uses

As previously mentioned, the primary use case for submodules is importing external code to a project. However, this also makes it a good case for avoiding code reuse and drift. For instance, one C++ project I use internally handles common features such as architecture-specific handling, string utilities and special math handling. This project is used by all other end projects and, as a result of being included as a submodule, the code for it has a single source of truth. This means that code isn’t needlessly duplicated across projects, and behavior doesn’t drift between projects because one feature exists in one project but not others.

Use of submodules is also a decent way to anchor commits. If a submodule’s repository frequently pushes breaking changes, declaring a specific commit as the one to be using is an easy solution, allowing updating to these submodules at one’s own pace, rather than being made to adjust to changes in the middle of another workflow.