Close

FAQ

1. General

1.1 What is Software Heritage (SWH)?

Software Heritage is an open, non-profit initiative unveiled in 2016 by Inria. It is supported by a broad panel of institutional and industry partners, in collaboration with UNESCO.

Expand for details

 

The long term goal is to collect all publicly available software in source code form together with its development history, replicate it massively to ensure its preservation, and share it with everyone who needs it.

For more information about the SWH mission.

1.2 What is the SWH archive?

The SWH archive is the largest public collection of source code in existence. Visit the archive on https://archive.softwareheritage.org.

1.3 What is the size of the archive?

The archive is growing over time as we crawl new source code from software projects and development forges. You can see live counters of the archive contents, as well as a breakdown by crawled origins, on https://archive.softwareheritage.org.

1.4 What are the services provided by SWH?

Software Heritage is a mutualised platform that offers a growing number of services to a large spectrum of users.

The features page provides an overview of the features currently available. This includes, for example, archiving software repositories, browsing the archived source code and providing persistent identification.

2. Archiving software

2.1 Which software platforms (forges, package managers, etc.) are archived?

The software origins that are currently regularly archived are listed on the main archive page.

Expand for details

 

Here is an excerpt of this list:

  • git repositories from multiple forges (github, bitbucket, gitlab instances, cgit instances, gitea instances, phabricator instances, etc.)
  • svn repositories…
  • mercurial repositories…
  • Debian packages in apt
  • Python packages in PyPI
  • R packages in CRAN
  • NPM packages in npm.org
  • zip or archives tarballs in gnu.org

2.2 If my code is on GitHub/GitLab/Bitbucket, is it already archived in SWH?

It might be, as we crawl these and other popular forges regularly.
Search for your code repository on https://archive.softwareheritage.org/browse/search/.

Expand for details

 

If it is not there yet, or if the latest snapshot is not the most recent state of your repository, you can trigger a new archival using the Save Code Now functionality
https://archive.softwareheritage.org/save/ or by clicking on the “Save again” button in the browse view.

2.3 If I delete the repository that contains my code, will the data stay in SWH?

Yes, all software source code artifacts are preserved for the long-term.

2.4 What is the policy for determining what deserves to be archived? are there requirements for a GitHub, Gitlab or XXX repository to be archived by SWH?

We do not inspect or filter the source code, and archive anything that we can get hold of. As a consequence, here are no requirements but we do suggest following the SWH guidelines for best results.

Expand for details

 

The reason for this approach is because the value of software source code cannot be known in advance. When a project starts, one cannot predict whether it will become a key software component or not. For example, when Rasmus Lerdorf released the first version on PHP back in 1995, who could have predicted that it would become one of the most popular tools for the Web.

XKCD Comics https://xkcd.com/2347/ (CC BY-NC 2.5)

And it also happens that very precious pieces of source code may be go unnoticed for decades, until one day some unexpected bug unveils that a big part of our digital infrastructure relies on them.

2.5 Is the code checked for LICENSE file or any specific characteristic in the repository before archiving?

Software Heritage archives everything that is publicly available, without preliminary tests or checks.

This means that you are responsible for checking whether the source code you find in the archive can be reused, and under which terms.

For the code that you produce, we do suggest following standard best practices, that are recalled in the SWH guidelines, and this include adding licensing information.

2.6 Do you also archive software executables (aka binaries)?

Our core mission is to preserve source code, because it is human readable and contains precious information that is stripped out in the executables. As a consequence, we do not actively archive binaries, but if binaries are included in a software repository, we do not filter them out in the archival process. Hence you can find a few binaries in the archive.

2.7 I can’t find all my “releases” in a git repository in Software Heritage, what should I do?

Do not worry, your repository has been saved in full.
What you are witnessing is just a terminological difference between what
platforms like GitHub calls “releases” (any non annotated git tag) and what we call “releases” (a node in the Merkle tree, which corresponds to a git annotated tag). This is a common issue, as you can see for example in this discussion thread.

Expand for details

 

Let’s say you tagged your release naming it “FinalSubmission”, but you did not use an annotated tag: in this case, it will not show up in the Releases tab on Software Heritage, but it is there nonetheless! Click on the branch dropdown menu on the Software Heritage Web interface and you’ll find it listed as “refs/tags/FinalSubmission”. If you want a release to appear in our web interface you should create your tags
using “git tag -a”, instead of simply “git tag”, or create the release directly on the code hosting platform, that uses the proper “git tag -a” behind the scenes, and then archive your
repository again.

3. Referencing and identification

3.1 What is a SWHID (Software Heritage Identifier)?

The SWHID (Software Heritage Identifier), is a persistent intrinsic identifier that is computed uniquely from the software artifact itself. See the dedicated blog post to learn more about intrinsic and extrinsic identifiers.

Expand for details

 

All details about the syntax, semantics, interoperability and implementation can be found in the formal specification.

The following diagram shows concisely the key components of a SWHID:

Credit: Software Heritage, 2019

The top yellow box in the diagram corresponds to the “core SWHID“. It is possible to add qualifiers to a core SWHID in order to provide additional information about the location of the object in the Software Heritage graph, or its origin, and to identify code fragments.

3.2 What can be identified with a SWHID?

First, let’s notice that software can be identified at quite different levels of granularities, ranging from a conceptual level (e.g. the name of a software project), to concrete software artifacts (e.g. a directory containing plenty of files).

Credit: Research Data Alliance/FORCE11 Software Source Code Identification WG, 2020, https://doi.org/10.15497/RDA00053

The Software Heritage identifiers are designed to identify permanently and intrinsically all the levels of granularity that correspond to concrete software artifacts: snapshots, releases, commits, directories, files and code fragments.

Expand for details

 

A core SWHID can be used to identify the following source code artifacts:

Using the “lines” qualifier, it is possible to also identify “code fragments“, i.e. selected lines of code.

For example, swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2;lines=4-6 pinpoint lines 4 to 6 of the full text of the GPL3 license.

More generally, using a fully qualified SWHID provides all relevant information for placing a software artifact in context. For example, the following SWHID pinpoints the core mapping algorithm contained in the file parmap.ml, located in the src directory of a specific revision of the Parmap project retrieved from https://github.com/rdicosmo/parmap

swh:1:cnt:3b997e8ef2e38d5b31fb353214a54686e72f0870;origin=https://github.com/rdicosmo/parmap;visit=swh:1:snp:25490d451af2414b2a08ece0df643dfdf2800084;anchor=swh:1:rev:db44dc9cf7a6af7b56d8ebda8c75be3375c89282;path=/src/parmap.ml;lines=192-237

3.3 How can I get a SWHID for my software?

The core SWHID identifier is intrinsic, so yes, you can compute the core SWHID of any software artifact locally on your machine! You can find instructions in the documentation available here!

You can also get the full SWHID for any archived software artifact directly from the Software Heritage archive: using the red vertical tab called “Permalinks”, present on every page that shows source code (see this HOWTO for more details). The advantage of this second approach is that you can get a SWHID with relevant contextual information (e.g. the position of your artefact in the global graph of software development).

Expand for details

 

The “Permalinks” tab let you obtain a SWHID for the content that you are browsing. Here is an example:

Clicking on “Copy identifier” you can get the SWHID in your clipboard. Clicking on “Copy permalink” you can get in your clipboard the corresponding URL.

The “Add contextual information” checkbox allows you to choose whether you will get a core SWHID or the SWHID with the extra qualifiers that provide contextual information.

Notice that the Permalinks tab offers a plurality of options to pick a SWHID (you may get the one for the file content, the directory that contains it, the revision, the release or the snapshot). See the following question to understand which is best for your use case.

3.4 Which type of SWHID should I use in my article/documentation?

It really depends on your use case, but as a general suggestion we recommend to take the full SWHID of a directory (with the contextual information).

Expand for details

 

When writing a research article, a blog post or technical documentation, one may face some tension between the need to provide the maximum amount of information, using the full SWHID, or keeping the reference short (for example due to page limitations).

Here is the recommended best practice to address this issue:

  1. get the full SWHID for the ‘directory’ containing the version of the code you want to reference. Here is an example of such a full SWHID:
    swh:1:dir:013573086777370b558b1a9ecb6d0dca9bb8ea18;origin=https://gitlab.com/lemta-rheosol/craft-virtual-dma;visit=swh:1:snp:ef1a939275f05b667e189afbeed5fd59cca51c9d;anchor=swh:1:rev:ad74f6a7f73c7906f9b36ee28dd006231f42552e
  2. ensure the “core SWHID” (swh:1:dir:013573086777370b558b1a9ecb6d0dca9bb8ea18 in the example above) is printed, and the full SWHID is available at least as an hyperlink.

This effect can be achieved as follows in LaTeX:

\href{https://archive.softwareheritage.org/swh:1:dir:013573086777370b558b1a9ecb6d0dca9bb8ea18;origin=https://gitlab.com/lemta-rheosol/craft-virtual-dma;visit=swh:1:snp:ef1a939275f05b667e189afbeed5fd59cca51c9d;anchor=swh:1:rev:ad74f6a7f73c7906f9b36ee28dd006231f42552e/}{swh:1:dir:013573086777370b558b1a9ecb6d0dca9bb8ea18}

or in Markdown: swh:1:dir:013573086777370b558b1a9ecb6d0dca9bb8ea18

This approach ensures that in any printed version the reader will find the identifier that is most useful for reproducibility: the core SWHID of the directory. Indeed, the core SWHID of a directory can be locally computed from any instance of a source code, independently of the release or commit that cointains it in a specific project.

In the digital version the clickable link uses the full SWHID to let the reader browse the code in Software Heritage with all the proper context (version, origin, etc. etc.).

3.5 I want the full SWHID for a source code that is not already in the archive. How can I proceed? How long will it take?

If your code (or the latest version of it) is not yet in the archive, you need first to trigger its archival. This can be done with a “Save Code Now” request, or via the deposit API.

Once a Save Code Now request is issued, the ingestion of the code is usually completed in a few minutes, depending on the size of the repository. Once it’s done, the status of the save request is updated and you can get the SWHID as shown before.

When a deposit is submitted, the ingestion is also usually completed in a few minutes and the SWHID is accessible through the SWORD status response.

4. Access and Reuse

4.1 Can I reuse the source code artifacts I find on SWH?

It depends on the license of the artifact, as stored alongside the source code: you must check this license before downloading or reusing it. If you cannot find the license information, you should assume that you have no right to reuse it.

Expand for details

 

All software components present in the Archive may be covered by copyright, or other rights like patents or trademarks. Software Heritage may provide automatically derived information on the software license(s) that may apply to a given software component, but it makes no claim of correctness and the licence information provided does not constitute legal advice. You are solely responsible for determining the license, or other rights that apply to any software component in the Archive, and you must abide by its terms.

4.2 Can I clone a repository using SWH?

Please do not clone a full repository directly from SWH: it is an archive, not a forge. Try first to clone a repository from the place where it is developed: it will be faster and as an added bonus you will be already in the right place to interact with its developers.

Expand for details

 

SWH stores all the software artifacts in a massive shared Merkle tree, so that exporting (a specific version of) an archived respository implies traversing the graph to get all the relevant contents and packaging them up for your consumption. This operation is much more expensive than downloading an existing tar file or cloning a repository from a forge.

If really SWH is your last resort, and you cannot find the source code of interest elsewhere, we recommend that you download only the version of interest for you, using the “directory” option of the Download button that you find when you browse the archive.

If absolutely needed, you can use the more expensive “revision” option of the Download button, that will prepare for you the equivalent of a git bare clone , which you will be able to use offline. This may require quite some time (hours, or even days for huge repositories).

4.3 Can I retrieve a source code artifact through the API?

Yes, you can. If you have the SWHID at hand, you can use the appropriate API method for it to navigate through the endpoints to follow the graph of project artifacts. Checkout the API documentation for the complete list of endpoints.

Expand for details

 

  • /api/1/snapshot/ which allows you to get the snapshot’s branches and tags, each with a target_url key that contains the URL to
  • /api/1/release/ or /api/1/revision/ which allow you to get the revision’s or releases’ data. Assuming you get a revision, the directory_url key contains a URL to:
  • /api/1/directory/ which lists entries of the root directory, with links to other directories and content objects
  • /api/1/content/ which returns all the information about a given file content, including a link to the raw data.

You can also lookup an origin, and follow its visits:

  • /api/1/origin/search/ allows you to search the exact URL of the code repository
  • /api/1/origin/visits/ allows you to list the times Software Heritage visited the repository, and get the snapshot associated with each visit. For each visit, this snapshot is available as a snapshot_url key, that contains the URL to get the corresponding snapshot object.

If you are interested in downloading a large part of the repository (a directory or a set of revisions), you should use the download service called the Vault. The Vault allows you to fetch them in batch and download a tarball. The list of vault endpoints is available at the end of the list of all API endpoints

5. Software metadata

5.1 Can I add metadata to my software?

A regular user can add metadata files in the repository which will be ingested and indexed when using a specific file format (codemeta.json, package.json, pom.xml, etc.).

Expand for details

 

Follow the SWH guidelines on how to prepare your code for archival.

More information about the formats that are indexed and some general overview of the metadata workflow in the blog-post about mining for software metadata

5.2 What metadata are preserved from a code repository, with save code now?

All metadata contained by the source code repository itself is preserved. This will include the development history and commit dates and messages.
At the moment, other metadata artifacts which are not part of the repository (known as extrinsic metadata) are not preserved when using the Save Code Now feature.

5.3 What metadata are preserved with a deposited software artifact?

All metadata which is sent via the SWORD protocol accompanying the software artifact. For more information visit the deposit documentation.

5.4 What is the codemeta.json file, why should I use it?

As software developers, we may want to provide a machine readable description of our projects, but there are (too) many metadata schemas for describing software, and one can easily get lost.

The CodeMeta initiative created a common vocabulary to address this issue, based on (a slight extension of) the SoftwareApplication and SoftwareSourceCode classes of the well established schema.org initiative, and provides tools to convert back and forth from other medatada schemas.

The codemeta.json file is a JSON-LD representation of the CodeMeta vocabulary, that can be easily created and validated using the Open Source codemeta generator tool. By adding a codemeta.json file to your project, you make it easy to share metadata information, and reduce the burden of retyping a lot of information in data entry forms.

Expand for details

 

For example, the french HAL national open access archive looks for a codemeta.json file when a software project archived in Software Heritage is deposited, and pre-fills the deposit form using the information it contains, a real time saver!

Last but not least, Software Heritage indexes the metadata contained in codemeta.json files and makes it searchable on the web-app using the CodeMeta crosswalk table. The crosswalk table is the Rosetta stone of software metadata, facilitating translation between ontologies and metadata standards for software.

Credit: Gruenpeter M. and Thornton K. (2018) Pathways for Discovery of Free Software (slide deck from LibrePlanet 2018). https://en.wikipedia.org/wiki/File:Pathways-discovery-free.pdf

5.5 Does Software Heritage perform a check on the metadata (e.g. to verify whether a licence is declared)?

The short answer is no. Software Heritage does not perform any a priory filtering of the repositories that are archived.