Git Images Tutorial Contact

Ambient CI is a CI/CD system that aims to make it safe and secure to run CI on other people’s code. It runs the dangerous actions for a project in a virtual machine that has no network access.

Maturity: Ambient is alpha level software. It just about works, but is not feature complete and will change in inconvenient ways. It is quite horrifyingly awkward to use. This web site describes what we want Ambient to become, not what it is yet.

  1. Overview: what Ambient does
  2. Glossary
  3. Motivation
  4. Visions of CI
  5. Threat modeling
  6. Architecture
  7. License

Overview: what Ambient does

Ambient processes source files into artifacts and optionally delivers or deploys them relevant parties.

At a very high level, this means that there is a source tree with some files that are part of a project. In the typical case, the source tree has the source code for some software, but it may also be the source code for a web site, the configuration for one or more servers, or anything else that can be expressed as files.

At various times, the source gets processed in an automated way. This is triggered by something, such as a change to the source tree, time passing, manually, or something else. For a software project, the processing might mean compiling the source code into binary form or running some automated tests.

After the processing, there may be artifacts and these may be automatically delivered to one or more recipients, which means they’re published so they may be downloaded. The artifacts may also be automatically deployed on some computer that will make use of them. In a common software scenario, the deployed code will be used to run a web site or service.

Each project gets to specify, in some way, how the processing is done. For many existing CI systems, this is done using some form of file in the source tree. In the simplest form, this is a shell script that gets run, but a Makefile is common. The GitLab .gitlab-ci.yml file is also an example of this.

In contemporary systems, the processing is usually more or less linear. Even when parts are concurrent, they usually run on a single machine. For Ambient, the long-term goal is to allow the processing to be spread over wildly and widely separated machines that don’t know about each other and may lack a way to communicate directly. This makes everything more complicated and error prone, but allows collaboration without tight co-ordination or strong trust.

Glossary

These are the terms we have chosen for the Ambient project. We hope they make sense without having to read the definition. The terms are meant to be generic for CI/CD systems, and not specific for Ambient.

  • artifact—the output of processing a source tree
  • delivery—transfer of artifacts to a recipient or publishing them for relevant parties
  • deploy, deployment—delivery of artifacts to a recipient that starts using them
  • plan—a specification how a source tree is processed into artifacts, and how the artifacts are delivered or deployed; consists of steps arranged in a directed acyclic graph
  • project—one or more source trees; often with additional organization to co-ordinate work on the files
  • run—an execution of a plan; may produce artifacts that get delivered or deployed
  • source tree—a set of files that can be processed according to a plan
  • step—an atomic step in a plan
  • worker—the machine the executes one or more steps in a plan

Motivation

Problems we see

These are not in any kind of order.

  • Debugging: when (not if) there is a build failure, it can be tedious and frustrating to figure what the cause is and to fix that. Often the failure can be difficult to reproduce locally, or otherwise in a way that can be inspected except via the build log.

  • Capacity: individuals and small organization often don’t have a much capacity to spare for CI purposes, which hampers and slows down development and collaboration.

  • Generality: many CI systems run jobs in containers, which has a low overhead, but limits what a job can do. For example, containers don’t allow building on a different operating system from the host or on a different computer architecture.

  • Security: typically a CI system doesn’t put many limits on what the job can do, which means that building and testing software can be a security risk.

Needs and wants we have

These are not in any kind of order.

  • We want to build software for different operating systems and computer architectures, and versions thereof, as one project, with the minimum of fuss. One project should be able to build binaries and installation packages for any number of targets, and update web sites, and so on, as part of one build.

  • It must be possible to construct and update the build environments within the CI system itself. For example, by building the virtual machine base image for build workers.

  • We want builds to be quick. The CI system should add only little overhead to a build. When it’s possible to break a build into smaller, independent parts, they are run concurrently as much hardware capacity allows.

  • We want it to be easy to provide build workers, without having to worry about the security of the worker host, or the security of the build artifacts.

  • If a build is going to fail for a reason that can be predicted before it even starts, the job should not start. For example, if a build step runs a shell command, the syntax should be checked before the job starts. Obviously this is not possible in every case, but in the common case it is.

  • Build failures should be easy to debug. Exactly what this means is unclear at the time of writing, but it should be a goal for all design and development work.

  • It’s easy to host both client and server components.

  • It’s possible, straightforward, and safe, for workers to require payment to run a build step. This needs to be done in a way that is unlikely to anyone being scammed. Paying for resources is necessary for continued availability of the resources.

  • Is integrated into major Git hosting platforms (GitHub, GitLab, etc), but is not tied to any Git platform, or Git at all.

  • Build logs can be post-processed by other programs.

Visions of CI

Megalomaniac vision: CI is everywhere, all the time

Ambient links together build workers and development projects to provide a CI system that just is there, all the time, everywhere. Anyone can provide a worker that anyone can use. The system constrains build jobs so that they can only do safe things, and can only use an acceptable amount of resources, and guarantees that the output of the build can be trusted.

(This may be impossible to achieve, but we can dream. If you don’t aim for the stars you are at risk of shooting yourself in the foot.)

Plausible vision: CI is easy to provide and use

The various components of Ambient are easy to set up, and to keep running, and to use. Within an organization it’s easy to access. It’s so easy provide a worker on one’s own machine, without worry, that everyone in the organization can be expected to do so. Builds can easily be reproduced locally.

Realistic vision

Ambient provides a command line tool to run a job in a safe, secure manner that is easily reproduced by different people, to allow collaborating on a software development project in a controlled way. (Some of this works now.)

Threat modeling

This model concerns itself with running a build locally. Some terminology:

  • project – what is being built
  • host – the computer where Ambient is run

Many software projects require running code from that project to be built, and certainly when it’s automated tests are run. The code might not be directly part of the project, but might come from a dependency specified by the project. This code can do anything. It might be malicious and attack the build host. It probably doesn’t, but Ambient must be able to safely and securely build and test projects that aren’t fully trusted and trustworthy.

The attacks we are concerned with are:

  • reading, modifying, or storing data on the host, in unexpected ways
  • using too much CPU on the host
  • using too much memory on the host
  • using too much disk space on the host
  • accessing the network from the host, in unexpected ways
  • leaking secrets (passwords, encryption keys) from the host

Prevention

We build and test in a local virtual machine.

The VM has no network access at all. We provide the VM the project source code via a read-only virtual disk. We provide the VM with another virtual disk where it can store any artifacts that should persist. Both virtual disks will contain no file system, but a tar archive.

We provide the VM with a pre-determined amount of virtual disk. The project won’t be able to use more.

We provide the VM with an operating system image with all the dependencies the project needs pre-installed. Anything that needs to be downloaded from online repositories is specified by URL and cryptographic checksum, and downloaded before the VM starts, and provided to the build via a virtual disk to the VM.

We interact with the VM via a serial console only.

We run the VM with a pre-determined amount of disk, number of CPUs, amount of memory. We fail the build if it exceeds a pre-determined time limit. We fail the build if the amount of output via the serial console exceeds a pre-determined limit.

Architecture

At a very abstract level, the Ambient architecture is as follows:

  • Ambient creates a virtual machine with four block devices (using virtio_blk) in addition to the system disk (/dev/vda on Linux):
    • /dev/vdb: the read-only source device: a tar archive of the project’s source tree
    • /dev/vdc: the read/write artifact device: for the project to write a tar archive of any build artifacts it wants to export
      • this would be write-only if that was possible
      • when the build starts, this contains only zeroes
      • after the build a tar archive is extracted from this
    • /dev/vdd: the read-only dependencies device: a tar archive of additional dependencies in a form that the project can use
    • /dev/vde: the read/write cache device: a tar archive of any files the project wants to persist across runs; for example, for a Rust project, this would contains the cargo target directory contents
      • when a build starts, this can be empty; the build must deal with an empty cache
  • The VM additionally has a serial port where it will write the build log. On Linux this is /dev/ttyS0.
  • The VM automatically, on boot, creates /workspace/{src,cache,deps}, and extracts the source, cache, and dependencies tar archives to those directories.
  • The VM then changes current working directory to /workspace/src and runs ./.ambient-script (if the script isn’t executable, the VM first makes it so). The script’s stdout and stderr are redirected to the serial port.

Data is transferred out of the VM as tar archives instead of disk images with file systems, to allow the data to be unpacked with user space tooling only. Mounting a disk image in a way that involves the kernel is risky. For example, an ext2/3/4 file system can tell the kernel to panic if there’s is corruption, and the CI run can corrupt the file system on purpose. See for example https://infosec.exchange/@wdormann/113625346544970814.

The ambient-build.service and ambient-run-script files in the Ambient source tree implement this for Linux with systemd, and have been tested with Debian.

License

All content on this web site is copyrighted by Lars Wirzenius, and other contributors, and licensed under a Creative Commons Attribution-Share Alike 4.0 Unported License.

The Ambient CI software is licensed under the GNU Affero General Public License, version 3, or (at your option) any later version.