Why build a new CI system from scratch?

Problems we see

These are not in any kind of order.

• Debugging: when (not if) there is a build failure, it can be tedious and frustrating to figure what the cause is and to fix that. Often the failure can be difficult to reproduce locally, or otherwise in a way that can be inspected except via the build log.

• Capacity: individuals and small organization often don’t have a much capacity to spare for CI purposes, which hampers and slows down development and collaboration.

• Generality: many CI systems run jobs in containers, which has a low overhead, but limits what a job can do. For example, containers don’t allow building on a different operating system from the host or on a different computer architecture.

• Security: typically a CI system doesn’t put many limits on what the job can do, which means that building and testing software can be a security risk.

  • example: exploiting Drone CI instances with open registration: https://manganiello.social/@fabio/posts/Arivp4ZjiD2KUwpwJs

Needs and wants we have

These are not in any kind of order.

• We want to build software for different operating systems and computer architectures, and versions thereof, as one project, with the minimum of fuss. One project should be able to build binaries and installation packages for any number of targets, and update web sites, and so on, as part of one build.

• It must be possible to construct and update the build environments within the CI system itself. For example, by building the virtual machine base image for build workers.

• We want builds to be quick. The CI system should add only little overhead to a build. When it’s possible to break a build into smaller, independent parts, they are run concurrently as much hardware capacity allows.

• We want it to be easy to provide build workers, without having to worry about the security of the worker host, or the security of the build artifacts.

• If a build is going to fail for a reason that can be predicted before it even starts, the job should not start. For example, if a build step runs a shell command, the syntax should be checked before the job starts. Obviously this is not possible in every case, but in the common case it is.

• Build failures should be easy to debug. Exactly what this means is unclear at the time of writing, but it should be a goal for all design and development work.

• It’s easy to host both client and server components.

• It’s possible, straightforward, and safe, for workers to require payment to run a build step. This needs to be done in a way that is unlikely to anyone being scammed. Paying for resources is necessary for continued availability of the resources.

• Is integrated into major Git hosting platforms (GitHub, GitLab, etc), but is not tied to any Git platform, or Git at all.

• Build logs can be post-processed by other programs.