ARM64 Bootstrap for VyOS

This repo can assist in bootstrapping a VyOS image for arm64, without depending on the artifacts already generated by Jenkins.

The generated image is an ISO that will boot on ARM Embedded and Server Base Boot systems (e.g those that use EFI)

There are also a few minor hacks/patches applied which should be upstreamed shortly.

If you are unfamiliar with the VyOS build process, see the Build VyOS page in the documentation.

Current hardware targets:

• QEMU ARM64 Virtual Machine with EDKII
• Traverse Technologies Ten64

Note: only basic routing and NAT functionality has been tested at this time.

Build requirements:

• ARM64 host (no cross compiling)

• It is best to use a Debian machine as the build host. Issues have been encountered with AppArmor on Ubuntu

• Working docker installation

• sudo access from the current user

Other efforts:

Graham Hayes built his own version here: https://github.com/grahamhayes/vyos-build/. The main difference is that this one doesn't use a container to do builds. See Ten64 forum discussion

Usage:

My vyos-build branch is setup as a submodule of this repository, so you should do a recursive clone first:

git clone --recursive https://github.com/mcbridematt/vyos-arm64-builder.git

Run ./build.sh (as root or sudo) to build the 'vyos-build' container and then packages and image.

sudo ./build.sh

An ISO image will be generated under vyos-build/build/vyos-{VYOS_VERSION}.{YYYYMMDDHHMM}-arm64.iso

Package commit pinning

The source versions of each component are "pinned" to a commit or branch in repos.txt to ensure this environment will produce a working image. To use the latest available current/development versions, copy repos.development.txt in its place.

Testing inside a VM

The ./testimg.sh script can be used to boot the generated ISO image as a QEMU/KVM virtual machine.

You need to run the script on a genuine arm64 host with virtualization capabilities.

(It is possible to run it under emulation, but YMMV).

An EDK2 "bios" binary is required, under Debian you can use the qemu-efi-aarch64 package or grab a build from retrage/edk2-nightly.

cp vyos-build/build/vyos-{VYOS_VERSION}.{YYYYMMDDHHMM}-arm64.iso vyos.iso
sudo ./testimg.sh

(Hint: Use Ctrl-X to immediately exit qemu).

Running on real hardware (Ten64)

On a Ten64 a suitable block medium (NVMe SSD, USB drive or SD card) is needed. It might be possible to fit VyOS into the onboard NAND/ubifs in the future.

Write the VyOS ISO to a USB drive using dd or a similar tool:

dd if=vyos.iso of=/dev/sda

Then boot into the 'live' VyOS system and do an install:

vyos@vyos:~$ install image

See the VyOS Installation doc for more information.

Known issues:

• On a Ten64, when the live image is booted from USB, there may be a hang on reboot as something tries to do a write to the USB. You will need to restart the system using the reset button if this happens.
• There are some hacks to bypass or remove certain aspects that can cause trouble:

  • udev interface renaming rules and biosdevname handling - need to debug issues with this not working properly.

  • Only one GRUB console entry - "VyOS" is added, vs x86 VyOS which has KVM/serial/USB serial choices. As long as the system firmware has the "chosen" console set correctly (in FDT or ACPI), there should be no issues as systemd and the kernel will chose the "right" console.

    Forcing an "external" console is a use-case we will need to consider. (Also, systems with the ARM standard UART, like qemu-system-aarch64 -M virt have ttyAMA0 instead of ttyS0. So I'd rather avoid specifying every possible console= combination)

  • Secure boot (SHIM) is removed from the Live ISO as the old version of U-Boot on the Ten64 doesn't support the secure boot EFI protocol. Once the Ten64 firmware is updated this can be put back in.

• Previously this repository generated a qcow2 image, but this is currently not possible due to recent changes in vyos-build. Hopefully it can be bought back soon.