Skip to content
This repository has been archived by the owner on Nov 30, 2023. It is now read-only.

Support ARM64 containers / images #558

Closed
Chuxel opened this issue Sep 15, 2020 · 28 comments
Closed

Support ARM64 containers / images #558

Chuxel opened this issue Sep 15, 2020 · 28 comments
Assignees
Labels
feature-request Request for new features or functionality notice

Comments

@Chuxel
Copy link
Member

Chuxel commented Sep 15, 2020

UPDATE - Images previously built out of this repository are now at https://github.com/devcontainers/images

Please raise issues there. Jammy and Bullseye images now exist and are broadly supported. We have no plans for arm64 support for Focal or Buster image nor other distros like centos:8 that experience random crashing. See here for details.

---- ORIGINAL TEXT ---

Currently the definitions in this repository are x86_64 based. Some scripts will likely need to be modified to support ARM64. In addition, pre-built definitions will need to be compiled for ARM64 after this has been done.


UPDATE

Images listed below should now all be released with arm64 variations (you can now omit the dev prefix and use 0- or another sub-version as you see fit). UX in the Add Dev Container Configuration Files command in VS Code will be getting updated incrementally in Remote - Containers and Codespaces to list -bullseye variants with amd64/Apple Silicon support (along with other variants where possible for backwards compatibility).

To update existing dev container configuration files from this repository, you can append -bullseye to the VARIANT argument. For many configuration, this will be in devcontainer.json. For example, for Python:

"build": {
    "dockerfile": "Dockerfile",
    "args": {
        "VARIANT": "3-bullseye"
    }
}

Cases where the VARIANT argument is in docker-compose.yaml are similar. For example, for Python 3 & Postgres:

#...
    build:
      dockerfile: Dockerfile
      args:
        VARIANT: 3-bullseye
#...

We have also updated mcr.microsoft.com/vscode/devcontainers/base:alpine and related tags now that Remote - Containers supports Alpine arm64 as of the release of VS Code 1.61.

After the C#/.NET extension added support for ARM, arm support for the dotnet images's bullseye variations have ARM64 support starting with version 0.203.0 of the .NET images (e.g. mcr.microsoft.com/vscode/devcontainers/dotnet:0.203.0-6.0-bullseye) in release v0.224.0.

@gfmatthews
Copy link
Member

Hey y'all - I have a Surface Pro X and am interested in helping to get some of these containers supported on the ARM64 platform. I've seen that javascript-node builds correctly without modification already for ARM64 so for that container, the only thing missing is getting the pre-built containers updated.

Is there a build pipeline or something similar somewhere that can be used to test this out? Happy to reach out internally (@microsoft) if that would be helpful.

@jasonsummers
Copy link

Hi all, new to GitHub - apologies in advance for breaches of etiquette, please correct any mistakes I make.

I'm keen to help make progress in on this issue and plan to look at how multi-arch images can be used as well as cleaning up / automating debugging linux on ARM. As a temporary solution I've managed to get a working .net core devcontainer working by building from the arm64 sdk image and using the scripts from this repo.

If work is already in progress on resolving this issue, please let me know if and how I can be of help.

@philliphoff
Copy link
Contributor

philliphoff commented Feb 28, 2021

I've run into a qemu issue when building upon one of the base amd64 images with a new M1 Mac (for testing our Docker tooling on the new platform). It would be great to have arm64 images to test.

@tkreindler
Copy link

I'm on an Apple M1 and although I totally understand this is beta and low priority, it would seem as simple as building and publishing an aarch64 container for each dev environment.

I got the dotnet devcontainer to work perfectly locally in docker simply by changing
FROM mcr.microsoft.com/dotnet/sdk:5.0-focal
to
FROM mcr.microsoft.com/dotnet/sdk:5.0-buster-slim-arm64v8
in the parent dockerfile.

@jroes
Copy link

jroes commented Mar 27, 2021

I've been hacking around with things to get my m1 mac going and I've had a little success. Here's a recent Dockerfile I put together in case it's helpful to anyone. It's pretty ugly and I'm sure could be done better, but it seems to work okay so far for now!

I imagine this is the path to officially supporting such things here in this repo: https://docs.docker.com/docker-for-mac/multi-arch/

@chrmarti
Copy link
Contributor

@Chuxel Maybe we should introduce an automatic build arg in Remote-Containers for the architecture that can be appended to the image name in FROM to avoid duplication?

@jamesthurley
Copy link

Would the eventual solution be likely to use multi-arch container builds? It feels like the aim should be to open the same repository on either an x86 or ARM machine without (either automatically or manually) modifying any configuration files.

@chrmarti
Copy link
Contributor

@jamesthurley Thanks for pointing this out. That looks like it might be the preferred solution.

@Chuxel
Copy link
Member Author

Chuxel commented Mar 29, 2021

@chrmarti Yeah we'll need some sort of architecture arg because of external feeds and curl'd tools. The build scripts need to be architecture aware. I'd added the idea of an env file in a sample Go implementation along with autodetection in an exploration here. The architecture needs to be translated into the appropriate format for the URI/packaging system which isn't consistent.

@pbrit
Copy link

pbrit commented May 20, 2021

Seriously, I'd love to have arm64 dev containers pre-built.

@njbmartin
Copy link

Recently purchased an M1 mac for dev purposes, and only use dev containers for all of my projects. Unfortunately, it's an expensive paperweight at the moment as the containers are fairly unusable (poor performance, constant disconnects), so I'd love to help test this out. I have the base model M1 Mac Mini.

image

These are the images I use currently:

  • mcr.microsoft.com/vscode/devcontainers/typescript-node:14-buster
  • mcr.microsoft.com/vscode/devcontainers/php:7

@Chuxel
Copy link
Member Author

Chuxel commented Jun 28, 2021

A number of us are working to get our hands on M1 macs but they're quite backlogged - we're hearing months to get our hands on them.

That said, there is a branch that has versions of the scripts in the script-library folder that support arm64 (and in some cases armhf). Right now I'm seeing a seg fault in curl when using buildx in a couple spots that is blocking merge. From there we can get the CI system set up to do automated image generation as described in #914.

@Chuxel
Copy link
Member Author

Chuxel commented Jul 12, 2021

Script updates have been merged and there's a branch that supports the build system updates. However, buildx is reporting a native crash in a simple curl call likely due to QEMU in CI that has no resolution at the moment. I'm seeing some indication of problems with openSSL. That's blocking publishing images.

@Chuxel Chuxel added the blocked label Jul 12, 2021
@finnatsea
Copy link

Anyone have an example of a good enough image to use for container development that is arm native?

@Chuxel
Copy link
Member Author

Chuxel commented Aug 25, 2021

Bullseye is very new, but in concept this problem that has been blocking us should be resolved there. However, many images have not moved their defaults yet. (E.g. The node image is slow to move - it was almost a year before it moved from stretch to buster.)

A hack that unblocks buster but is questionable from a stability standpoint is to upgrade the version of libssl1.1 in buster to the version that is in bullseye as follows:

RUN . /etc/os-release \ 
    && if [ "${VERSION_CODENAME}" = "buster" ]; then \
        apt-get update && apt-get -y install wget \
        && wget https://cloudfront.debian.net/debian/pool/main/o/openssl/libssl1.1_1.1.1k-1_arm64.deb -O /tmp/libssl1.1_1.1.1k-1_arm64.deb \
        && dpkg -i /tmp/libssl1.1_1.1.1k-1_arm64.deb; \
    fi

Update: This issue repros on both physical mac M1 hardware with Docker Desktop and cross-compiling using docker buildx (aka buildkit) on Linux x86_64. Any image based on debian buster exhibits this near as I can tell without adding the above RUN statement as a workaround. This is also reported as a known issue in Docker Desktop release notes for 3.4.0 (which I noticed after getting real M1 hardware and realizing that this was not a buildx problem). You see reports like this in various forms like this one or this one for php which confirms this affects all debian:buster based images in addition to apparently ubuntu:focal and I've even seen reports of centos:8. Combined that's a significant portion of images in the ecosystem.

@Chuxel Chuxel pinned this issue Aug 25, 2021
@Chuxel
Copy link
Member Author

Chuxel commented Aug 27, 2021

The primary reason for this feature request has been M1 Macs. With Docker Desktop not working with Debian buster or Ubuntu focal, it makes little sense for us to spend time trying to get these variations working with pre-built images as they'll suffer problems when you try to use them on Desktop. From what I'm hearing from Docker, it is unlikely that a fix will be back-ported to buster given the small fraction of Debian's user base that would be in this situation. The same is possibly true for Ubuntu focal. Instead we can focus on bullseye based images.

Upstream images that have switched to bullseye by default, should add buster variants for back compat:

  • debian
  • golang
  • php
  • python
  • ruby
  • continuumio/anaconda3
  • continuumio/miniconda3

Default not updated, so add bullseye variant, but pin to buster:

  • node - make bullseye variant visible, control when we switch
  • rust - add bullseye variant, control when we switch
  • openjdk - Default is Oracle Linux but we pin to buster, but bullseye is available.

Uses Ubuntu 20.04/focal:

  • ✋ mcr.microsoft.com/dotnet/sdk - However, C# extension doesn't support ARM64 anyway. Bullseye is the default for 6.0 preview, but not available for other versions.
  • Ubuntu definition - bulidpack-deps:focal-curl <= No known workaround. Could build 21.04 and 18.04 instead.

We will leave the codespaces-linux image as is and stay with x86_64 support since its primary use case is in the cloud (on x86 hardware) and the upstream Oryx image is also x86_64 and switching off of Ubuntu focal would affect quite a bit.

@Chuxel
Copy link
Member Author

Chuxel commented Sep 1, 2021

There are now development builds of the following images with arm64/aarch64 support that can be tested. We will update this list as new image builds are available:

Base images

  • mcr.microsoft.com/vscode/devcontainers/base:dev-bullseye (Debian 11) - Released in v0.194.2
  • mcr.microsoft.com/vscode/devcontainers/base:dev-stretch (Debian 9) - Released in v0.194.2
  • mcr.microsoft.com/vscode/devcontainers/base:dev-hirsute (Ubuntu 21.04) - Released in v0.194.2
  • mcr.microsoft.com/vscode/devcontainers/base:dev-bionic (Ubuntu 18.04) - Released in v0.194.2
  • mcr.microsoft.com/vscode/devcontainers/base:dev-alpine-3.11 (Ubuntu 18.04) - Released in v0.202.2
  • mcr.microsoft.com/vscode/devcontainers/base:dev-alpine-3.12 (Ubuntu 18.04) - Released in v0.202.2
  • mcr.microsoft.com/vscode/devcontainers/base:dev-alpine-3.13 (Ubuntu 18.04) - Released in v0.202.2
  • mcr.microsoft.com/vscode/devcontainers/base:dev-alpine-3.14 (Ubuntu 18.04) - Released in v0.202.2

Node

  • mcr.microsoft.com/vscode/devcontainers/javascript-node:dev-16-bullseye - Released in v0.194.2
  • mcr.microsoft.com/vscode/devcontainers/javascript-node:dev-14-bullseye - Released in v0.194.2
  • mcr.microsoft.com/vscode/devcontainers/javascript-node:dev-12-bullseye - Released in v0.194.2
  • mcr.microsoft.com/vscode/devcontainers/typescript-node:dev-16-bullseye - Released in v0.194.2
  • mcr.microsoft.com/vscode/devcontainers/typescript-node:dev-14-bullseye - Released in v0.194.2
  • mcr.microsoft.com/vscode/devcontainers/typescript-node:dev-12-bullseye - Released in v0.194.2

Python

  • mcr.microsoft.com/vscode/devcontainers/python:dev-3-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/python:dev-3.10-bullseye - Released in v0.202.5
  • mcr.microsoft.com/vscode/devcontainers/python:dev-3.9-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/python:dev-3.8-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/python:dev-3.7-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/python:dev-3.6-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/anaconda:dev-3 - Released in v0.195.0

PHP

  • mcr.microsoft.com/vscode/devcontainers/php:dev-8-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/php:dev-8.0-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/php:dev-7-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/php:dev-7.4-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/php:dev-7.3-bullseye - Released in v0.195.0

.NET

  • mcr.microsoft.com/vscode/devcontainers/dotnet:dev-6.0-bullseye - Released in v0.224.0
  • mcr.microsoft.com/vscode/devcontainers/dotnet:dev-5.0-bullseye - Released in v0.224.0
  • mcr.microsoft.com/vscode/devcontainers/dotnet:dev-3.1-bullseye - Released in v0.224.0

Go

  • mcr.microsoft.com/vscode/devcontainers/go:dev-1-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/go:dev-1.17-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/go:dev-1.16-bullseye - Released in v0.195.0

Ruby, Jekyll

  • mcr.microsoft.com/vscode/devcontainers/ruby:dev-3-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/ruby:dev-3.0-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/ruby:dev-2-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/ruby:dev-2.7-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/ruby:dev-2.6-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/jekyll:dev-bullseye - Released in v0.195.0

Java

  • mcr.microsoft.com/vscode/devcontainers/java:dev-8-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/java:dev-11-bullseye - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/java:dev-17-bullseye - Released in v0.203.0

Rust

  • mcr.microsoft.com/vscode/devcontainers/rust:dev-1-bullseye - Released in v0.195.0

C++

  • mcr.microsoft.com/vscode/devcontainers/cpp:dev-bullseye (Debian 11) - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/cpp:dev-stretch (Debian 9) - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/cpp:dev-hirsute (Ubuntu 21.04) - Released in v0.195.0
  • mcr.microsoft.com/vscode/devcontainers/cpp:dev-bionic (Ubuntu 18.04) - Released in v0.195.0

@fdanielsen
Copy link

There are now development builds of the following images with arm64/aarch64 support that can be tested. We will update this list as new image builds are available:

[snip]

Python

  • mcr.microsoft.com/vscode/devcontainers/javascript-node:dev-3-bullseye
  • mcr.microsoft.com/vscode/devcontainers/javascript-node:dev-3.9-bullseye
  • mcr.microsoft.com/vscode/devcontainers/javascript-node:dev-3.8-bullseye
  • mcr.microsoft.com/vscode/devcontainers/javascript-node:dev-3.7-bullseye
  • mcr.microsoft.com/vscode/devcontainers/javascript-node:dev-3.6-bullseye

Shouldn't these Python images be mcr.microsoft.com/vscode/devcontainers/python:<tag>?

@Chuxel
Copy link
Member Author

Chuxel commented Sep 6, 2021

@fdanielsen Yes, thanks - corrected.

@fdanielsen
Copy link

@fdanielsen Yes, thanks - corrected.

Great, thanks. I assume once these pass development stage all new builds of these devcontainers will be multi arch with arm64 support, eg. the mcr.microsoft.com/vscode/devcontainers/python:3.8 image? We're currently using a devcontainer based on one of the examples, and it uses the mcr.microsoft.com/vscode/devcontainers/python:0-3.8 image. I've tried finding info on what really is the difference between the tags 0-3.8 and simply 3.8, but all I see is some reference to semver. Could you briefly explain or point to an explanation of the real difference in the images?

@Chuxel
Copy link
Member Author

Chuxel commented Sep 7, 2021

@fdanielsen For a given image, type devcontainer-info in a terminal. This will include a link to a history markdown file with exact contents. E.g. one of these for python. The purpose of the prefixes are described in each README:

You can decide how often you want updates by referencing a semantic version of each image. For example:

  • mcr.microsoft.com/vscode/devcontainers/python:0-3.9
  • mcr.microsoft.com/vscode/devcontainers/python:0.202-3.9
  • mcr.microsoft.com/vscode/devcontainers/python:0.202.0-3.9

However, we only do security patching on the latest non-breaking, in support versions of images (e.g. 0-3.9). You may want to run apt-get update && apt-get upgrade in your Dockerfile if you lock to a more specific version to at least pick up OS security updates.

The dev version is a pre-release version indicator. On release it will be come a full version. If you stick with the "Add Development Container Configuration Files" command to add one to your project, you'll be kept on the released path.

@Chuxel
Copy link
Member Author

Chuxel commented Sep 14, 2021

Images listed above should now all be released so you can now omit the dev prefix and use 0- or another sub-version as you see fit. Dropdowns, etc in the Add Dev Container Configuration Files UX will be getting updated incrementally in Remote - Containers and Codespaces to list -bullseye variants along with -buster variants where possible for backwards compatibility.

For images like dotnet that are still x86 only, see here and here for explanation.

Locking this issue so this note remains easily visible to people new to the discussion. Please open up new issues if you run into problems.

UPDATE .NET Bullseye images are available and arm64 support is in the extension, so this is also resolved.

@microsoft microsoft locked as resolved and limited conversation to collaborators Sep 14, 2021
@Chuxel Chuxel added notice and removed blocked help-wanted Extra attention is needed labels Sep 14, 2021
@Chuxel Chuxel closed this as completed Oct 1, 2021
vscode-dev-containers Planning automation moved this from In progress 🏃‍♀️ to Done 🎉 Oct 1, 2021
@Chuxel Chuxel unpinned this issue Aug 19, 2022
@Chuxel Chuxel pinned this issue Nov 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature-request Request for new features or functionality notice
Development

No branches or pull requests