Dev container improvements

[jonah-iden]

What it does

This adds 4 new supported properties of the devcontainer JSON.

• remoteUser which sets the user theia is started with in the container
• postCreateCommand a command executed after creating and initally starting the container
• extensions to set extensions installed when creating the container
• settings preferences to set when creating the container

Also adds a new command line option to theia which sets preferences on startup.

How to test

Here is a devcontainer.json and a Dockerfile for testing.
devcontainer.json
Dockerfile

Put both of these in a .devconainer folder and run Dev Container: Reopen in Container

When creating the container in the output tab you should see the output of the postCreateCommand at the end

In the opened Container the following properties should be set:

• the user (for example when opening a terminal and executing bin/bash) should be node
• the extensions defined in devcontainer.json should be installed.
• the html.format.templating property should be set to true

Follow-

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[planger]

Thank you very much for this great evolution of the devcontainer support!

I tested this change with the provided Dockerfile and devcontainer.json and observed two issues:

1. The environment in the container wasn't set correctly; that is the line ENV PATH="/go/bin:${PATH}" didn't seem to have an impact so that go wasn't available on the path.
2. The port forwarding didn't seem to work. Even though the devcontainer.json specifies a forwardPorts, this port didn't show up in the Ports view. If I try to manually add it, I get an EADDRINUSE error in the console. Before and after I tried adding it to the Port view, the port wasn't reachable from the host.

Maybe this is also related to something local for me, but I'm not sure. Do you have an idea?

Thanks again!

[planger]

Suggested change

	protected readonly CliPreferences: CliPreferences;
	protected readonly cliPreferences: CliPreferences;

[planger]

Suggested change

	this.CliPreferences.getPreferences().then(async preferences => {
	this.cliPreferences.getPreferences().then(async preferences => {

[jonah-iden]

Thats weird both go and the forwarded port worked fine for me locally. Did you by any chance have a service allready running on 1313 thats the only thing i could explain this with.
I'll take another look tomorrow maybe i can reproduce this

[planger]

Thanks for the fast response! I'll double-check at latest tomorrow and report back. I think I made sure to kill and prune all containers in order to test the setup. But I'll check.

[planger]

I can confirm that remoteUser and extensions work fine with this change. This is awesome!

Unfortunately, however, I don't really have better news after retrying from scratch regarding the ENV and forwardPorts.
I ran the following:

• docker stop $(docker ps -q) to stop all containers
• docker system prune -a --volumes to ensure all containers are rebuilt from scratch
• ss -tulnp to check that 1313 is not used anywhere yet
• Started Theia and re-opened folder in devcontainer
• Now ss -tulnp shows 0.0.0.0:1313 (local) / 0.0.0.0:* (peer) and [::]:1313 / [::]:*
• The Ports view however still shows no forwarded ports (which it would in VS Code)
• Opening a Terminal in Theia and typing go fails (because it isn't on the path), so does hugo server
• Running export PATH="/go/bin:${PATH}" in the terminal fixes that tough
• Now hugo server works and is listening on port 1313
• Opening localhost:1313 fails though (ERR_CONNECTION_RESET)
• Trying to add a port in the Port view manually fails with EADDRINUSE (because the container already uses that port, I guess)

When I close Theia, 1313 remains blocked and docker ps still shows the container, which it shouldn't as I've closed Theia.

Comparing to VS Code, I see that ss -tulnp shows 127.0.0.1:1313 / 0.0.0.0:* (VS Code) instead of 0.0.0.0:1313 / 0.0.0.0:* (Theia). Also Theia has another entry [::]:1313 / [::]:*, which isn't there at all in VS Code.

I didn't debug into all of that in more detail, but in summary I have the following observations:

• Something seems to be different with respect to the port setup as mentioned above (maybe this is the reason for the port not being reachable from the host and the port not showing up in the Port view)
• The terminal in the containerized Theia backend doesn't seem to respect the ENV command in the Dockerfile for me
• The container keeps running after I close Theia

I hope this helps! If it doesn't, please let me know. I'm happy to try provide more info or if you like even jump on a short screenshare.

Thank you for all your work on this!

[jonah-iden]

thanks for this in depth analysis. I'll take a look today

[jonah-iden]

@planger Small update: I am able to reproduce the problem with the PATH. It seems it works fine when using a docker exec instance (for example when using the docker desktop exec tab) but not when opening a terminal through theia. Still no idea why though yet. But seems it has something to do with the temrinal library theia is using. Launching an application from theia also has the correct path

Regarding the port forwarding it works fine for me. But by default there is no service running in my container listening on port 1313 so i had to start one. Also from the Dockerfile it seems there is no service started

[planger]

@jonah-iden Thanks for the update and great you have a lead for the path issue!

Regarding the port: Yeah the Dockerfile / devcontainer doesn't start a process on the port automatically, but only if you run hugo server (or the corresponding task from the tasks.json).
In VS Code the behavior is that it already shows the port in the port view (and seems to reserve it on the host), even though nothing is reachable on that port in the container. Once there is a process listening to the port in the container, it starts actively forwarding it to the host. I'm not sure if this can be done statically, or if VS Code actually listens in the container for a process coming up on that port and the re-initializes the port forwarding.

[jonah-iden]

I Found the issue with the PATH environement. It seems the /etc/profile file was overwriting the PATH and thus removing the docker changes. VSCode seems to be modifiying the profile file so that the path is not overwritten.
Will push a fix soon

[planger]

Thank you very much @jonah-iden for the great work! I can confirm that the latest changes fix the following issues

1. go is now correctly on the PATH and the build task works perfectly fine! 💯
2. the container is correctly stopped when I close Theia 🥳
3. the ports now show up in the ports view 👁️

I still have the issue that the port is not available on the host. As you say this is working for you, I think it is worth pulling in another test to ensure it is not something specific to me.

@JonasHelming Can you please test if it works for you to open a container based on the test devcontainer file and run hugo server (with a hugo project in the workspace) and see if you can access port 1313 from your host's browser?

Thank you!

[jonah-iden]

the problem may be, that currently the forwarded ports are forwarded via the docker container. VSCode is using their own dynamic port forwarding mechanism for this. Could it be the hugo server is running on localhost and not 0.0.0.0. Maybe that could explain it?

[planger]

the problem may be, that currently the forwarded ports are forwarded via the docker container. VSCode is using their own dynamic port forwarding mechanism for this. Could it be the hugo server is running on localhost and not 0.0.0.0. Maybe that could explain it?

Excellent hint! That is indeed the reason. By default hugo server binds to 127.0.0.1. If I explicitly bind to 0.0.0.0 (hugo server --bind=0.0.0.0), then it works also in Theia. VS Code indeed seems to do some dynamic forwarding there. Thanks for pointing this out!

Do you think we should also look into replicating this behavior of VS Code or is it too much effort?

[jonah-iden]

The dynamic forwarding feature allready exists for theia, so it shouldn't be to difficult to change this behaviour.
Should be at maximum a day of work i would guess.
I think it would be good to mimic vscodes bahaviour here. Otherwise others could have similar problems to yours where it works in vscode but not theia

[planger]

Right, I agree. For compatibility reasons, I think it makes sense to also support that in Theia. Thank you very much!

[jonah-iden]

forwardPorts should now use the dynamic port forwarding mechanism

[planger]

Thank you very much! The dynamic port forwarding works like a charm. This PR is an excellent addition to the dev-container support. Thank you!