0.60.0 broken on arm64

[hellmund]

Platform Version

0.60.0

What step the error happened?

On deploy

Relevant information

Upgrading from a previous airbyte deployment to 0.60.0 resulted in an unusable installation on arm64 platforms. The root cause appears to be a bug in temporal temporalio/docker-builds#194

Possible workaround might be to link to temporalio/auto-setup:1.23.0 instead of temporalio/auto-setup:1.23 (which currently links to the defective auto-setup:1.23.1 effectively)

Relevant log output

airbyte-temporal                  | ./update-and-start-temporal.sh: line 54: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | ./update-and-start-temporal.sh: line 55: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | Starting to update the temporal DB
airbyte-temporal                  | ./update-and-start-temporal.sh: line 58: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | Update the temporal DB is done
airbyte-temporal                  | Starting to update the temporal visibility DB
airbyte-temporal                  | ./update-and-start-temporal.sh: line 62: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | Update the temporal visibility DB is done
airbyte-temporal                  | starting temporal server
airbyte-temporal                  | Temporal CLI address: 192.168.48.3:7233.
airbyte-temporal                  | ./start-temporal.sh: line 16: /usr/local/bin/temporal-server: cannot execute binary file: Exec format error
airbyte-temporal                  | ./start-temporal.sh: line 16: /usr/local/bin/temporal-server: No error information
airbyte-bootloader                | Unsetting empty environment variable 'LOCAL_CONNECTOR_CATALOG_PATH'
airbyte-bootloader                | Unsetting empty environment variable 'CONNECTOR_REGISTRY_SEED_PROVIDER'
airbyte-bootloader                | Unsetting empty environment variable 'CONNECTOR_REGISTRY_BASE_URL'
airbyte-temporal exited with code 1
airbyte-temporal                  | Done init
airbyte-temporal                  | PostgreSQL started.
airbyte-temporal                  | touch: CONTAINER_ALREADY_STARTED_PLACEHOLDER: Permission denied
airbyte-temporal                  | ./update-and-start-temporal.sh: line 50: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | ./update-and-start-temporal.sh: line 51: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | ./update-and-start-temporal.sh: line 54: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | ./update-and-start-temporal.sh: line 55: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | Starting to update the temporal DB
airbyte-temporal                  | ./update-and-start-temporal.sh: line 58: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | Update the temporal DB is done
airbyte-temporal                  | Starting to update the temporal visibility DB
airbyte-temporal                  | ./update-and-start-temporal.sh: line 62: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | Update the temporal visibility DB is done
airbyte-temporal                  | starting temporal server
airbyte-temporal                  | Temporal CLI address: 192.168.48.3:7233.
airbyte-temporal                  | ./start-temporal.sh: line 16: /usr/local/bin/temporal-server: cannot execute binary file: Exec format error
airbyte-temporal                  | ./start-temporal.sh: line 16: /usr/local/bin/temporal-server: No error information
airbyte-temporal                  | Done init
airbyte-temporal                  | PostgreSQL started.
airbyte-temporal                  | touch: CONTAINER_ALREADY_STARTED_PLACEHOLDER: Permission denied
airbyte-temporal                  | ./update-and-start-temporal.sh: line 50: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | ./update-and-start-temporal.sh: line 51: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | ./update-and-start-temporal.sh: line 54: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | ./update-and-start-temporal.sh: line 55: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | Starting to update the temporal DB
airbyte-temporal                  | ./update-and-start-temporal.sh: line 58: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | Update the temporal DB is done
airbyte-temporal                  | Starting to update the temporal visibility DB
airbyte-temporal                  | ./update-and-start-temporal.sh: line 62: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
airbyte-temporal                  | Update the temporal visibility DB is done
airbyte-temporal                  | starting temporal server
airbyte-temporal                  | Temporal CLI address: 192.168.48.3:7233.
airbyte-temporal                  | ./start-temporal.sh: line 16: /usr/local/bin/temporal-server: cannot execute binary file: Exec format error
airbyte-temporal                  | ./start-temporal.sh: line 16: /usr/local/bin/temporal-server: No error information

[downlz]

I am facing a similar issue on docker running on arm64 ubuntu. Its failing to start container "airbyte/temporal:0.60.0" which is restarting automatically.

Error while accessing localhost:8000

// HttpError
{
  "i18nKey": "errors.http.badGateway",
  "i18nParams": {
    "status": 502
  },
  "name": "HttpError",
  "requestId": "oYiGacade9B9odGSkjbH2y",
  "request": {
    "url": "/api/v1/instance_configuration",
    "method": "get"
  },
  "status": 502,
  "response": "<!DOCTYPE html>\n<html>\n<head>\n<title>Error</title>\n<style>\nhtml { color-scheme: light dark; }\nbody { width: 35em; margin: 0 auto;\nfont-family: Tahoma, Verdana, Arial, sans-serif; }\n</style>\n</head>\n<body>\n<h1>An error occurred.</h1>\n<p>Sorry, the page you are looking for is currently unavailable.<br/>\nPlease try again later.</p>\n<p>If you are the system administrator of this resource then you should check\nthe error log for details.</p>\n<p><em>Faithfully yours, nginx.</em></p>\n</body>\n</html>\n"
}

Error Same as mentioned above, Looks like permission issue.

touch: CONTAINER_ALREADY_STARTED_PLACEHOLDER: Permission denied
PostgreSQL started.
./update-and-start-temporal.sh: line 50: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
./update-and-start-temporal.sh: line 51: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
./update-and-start-temporal.sh: line 54: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
./update-and-start-temporal.sh: line 55: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
Starting to update the temporal DB
./update-and-start-temporal.sh: line 58: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
Update the temporal DB is done
Starting to update the temporal visibility DB
./update-and-start-temporal.sh: line 62: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error
Update the temporal visibility DB is done
starting temporal server
Temporal CLI address: 172.21.0.11:7233.
./start-temporal.sh: line 16: /usr/local/bin/temporal-server: cannot execute binary file: Exec format error
./start-temporal.sh: line 16: /usr/local/bin/temporal-server: No error information

@hellmund how can we apply the proposed workaround as mentioned change to "temporalio/auto-setup:1.23.0"

[ivan-sukhomlyn]

The same situation on the ARM Graviton instances with the error:

/etc/temporal/auto-setup.sh: line 220: /usr/local/bin/temporal-sql-tool: cannot execute binary file: Exec format error

[JamesWhyley]

Was testing this release today and can confirm the same behaviour.

Looks very much as though when they've made a change to automatically select the build architecture they've overlooked the fact that that'll then default to whatever the architecture of the machine doing the build is. In this case, not ARM64. :\

[malikdiarra]

Hi, thanks a lot for reporting this. We are looking into fixing this.

[wennergr]

A new version of our helm chart has been published that depends on temporalio/auto-setup:1.23.0 instead of the non functioning version temporalio/auto-setup:1.23 (1.23.1). This should resolve the issue.

[downlz]

A new version of our helm chart has been published that depends on temporalio/auto-setup:1.23.0 instead of the non functioning version temporalio/auto-setup:1.23 (1.23.1). This should resolve the issue.

I tried removing the old image for airbyte/temporal. And did docker compose up. Still the container is restarting and throwing the same error.

[wennergr]

@downlz Can you check again? The latest helm charts (0.60.1) should use airbyte/temporal:0.60.1

[downlz]

@downlz Can you check again? The latest helm charts (0.60.1) should use airbyte/temporal:0.60.1

This is resolved in 0.60.1 release. Airbyte is deployed successfully but now connections fails with error

at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: io.temporal.failure.ApplicationFailure: message='Unexpected error while getting checking connection.', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:135) at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142) at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:226) at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ... 14 more Caused by: io.temporal.failure.ApplicationFailure: message='/tmp/workspace/ff41d2f8-04b8-4595-9b59-e3557956598c/0', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:191) at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:149) at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ... 18 more Caused by: io.temporal.failure.ApplicationFailure: message='/tmp/workspace/ff41d2f8-04b8-4595-9b59-e3557956598c/0', type='java.nio.file.NoSuchFileException', nonRetryable=false at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:438) at java.base/java.nio.file.Files.createDirectory(Files.java:699) at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:121) ... 20 more

[JamesWhyley]

@downlz that looks similar to the issue mentioned here regarding permissions in the workspace directory.

I've not had a chance to test the v0.60.1 release yet however so can't confirm.

[hellmund]

Thank you @wennergr - in my deployment (with Stripe sources and Redshift destinations), upgrading to v0.60.1 resolved the issue.

[downlz]

@JamesWhyley Thank you so much for supporting with a solution on permissions. I applied the same on airbyte 0.60.1 and now it works smoothly.