Internal Refactoring: Towards a Multi-Distribution Setup

[dominikschubert]

Over the next few weeks and months we’re refactoring the code in this repository to move toward a multi-distribution setup.

For now this only affects active contributors as well as any developers that depend on code existing in the published container under the path /opt/code/localstack/localstack. Most users should not be affected by this change.

Motivation

This will enable us to define clearer boundaries and allows for easier re-use of individual components.
Some of the previously internal code has already been moved into external open repositories such as localstack/rolo.
Other parts of the codebase will keep living in this repository but under its own distribution.
How we will map this to PyPI items is still being discussed and should become clearer over the next weeks during the initial refactorings.

The code layout is not part of any official API or semver guarantees, nevertheless we still want to use this chance to give you a heads up and some guidance in how to make your existing code compatible with the new structure.

Detailed instructions

1. Moving everything into localstack-core

As a first step, the entirety of the localstack module is moved into a localstack-core directory with #10800, which will make up one of the multiple distributions.

In this initial step on our way to a multi-distribution system, only the additional root level of localstack-core is introduced and the rest of the directory structure is unchanged.

• If you have an open PR, you can rebase onto master after move source code into localstack-core #10800 has been merged.

• After the PR is merged, update your local repository with git pull, remove the now empty localstack directory rm -r localstack, and run a make clean install. You should see an localstack_core.egg-info directory in localstack-core/

• If you are an active contributor and you're using the PyCharm IDE, you need to adapt your project structure by marking the new localstack-core module as a source folder. Otherwise you will encounter errors where it will complain about not being able to find the localstack module.

  • 

• If you want to call code from the localstack module, you now need to perform an installation of the project (e.g. with pip install -e .). Previously, since localstack was a root-level module, python automatically included it in its import path. With a source directory layout there is a more strict boundary now which also helps avoiding unintentional imports. See here for more information on the differences between a flat and a src layout.

• The location of test files is unchanged.

• Locally the code moves from .../localstack/localstack/... => .../localstack/localstack-core/localstack/...

• In the published container the code moves from /opt/code/localstack/localstack => /opt/code/localstack/localstack-core/localstack

?. Next steps

After the initial move is over the line, additional code will be extracted from localstack-core into new distributions such as localstack-cli.
This issue will be updated with new information as the project progresses.