Testing speedup

[alovak]

Currently on my machine running all tests takes ~2m:50s. This makes me run tests less frequently or rely on CI (which takes even more time because it runs linters, etc.). Main time consumers are tests that run against MySQL. Specifically: run docker container with MySQL and run migrations. We do this multiple times if we run full test suite. Launching MySQL instance using docker takes ~12-15 seconds.

Here is how we can speedup our tests:

1. Run MySQL database instance locally and use it for all our tests
2. Create database and run migrations for this MySQL instance
3. Use transaction isolated sql driver for our tests to rollback all changes after db connection is closed.

To make this work we should extract db management process into separate command. We need this to setup DB before we run any tests (create, migrate, drop). I also think that this maybe beneficial for users that want to have a control over DB migrations. For example, if migration takes time and it adds only indexes it may be better to run this migration separately.

Questions

1. How many MySQL containers are being launched?

Each time we call CreateTestMySQLDB(t) we run new container. Launching MySQL database inside container takes ~14 seconds. We call CreateTestMySQLDB 11 times.

2. Which specific Customers tests are the slowest? Why?

Tests are not slow. Launching MySQL instance is slow. Running all tests (including SQLite) with one MySQL instance takes < 10 seconds.

3. Can we consolidate tests so we start less MySQL containers?

I don't see how we can make it work without using transactions (or any other db cleaning method). Each test modifies DB by adding records, etc. So if you expect repository to have 1 account then next time test will fail.

Using transactions we can consolidate tests and run one MySQL container per package. But still each package that has DB repo will run own container (+13 seconds to the test suite running time).

4. What would it look like to spin up one instance of MySQL and create random databases for each test that needs it?

It's possible, but I don't see how it's better than using transactions (which are hidden by txdb driver and for developer it looks like Open/Close DB connection). We may benefit from using transactions as it's a familiar way for Python/Django, Ruby/Rails, Buffalo/Go developers that will join Moov :D

5. Should we look at gorm for this project? We're clearly fighting my half-baked setup in this app, so is that the problem?

It may be so. But with a tiny change we can get it now and then continue our work with GORM, etc.

[adamdecaf]

cc @vxio