Unable to upgrade from 1.5.3 to 2.0.0 or 3.1.3

[lloonngg]

Bug description

This issue is related to a prior one that was closed without a resolution Related issue

We are using Rancher Kubernetes 1.20.9. I don't think this is Kubernetes specific.

We were able to rebuild our Docker image including libraries to connect to various databases (pyodbc, dremio, sybase, etc.)

1.3.0 --> 1.5.0 (successful)
1.5.0 --> 1.5.3 (successful)
1.5.0 --> 2.0.0 (failed)
1.5.3 --> 2.0.0 (failed)
1.5.0 --> 3.1.3 (failed)
1.5.3 --> 3.1.3 (failed)

We are using sqlite for the database. When running superset db upgrade, we get this error below in the log. Any workarounds or solution for this?

`INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade ad07e4fdbaba -> a9422eeaae74, new_dataset_models_take_2

Copy 23 physical tables to sl_tables...
Copy 899 SqlaTable to sl_datasets...
Copy dataset owners...
Link physical datasets with tables...
Copy 8,979 table columns to sl_columns...
Link all columns to sl_datasets...
Copy 895 metrics to sl_columns...
Link metric columns to datasets...
Run postprocessing on 9,874 columns
[Column 1 to 9,874] 8,929 may be updated
Assign table column relations...
Run postprocessing on 899 datasets
Process dataset 1~899...
Traceback (most recent call last):
File "/usr/local/bin/superset", line 33, in
sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 357, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask_migrate/cli.py", line 149, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/usr/local/lib/python3.10/site-packages/flask_migrate/init.py", line 98, in wrapped
f(args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask_migrate/init.py", line 185, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 294, in upgrade
script.run_env()
File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 490, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.10/site-packages/alembic/util/compat.py", line 184, in load_module_py
spec.loader.exec_module(module)
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/app/superset/extensions/../migrations/env.py", line 126, in
run_migrations_online()
File "/app/superset/extensions/../migrations/env.py", line 118, in run_migrations_online
context.run_migrations()
File "", line 8, in run_migrations
File "/usr/local/lib/python3.10/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.10/site-packages/alembic/runtime/migration.py", line 561, in run_migrations
step.migration_fn(**kw)
File "/app/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 895, in upgrade
postprocess_datasets(session)
File "/app/superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py", line 594, in postprocess_datasets
table_refrences = extract_table_references(
File "/app/superset/sql_parse.py", line 1063, in extract_table_references
return {
File "/app/superset/sql_parse.py", line 1064, in
Table([part["value"] for part in table["name"][::-1]])
TypeError: Table.init() takes from 2 to 4 positional arguments but 5 were given`

How to reproduce the bug

1. Start from version 1.3.0 or 1.5.0 or 1.5.3
2. Upgrade to 2.0.0 or 3.1.3
3. Run superset db upgrade

Screenshots/recordings

No response

Superset version

3.1.3

Python version

3.10

Node version

I don't know

Browser

Not applicable

Additional context

No response

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

The error you're encountering, "TypeError: Table.init() takes from 2 to 4 positional arguments but 5 were given," during the superset db upgrade process is indicative of a problem with how a Table object is being initialized. This issue likely stems from a version mismatch or an incompatibility between the migration scripts and the version of SQLAlchemy being used by Superset.

To address this issue, you should:

1. Check SQLAlchemy Version: Ensure that the version of SQLAlchemy used by Superset is compatible with the migration scripts. Incompatibilities between versions could lead to unexpected errors during migrations.

2. Review Custom Code: If you have custom code or overrides in your Superset installation, review them to ensure they are not contributing to the issue. This includes any custom migration scripts or modifications to the Superset codebase.

3. Enable Detailed Logging: Turning on detailed logging for the migration process can help pinpoint exactly where the error occurs, providing more context on the erroneous call to Table.__init__().

4. Consult Documentation: Review the latest Apache Superset and SQLAlchemy documentation for any notes on migration issues or breaking changes in the versions you are upgrading to.

It's also worth noting that while SQLite is supported for Superset's metadata database, it's generally recommended for lightweight or development purposes only. For production environments, more robust databases like PostgreSQL or MySQL are recommended due to their better scalability and concurrency support.

Details

Sources

• Error while upgrading superset to 2.0.0 psycopg2.errors.UndefinedColumn
• superset/migrations/versions/2022-04-01_14-38_a9422eeaae74_new_dataset_models_take_2.py
• docs/docs/faq.mdx
• docs/docs/configuration/databases.mdx
• CHANGELOG/2.0.0.md
• RELEASING/release-notes-2-0/changelog.md
• UPDATING.md
• CHANGELOG/3.0.0.md

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.