-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Storage] Add storage_compatibility_version
to control for what version the DB has to be serialized.
#12110
Conversation
…alue has a 'default', and is the last property?
… break when serialization is introduced
… property should be serialized or not
…t and the storage_info.cpp file
…g and the comparison
…he future we will remove properties we previously serialized?
…' to serialize everything
…LTERNATIVE_VERIFY is set
…unreleased version number to 'latest'
I think this is a cool feature to have, but I worry a bit of the ramifications of this, in particular that using this setting will allow user to lose data / constraint / information and produce a DB file that will not keep track of the fact that this has been produced while being instructed to skip some serialization. In particular, how can you tell given a db file on your machine whether it's the 'complete' DB version or some limited version of it? One proposal that could handle this goals could be having this setting be at the attached database level, and with this information serialized to the header of the Database file. This achieve a few things: no impact on the running database (I feel strange that |
Having the target serialization version attached to a database file makes sense to me, but can likely be added later. (1) we still need a default version in case none is specified, this setting can serve as that default setting, and (2) we might also need this for serialization of e.g. logical plans to e.g. JSON, which does not involve an attached database. |
Thanks! |
Merge pull request duckdb/duckdb#12110 from Tishj/serialized_dependencies Merge pull request duckdb/duckdb#12153 from carlopi/pyodide_in_nightly
The idea behind the storage compatibility version is that we want to be able to add serialization, without losing the ability for forwards compatibility.
What inspired this functionality (and is introduces by this PR as well) is the ability to serialize CatalogEntry dependencies.
Because we won't rebind entries on load of a database anymore (think indexes, default values, views etc..), we also won't rediscover dependencies, as that is done during binding.
We do not serialize them by default currently.
Using `storage_compatibility_version='v0.10.3' you can opt-in to this, it will be the default in a future release.
storage_compatibility_version
This is a new setting on the DBConfig, it takes a VARCHAR value in the form of:
v<major>.<minor>.<patch>
orlatest
The string internally gets mapped to a serialization version, which is checked at serialization time.
The
generate_serialization.py
was updated to conditionally serialize items that have been marked with aversion
field based on this setting.