[Appender] Add AppendDefault

[Tishj]

This feature has been requested many times, so I figured it was time to add it.

We add AppendDefault to the Appender class (not BaseAppender).
This method can be used in place of Append(Value val)

The behavior of AppendDefault is the same as calling INSERT INTO <name> VALUES (DEFAULT) through SQL

[Mytherin]

Thanks for the PR! Great idea. Some comments:

[taniabogatsch]

Hey, thanks for the PR! These changes come in handy!

The main functionality I am missing in this PR is writing a default value directly to a vector in a data chunk. Any performance-focused appender implementation most likely uses duckdb_append_data_chunk instead of appending value-by-value.

From my understanding, in order to write default values to a vector in a data chunk, we need the context of the appender, as we need to know the context of the table. I.e., we cannot add this functionality to the data chunk API. So, maybe it is sensible to extend this PR with a function Appender::AppendDefaultToVector, which takes a vector and a row index and writes the default value to the expected position... 🤔

[Tishj]

So, maybe it is sensible to extend this PR with a function Appender::AppendDefaultToVector, which takes a vector and a row index and writes the default value to the expected position... 🤔

That makes sense, I think to be performant it makes more sense to take a list of integers instead of a single row index.
Since internally we use an ExpressionExecutor, running this expression on multiple rows at once is likely more performant.

[Tishj]

So, maybe it is sensible to extend this PR with a function Appender::AppendDefaultToVector, which takes a vector and a row index and writes the default value to the expected position... 🤔

That makes sense, I think to be performant it makes more sense to take a list of integers instead of a single row index. Since internally we use an ExpressionExecutor, running this expression on multiple rows at once is likely more performant.

@taniabogatsch I've made a commit that starts this, taking a SelectionVector and a count for the amount of defaults that need to be appended (taking the indices from the SelectionVector)

I've had to make some adjustments to the ExpressionExecutor that make me think this is not it's intended purpose (even though it does already take a const SelectionVector *sel)
Please check out this commit first PoC for AppendDefaultToVector

[Giorgi]

Any performance-focused appender implementation most likely uses duckdb_append_data_chunk instead of appending value-by-value.

From my understanding, in order to write default values to a vector in a data chunk, we need the context of the appender, as we need to know the context of the table. I.e., we cannot add this functionality to the data chunk API. So, maybe it is sensible to extend this PR with a function Appender::AppendDefaultToVector, which takes a vector and a row index and writes the default value to the expected position... 🤔

Not only performance-focused, duckdb_append_data_chunk is needed to append more "complex" data types (uuid, decimal, enum, list, etc) so we definitely need a way to append a default value to a vector.

[Tishj]

I didn't feel like the appender was the right place to add methods that query table information.
As such I've created a duckdb_table_description object, which wraps a TableDescription object, which is also present inside the Appender.

This can then be queried for information; duckdb_column_has_default(table, 0)
I imagine this can be extended in the future to include duckdb_column_is_generated, duckdb_column_get_type, duckdb_column_get_name, etc..

[taniabogatsch]

Hi @Tishj, the changes and improvements look nice! 😄 I've added mostly nits and also some comments.

[Giorgi]

@Tishj Will you expose AppendDefaultsToVector in the C API as part of this PR?

[taniabogatsch]

@Giorgi, we plan to expose a way for the appender to write default values when using append_data_chunk in the C API. We'll expose it in a follow-up PR if it is not included in this PR.

[Giorgi]

@Giorgi, we plan to expose a way for the appender to write default values when using append_data_chunk in the C API. We'll expose it in a follow-up PR if it is not included in this PR.

Great! If we don't specify that we want to use default values for some of the columns, will the DuckDB appender automatically use default values for that column? For example, if I have a table with ID auto-increment, will we still need to tell DuckDB that we want to use default for that column?

[taniabogatsch]

Great! If we don't specify that we want to use default values for some of the columns, will the DuckDB appender automatically use default values for that column? For example, if I have a table with ID auto-increment, will we still need to tell DuckDB that we want to use default for that column?

Currently, yes. In this first iteration, it is necessary to specify that a value is a default value (AppendDefault or by using AppendDefaultsToVector). For example, not writing any value (specific or default) to an auto-increment ID will result in an error.

[Mytherin]

Thanks for the changes! I think this PR has too much stuff going on in it at this point - and a few of the things are also not correct and need to be reworked. I propose the following changes to make this PR simpler and get it merged, then we can do follow-up work in subsequent PRs.

[Mytherin]

Thanks!

[Giorgi]

Will append default to vector be implemented in a future PR? The current one isn't very helpful.