INCREMENTAL_BY_UNIQUE_KEY models not taking batch_size
into account
#2609
Labels
Bug
Something isn't working
batch_size
into account
#2609
On Trino/Iceberg (and potentially other engines; unconfirmed), the following model doesnt backfill correctly:
The goal here is to backfill a "daily" model, in batches of 1 day.
However, it does not execute correctly. It triggers a DELETE+INSERT (where the DELETE has
WHERE TRUE
and clears the whole table) for every interval/batch, even though it should only trigger a DELETE+INSERT for the first batch (to support restatements) and then a MERGE for subsequent batches.Killing
sqlmesh plan
before it finishes with ctrl+c and then runningsqlmesh plan
again, which picks up where it left off, triggers the correct MERGE behaviour.It appears that the
batch_size
is not being taken into account correctly when deciding whether to clear the table or merge into the tableThe text was updated successfully, but these errors were encountered: