dlt.extract.incremental.transform
IncrementalTransform Objects
class IncrementalTransform()
A base class for handling extraction and stateful tracking of incremental data from input data items.
By default, the descendant classes are instantiated within the
dlt.extract.incremental.Incremental class.
Subclasses must implement the __call__ method which will be called
for each data item in the extracted data.
compute_deduplication_disabled
def compute_deduplication_disabled() -> bool
Skip deduplication when length of the key is 0 or if lag is applied.
JsonIncremental Objects
class JsonIncremental(IncrementalTransform)
Extracts incremental data from JSON data items.
find_cursor_value
def find_cursor_value(row: TDataItem) -> Any
Finds value in row at cursor defined by self.cursor_path.
Will use compiled JSONPath if present. Otherwise, reverts to field access if row is dict, Pydantic model, or of other class.
__call__
def __call__(row: TDataItem) -> Tuple[Optional[TDataItem], bool, bool]
Returns:
Tuple (row, start_out_of_range, end_out_of_range) where row is either the data item or None if it is completely filtered out
ModelIncremental Objects
class ModelIncremental(IncrementalTransform)
Incremental transform for Relation items.
Filtering happens via SQL pushdown when Relation.incremental(cursor) is
applied.
Modes:
end_valueis set: external scheduler/ephemeral: no aggregate, state is not advanced from observed data.range_start="open", noend_value: stateful open-range: aggregate runs andlast_valueadvances. Open range on the next run excludes the boundary, so no deduplication is required.- Otherwise (closed-range stateful): rejected as boundary deduplication via
unique_hashescannot be reproduced from a single aggregate.