Key | Default | Type | Description
--- | --- | --- | ---

##### state.changelog.dstl.dfs.base-path

(none)

String

Base path to store changelog files.

##### state.changelog.dstl.dfs.batch.persist-delay

10 ms

Duration

Delay before persisting changelog after receiving persist request (on checkpoint). Minimizes the number of files and requests if multiple operators (backends) or sub-tasks are using the same store. Correspondingly increases checkpoint time (async phase).

##### state.changelog.dstl.dfs.batch.persist-size-threshold

10 mb

MemorySize

Size threshold for state changes that were requested to be persisted but are waiting for state.changelog.dstl.dfs.batch.persist-delay (from all operators). . Once reached, accumulated changes are persisted immediately. This is different from state.changelog.dstl.dfs.preemptive-persist-threshold as it happens AFTER the checkpoint and potentially for state changes of multiple operators. Must not exceed in-flight data limit (see below)

##### state.changelog.dstl.dfs.compression.enabled

false

Boolean

Whether to enable compression when serializing changelog.

##### state.changelog.dstl.dfs.discard.num-threads

1

Integer

Number of threads to use to discard changelog (e.g. pre-emptively uploaded unused state).

##### state.changelog.dstl.dfs.download.local-cache.idle-timeout-ms

10 min

Duration

Maximum idle time for cache files of distributed changelog file, after which the cache files will be deleted.

##### state.changelog.dstl.dfs.preemptive-persist-threshold

5 mb

MemorySize

Size threshold for state changes of a single operator beyond which they are persisted pre-emptively without waiting for a checkpoint. Improves checkpointing time by allowing quasi-continuous uploading of state changes (as opposed to uploading all accumulated changes on checkpoint).

##### state.changelog.dstl.dfs.upload.buffer-size

1 mb

MemorySize

Buffer size used when uploading change sets

##### state.changelog.dstl.dfs.upload.max-attempts

3

Integer

Maximum number of attempts (including the initial one) to perform a particular upload. Only takes effect if state.changelog.dstl.dfs.upload.retry-policy is fixed.

##### state.changelog.dstl.dfs.upload.max-in-flight

100 mb

MemorySize

Max amount of data allowed to be in-flight. Upon reaching this limit the task will be back-pressured. I.e., snapshotting will block; normal processing will block if state.changelog.dstl.dfs.preemptive-persist-threshold is set and reached. The limit is applied to the total size of in-flight changes if multiple operators/backends are using the same changelog storage. Must be greater than or equal to state.changelog.dstl.dfs.batch.persist-size-threshold

##### state.changelog.dstl.dfs.upload.next-attempt-delay

500 ms

Duration

Delay before the next attempt (if the failure was not caused by a timeout).

##### state.changelog.dstl.dfs.upload.num-threads

5

Integer

Number of threads to use for upload.

##### state.changelog.dstl.dfs.upload.retry-policy

"fixed"

String

Retry policy for the failed uploads (in particular, timed out). Valid values: none, fixed.

##### state.changelog.dstl.dfs.upload.timeout

1 s

Duration

Time threshold beyond which an upload is considered timed out. If a new attempt is made but this upload succeeds earlier then this upload result will be used. May improve upload times if tail latencies of upload requests are significantly high. Only takes effect if state.changelog.dstl.dfs.upload.retry-policy is fixed. Please note that timeout * max_attempts should be less than execution.checkpointing.timeout