org.apache.flink.streaming.runtime.operators

Class CheckpointCommitter

java.lang.Object

org.apache.flink.streaming.runtime.operators.CheckpointCommitter

All Implemented Interfaces:

Serializable

public abstract class CheckpointCommitter
extends Object
implements Serializable

This class is used to save information about which sink operator instance has committed checkpoints to a backend.

The current checkpointing mechanism is ill-suited for sinks relying on backends that do not support roll-backs. When dealing with such a system, while trying to get exactly-once semantics, one may neither commit data while creating the snapshot (since another sink instance may fail, leading to a replay on the same data) nor when receiving a checkpoint-complete notification (since a subsequent failure would leave us with no knowledge as to whether data was committed or not).

A CheckpointCommitter can be used to solve the second problem by saving whether an instance committed all data belonging to a checkpoint. This data must be stored in a backend that is persistent across retries (which rules out Flink's state mechanism) and accessible from all machines, like a database or distributed file.

There is no mandate as to how the resource is shared; there may be one resource for all Flink jobs, or one for each job/operator/-instance separately. This implies that the resource must not be cleaned up by the system itself, and as such should kept as small as possible.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected String	jobId
protected static org.slf4j.Logger	LOG
protected String	operatorId

Constructor Summary

Constructors

Constructor and Description
CheckpointCommitter()

Method Summary

Modifier and Type	Method and Description
abstract void	close() Closes the resource/connection to it.
abstract void	commitCheckpoint(int subtaskIdx, long checkpointID) Mark the given checkpoint as completed in the resource.
abstract void	createResource() Creates/opens/connects to the resource that is used to store information.
abstract boolean	isCheckpointCommitted(int subtaskIdx, long checkpointID) Checked the resource whether the given checkpoint was committed completely.
abstract void	open() Opens/connects to the resource, and possibly creates it beforehand.
void	setJobId(String id) Internally used to set the job ID after instantiation.
void	setOperatorId(String id) Internally used to set the operator ID after instantiation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

jobId

protected String jobId

operatorId

protected String operatorId

Constructor Detail

CheckpointCommitter

public CheckpointCommitter()

Method Detail

setJobId

public void setJobId(String id)
              throws Exception

Internally used to set the job ID after instantiation.

Parameters:

id -

Throws:

Exception

setOperatorId

public void setOperatorId(String id)
                   throws Exception

Internally used to set the operator ID after instantiation.

Parameters:

id -

Throws:

Exception

open

public abstract void open()
                   throws Exception

Opens/connects to the resource, and possibly creates it beforehand.

Throws:

Exception

close

public abstract void close()
                    throws Exception

Closes the resource/connection to it. The resource should generally still exist after this call.

Throws:

Exception

createResource

public abstract void createResource()
                             throws Exception

Creates/opens/connects to the resource that is used to store information. Called once directly after instantiation.

Throws:

Exception

commitCheckpoint

public abstract void commitCheckpoint(int subtaskIdx,
                                      long checkpointID)
                               throws Exception

Mark the given checkpoint as completed in the resource.

Parameters:

subtaskIdx - the index of the subtask responsible for committing the checkpoint.

checkpointID - the id of the checkpoint to be committed.

Throws:

Exception

isCheckpointCommitted

public abstract boolean isCheckpointCommitted(int subtaskIdx,
                                              long checkpointID)
                                       throws Exception

Checked the resource whether the given checkpoint was committed completely.

Parameters:

subtaskIdx - the index of the subtask responsible for committing the checkpoint.

checkpointID - the id of the checkpoint we are interested in.

Returns:

true if the checkpoint was committed completely, false otherwise

Throws:

Exception