org.apache.flink.streaming.runtime.io.recovery

Class RescalingStreamTaskNetworkInput<T>

java.lang.Object

org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput<T,org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>

org.apache.flink.streaming.runtime.io.recovery.RescalingStreamTaskNetworkInput<T>

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.flink.runtime.io.AvailabilityProvider, PushingAsyncDataInput<T>, RecoverableStreamTaskInput<T>, StreamTaskInput<T>

@Internal
public final class RescalingStreamTaskNetworkInput<T>
extends AbstractStreamTaskNetworkInput<T,org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>
implements RecoverableStreamTaskInput<T>

A StreamTaskNetworkInput implementation that demultiplexes virtual channels.

The demultiplexing works in two dimensions for the following cases. *

• Subtasks of the current operator have been collapsed in a round-robin fashion.
• The connected output operator has been rescaled (up and down!) and there is an overlap of channels (mostly relevant to keyed exchanges).

In both cases, records from multiple old channels are received over one new physical channel, which need to demultiplex the record to correctly restore spanning records (similar to how StreamTaskNetworkInput works).

Note that when both cases occur at the same time (downscaling of several operators), there is the cross product of channels. So if two subtasks are collapsed and two channels overlap from the output side, there is a total of 4 virtual channels.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.runtime.io.PushingAsyncDataInput

PushingAsyncDataInput.DataOutput<T>

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

org.apache.flink.runtime.io.AvailabilityProvider.AvailabilityHelper

Field Summary

Fields inherited from class org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput

checkpointedInputGate, deserializationDelegate, flattenedChannelIndices, inputIndex, inputSerializer, recordDeserializers, statusWatermarkValve

Fields inherited from interface org.apache.flink.streaming.runtime.io.StreamTaskInput

UNSPECIFIED

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description

RescalingStreamTaskNetworkInput(CheckpointedInputGate checkpointedInputGate, org.apache.flink.api.common.typeutils.TypeSerializer<T> inputSerializer, org.apache.flink.runtime.io.disk.iomanager.IOManager ioManager, StatusWatermarkValve statusWatermarkValve, int inputIndex, org.apache.flink.runtime.checkpoint.InflightDataRescalingDescriptor inflightDataRescalingDescriptor, java.util.function.Function<Integer,StreamPartitioner<?>> gatePartitioners, org.apache.flink.api.common.TaskInfo taskInfo)

Method Summary

Modifier and Type	Method and Description
StreamTaskInput<T>	finishRecovery()
protected org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>	getActiveSerializer(org.apache.flink.runtime.checkpoint.channel.InputChannelInfo channelInfo)
CompletableFuture<Void>	prepareSnapshot(org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter channelStateWriter, long checkpointId) Prepares to spill the in-flight input buffers as checkpoint snapshot.
protected org.apache.flink.core.io.InputStatus	processEvent(org.apache.flink.runtime.io.network.partition.consumer.BufferOrEvent bufferOrEvent)

Methods inherited from class org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput

close, emitNext, getAvailableFuture, getInputIndex, processBuffer, releaseDeserializer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.runtime.io.StreamTaskInput

getInputIndex

Methods inherited from interface org.apache.flink.streaming.runtime.io.PushingAsyncDataInput

emitNext

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

and, getAvailableFuture, isApproximatelyAvailable, isAvailable, or

Methods inherited from interface java.io.Closeable

close

Constructor Detail

RescalingStreamTaskNetworkInput

public RescalingStreamTaskNetworkInput(CheckpointedInputGate checkpointedInputGate,
                                       org.apache.flink.api.common.typeutils.TypeSerializer<T> inputSerializer,
                                       org.apache.flink.runtime.io.disk.iomanager.IOManager ioManager,
                                       StatusWatermarkValve statusWatermarkValve,
                                       int inputIndex,
                                       org.apache.flink.runtime.checkpoint.InflightDataRescalingDescriptor inflightDataRescalingDescriptor,
                                       java.util.function.Function<Integer,StreamPartitioner<?>> gatePartitioners,
                                       org.apache.flink.api.common.TaskInfo taskInfo)

Method Detail

finishRecovery

public StreamTaskInput<T> finishRecovery()
                                  throws IOException

Specified by:

finishRecovery in interface RecoverableStreamTaskInput<T>

Throws:

IOException

getActiveSerializer

protected org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T> getActiveSerializer(org.apache.flink.runtime.checkpoint.channel.InputChannelInfo channelInfo)

Overrides:

getActiveSerializer in class AbstractStreamTaskNetworkInput<T,org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>

processEvent

protected org.apache.flink.core.io.InputStatus processEvent(org.apache.flink.runtime.io.network.partition.consumer.BufferOrEvent bufferOrEvent)

Overrides:

processEvent in class AbstractStreamTaskNetworkInput<T,org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>

prepareSnapshot

public CompletableFuture<Void> prepareSnapshot(org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter channelStateWriter,
                                               long checkpointId)
                                        throws org.apache.flink.runtime.checkpoint.CheckpointException

Description copied from interface: StreamTaskInput

Prepares to spill the in-flight input buffers as checkpoint snapshot.

Specified by:

prepareSnapshot in interface StreamTaskInput<T>

Throws:

org.apache.flink.runtime.checkpoint.CheckpointException