Package org.apache.nifi.processor.util.file.transfer

Class FetchFileTransfer

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processor.util.file.transfer.FetchFileTransfer

All Implemented Interfaces:

ConfigurableComponent, Processor

public abstract class FetchFileTransfer
extends AbstractProcessor

A base class for FetchSFTP, FetchFTP processors. Note that implementations of this class should never use the @SupportsBatching annotation! Doing so could result in data loss!

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	FetchFileTransfer.FileTransferIdleWrapper	Wrapper around a FileTransfer object that is used to know when the FileTransfer was last used, so that we have the ability to close connections that are "idle," or unused for some period of time.

Field Summary

Fields

Modifier and Type	Field	Description
static final AllowableValue	COMPLETION_DELETE
static final AllowableValue	COMPLETION_MOVE
static final AllowableValue	COMPLETION_NONE
static final PropertyDescriptor	COMPLETION_STRATEGY
static final String	FAILURE_REASON_ATTRIBUTE
static final PropertyDescriptor	FILE_NOT_FOUND_LOG_LEVEL
private final Map<Tuple<String,Integer>,BlockingQueue<FetchFileTransfer.FileTransferIdleWrapper>>	fileTransferMap
static final PropertyDescriptor	HOSTNAME
private final long	IDLE_CONNECTION_MILLIS
private long	lastClearTime
private LogLevel	levelFileNotFound
static final PropertyDescriptor	MOVE_CREATE_DIRECTORY
static final PropertyDescriptor	MOVE_DESTINATION_DIR
static final Relationship	REL_COMMS_FAILURE
static final Relationship	REL_NOT_FOUND
static final Relationship	REL_PERMISSION_DENIED
static final Relationship	REL_SUCCESS
static final PropertyDescriptor	REMOTE_FILENAME
static final PropertyDescriptor	UNDEFAULTED_PORT
static final PropertyDescriptor	USERNAME

Constructor Summary

Constructors

Constructor	Description
FetchFileTransfer()

Method Summary

Modifier and Type	Method	Description
void	cleanup()
private void	cleanupTransfer(FileTransfer transfer, boolean closeConnection, BlockingQueue<FetchFileTransfer.FileTransferIdleWrapper> transferQueue, String host, int port)
private void	closeConnections(boolean closeNonIdleConnections)	Close connections that are idle or optionally close all connections.
protected abstract FileTransfer	createFileTransfer(ProcessContext context)	Creates a new instance of a FileTransfer that can be used to pull files from a remote system.
Set<Relationship>	getRelationships()
protected List<PropertyDescriptor>	getSupportedPropertyDescriptors()
void	onScheduled(ProcessContext context)
void	onTrigger(ProcessContext context, ProcessSession session)
private void	performCompletionStrategy(FileTransfer transfer, ProcessContext context, FlowFile flowFile, String filename, String host, int port)

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

customValidate, equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

Methods inherited from interface org.apache.nifi.processor.Processor

isStateful, migrateProperties, migrateRelationships

Field Details

COMPLETION_NONE

public static final AllowableValue COMPLETION_NONE

COMPLETION_MOVE

public static final AllowableValue COMPLETION_MOVE

COMPLETION_DELETE

public static final AllowableValue COMPLETION_DELETE

FAILURE_REASON_ATTRIBUTE

public static final String FAILURE_REASON_ATTRIBUTE

See Also:

• Constant Field Values

HOSTNAME

public static final PropertyDescriptor HOSTNAME

UNDEFAULTED_PORT

public static final PropertyDescriptor UNDEFAULTED_PORT

USERNAME

public static final PropertyDescriptor USERNAME

REMOTE_FILENAME

public static final PropertyDescriptor REMOTE_FILENAME

COMPLETION_STRATEGY

public static final PropertyDescriptor COMPLETION_STRATEGY

MOVE_CREATE_DIRECTORY

public static final PropertyDescriptor MOVE_CREATE_DIRECTORY

MOVE_DESTINATION_DIR

public static final PropertyDescriptor MOVE_DESTINATION_DIR

FILE_NOT_FOUND_LOG_LEVEL

public static final PropertyDescriptor FILE_NOT_FOUND_LOG_LEVEL

REL_SUCCESS

public static final Relationship REL_SUCCESS

REL_COMMS_FAILURE

public static final Relationship REL_COMMS_FAILURE

REL_NOT_FOUND

public static final Relationship REL_NOT_FOUND

REL_PERMISSION_DENIED

public static final Relationship REL_PERMISSION_DENIED

fileTransferMap

private final Map<Tuple<String,Integer>,BlockingQueue<FetchFileTransfer.FileTransferIdleWrapper>> fileTransferMap

IDLE_CONNECTION_MILLIS

private final long IDLE_CONNECTION_MILLIS

lastClearTime

private volatile long lastClearTime

levelFileNotFound

private LogLevel levelFileNotFound

Constructor Details

FetchFileTransfer

public FetchFileTransfer()

Method Details

getRelationships

public Set<Relationship> getRelationships()

Specified by:

getRelationships in interface Processor

Overrides:

getRelationships in class AbstractSessionFactoryProcessor

onScheduled

@OnScheduled
public void onScheduled(ProcessContext context)

closeConnections

private void closeConnections(boolean closeNonIdleConnections)

Close connections that are idle or optionally close all connections. Connections are considered "idle" if they have not been used in 10 seconds.

Parameters:

closeNonIdleConnections - if true will close all connection; if false will close only idle connections

cleanup

@OnStopped
public void cleanup()

getSupportedPropertyDescriptors

protected List<PropertyDescriptor> getSupportedPropertyDescriptors()

Overrides:

getSupportedPropertyDescriptors in class AbstractConfigurableComponent

onTrigger

public void onTrigger(ProcessContext context,
 ProcessSession session)
               throws ProcessException

Specified by:

onTrigger in class AbstractProcessor

Throws:

ProcessException

cleanupTransfer

private void cleanupTransfer(FileTransfer transfer,
 boolean closeConnection,
 BlockingQueue<FetchFileTransfer.FileTransferIdleWrapper> transferQueue,
 String host,
 int port)

performCompletionStrategy

private void performCompletionStrategy(FileTransfer transfer,
 ProcessContext context,
 FlowFile flowFile,
 String filename,
 String host,
 int port)

createFileTransfer

protected abstract FileTransfer createFileTransfer(ProcessContext context)

Creates a new instance of a FileTransfer that can be used to pull files from a remote system.

Parameters:

context - the ProcessContext to use in order to obtain configured properties

Returns:

a FileTransfer that can be used to pull files from a remote system