org.apache.flume.EventDeliveryException: Failed to open file


Error Scenario:

org.apache.flume.EventDeliveryException: Failed to open file 

We will receive this error message when a flume agent is started and it is trying to start FILE_ROLL sink with a given target sink directory. Below are the error messages sequence from ~/logs/flume.log file.

Root Cause:

When a flume agent is started with FILE_ROLL sink type and without creating <sink.directory> folder prior to starting agent or 

Flume user doesn’t have enough permissions (write access) to <sink.directory>  to create files under it, then we will receive this error message. 

In our above error scenario,  the directory /home/hadoop/flume/files is not created prior to the start of agent Agent2. So, we have received the above error message.

If we didn’t have the destination directory already created but flume user has write access to create the folder, even then we will receive the above error message because Flume JVM process will not create the <sink.directory> on the fly while running Flume agent.

Resolution:

Parent directory given in AgentX.sinks.XXXXX.sink.directory property should already be created before starting flume agent and the flume user should have write access to it.

Even if the flume user has access to create files and parent directory is not created prior to starting agent,flume process will not create the directory on the fly.


Profile photo of Siva

About Siva

Senior Hadoop developer with 4 years of experience in designing and architecture solutions for the Big Data domain and has been involved with several complex engagements. Technical strengths include Hadoop, YARN, Mapreduce, Hive, Sqoop, Flume, Pig, HBase, Phoenix, Oozie, Falcon, Kafka, Storm, Spark, MySQL and Java.

Leave a comment

Your email address will not be published. Required fields are marked *