Flume to Phoenix as Sink


Hadoop Eco System Forums Hadoop Discussion Forum Flume to Phoenix as Sink

This topic contains 2 replies, has 2 voices, and was last updated by Profile photo of Mahesh Mahesh 4 years, 9 months ago.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #2629 Reply
    Profile photo of Mahesh
    Mahesh
    Participant

    Hi,

    AS we are working in flume to store apache log into phoenix using phoenix-plugin & access log properties configuration .we could run flume and see the events get process,when quering hbase through phoenix there is no record stored.Can you provide tutorial for flume-phoenix connection.

    14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
    14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14a5b888d18003e
    14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003e closed
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down
    14/12/18 12:53:09 INFO query.ConnectionQueryServicesImpl: Found quorum: nn01:2181
    14/12/18 12:53:09 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x843735 connecting to ZooKeeper ensemble=nn01:2181
    14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=nn01:2181 sessionTimeout=180000 watcher=hconnection-0x843735, quorum=nn01:2181, baseZNode=/hbase
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Opening socket connection to server nn01/10.10.10.25:2181. Will not attempt to authenticate using SASL (unknown error)
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Socket connection established to nn01/10.10.10.25:2181, initiating session
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Session establishment complete on server nn01/10.10.10.25:2181, sessionid = 0x14a5b888d18003f, negotiated timeout = 40000
    14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
    14/12/18 12:53:09 INFO client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14a5b888d18003f
    14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003f closed
    14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down
    14/12/18 12:53:15 INFO serializer.BaseEventSerializer: the upsert statement is UPSERT INTO APACHE_LOG4 (“HOST”, “IDENTITY”, “USER”, “TIME”, “METHOD”, “REQUEST”, “PROTOCOL”, “STATUS”, “SIZE”, “REFERER”, “AGENT”, “F_HOST”, “UID”) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    14/12/18 12:53:18 INFO sink.PhoenixSink: Time taken to process [10] events was [3] seconds
    14/12/18 12:53:22 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:27 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:33 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:40 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:48 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:53:56 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:54:04 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:54:12 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:54:20 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds
    14/12/18 12:54:28 INFO sink.PhoenixSink: Time taken to process [1] events was [3] seconds
    14/12/18 12:54:36 INFO sink.PhoenixSink: Time taken to process [0] events was [3] seconds

    Thanks
    Mahesh.S

    #2634 Reply
    Profile photo of Siva
    Siva
    Keymaster

    Hi Mahesh

    There is some issue with the BaseEventSerializer in the flume sink, so it is not able to parse the records properly and values are updated in the Phoenix/Hbase Table. Even I tried it multiple ways but facing the same issue again and again. So lets wait for some more time so that Phoenix will release stable version of flume plugin.

    Please let everybody know if this issue is solved to you by any chance.

    Regards,
    Siva

    #2645 Reply
    Profile photo of Mahesh
    Mahesh
    Participant

    Hi Siva,
    Thanks for your valuable reply.Lets wait for next stable version.

    Thanks,
    Mahesh

Viewing 3 posts - 1 through 3 (of 3 total)
Reply To: Flume to Phoenix as Sink
Your information: