Monthly Archives: October 2014

Hive UDF usage — In this post we will describe about the process of creating custom UDF in Hive. Though there are many generic UDFs (User defined functions) provided by Hive we might need to write our custom UDFs sometime to meet our requirements. In this post, we will discuss about one of the […]

Creating Custom UDF in Hive – Auto Increment Column in ...

30 Oct, 2014 in Hive tagged Auto Increment Column UDF in Hive / Changes in hive-site.xml or hive-env.sh file / Hadoop Hive UDF Tutorial / hive udf example java / hive udf type stateful / hive.aux.jars.path property in hive-site.xml file / how to add a permanent function in hive / how to add auto increment column in a table using hive / How to create UDFs in Hive / how to use udfs in hive example / How to write a custom udfs in Hive hadoop / udf example in hive / Updating HIVE_AUX_JARS_PATH in hive-env.sh file / Using a custom UDF in hive / writing udfs in hive example by Siva

Healthy Cluster — In this post, we will discuss about hadoop installation on cloud storage. Though there are number of posts available across internet on this topic, we are documenting the procedure for Cloudera Manager Installation on Amazon EC2 instances with some of our practical views on installation and tips and hints to […]

Cloudera Manager Installation on Amazon EC2

27 Oct, 2014 in Hadoop tagged cdh on aws / cdh on ec2 / cdh parcels install on ec2 / cloud hadoop cloudera / cloud hadoop cloudera install guide / cloud hadoop cluster / cloud hadoop tutorial / cloudera hadoop installation on amazon ec2 / Cloudera Hadoop multi-node cluster setup on Amazon EC2 instances / Cloudera hadoop setup on Amazon AWS cloud services / cloudera manager installation guide / cloudera manager installation on amazon ec2 / cloudera manager installation on amazon vpc / cloudera manager installation on ubuntu / cloudera manager installation steps / cloudera on aws / cloudera on ec2 / cloudera setup hadoop cluster on cloud storage / hadoop amazon cloud / hadoop cloud computing / hadoop installation on amazon ec2 / Hadoop Installation on Cloud Storage / hadoop on cloud free / hadoop on cloudera / hadoop on cloudstack / hive metastore error in cloudera manager / install hadoop on amazon ec2 / install hadoop on cloud / postgresql cloudera / reverse dns look failed while cloudera manager installation on cloud / running hadoop on aws by Siva

Below are a few important Hadoop HBase Interview Questions and Answers that are suitable for hadoop freshers or experienced developers. 1. What is HBase? HBase is Column-Oriented , Open-Source, Multidimensional, Distributed database. It run on the top of HDFS. 2. Why do we use HBase? HBase provide random read and […]

HBase Interview Questions and Answers Part – 1

26 Oct, 2014 in HBase Interview Questions for experienced and freshers tagged bigdata and hadoop hbase interview questions and answers for freshers / difference between hbase and rdbms / hadoop hbase interview questions and answers / hbase feature list / hbase vs hdfs / hbase vs rdbms example / nosql vs sql databases / what are the catalog tables in hbase / when do we use hbase by Siva

Hunk Visualization — In this post we will discuss about the configuration required for Hive connectivity with Hunk, Hadoop flavor of Splunk, the famous visualization tool. Splunk Overview: Splunk tool captures, indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, dashboards and visualizations. Splunk released a product […]

Hive Connectivity With Hunk (Splunk)

24 Oct, 2014 in Hadoop / Hive tagged Configure Hive to connect with Hunk / hadoop splunk use case / Hunk connectivity to Hive / hunk splunk analytics for hadoop configuration on linux for hive / splunk app for hadoop operations / Splunk for Hadoop / Splunk for Hive / splunk hadoop connect documentation / splunk hadoop hdfs / splunk hadoop hunk conectivity with hive metastore / splunk hunk hadoop hive provider index metastore / splunk hunk installation on ubuntu for hadoop / splunk on hadoop by Siva

Sqlline — Phoenix HBase Overview What is Apache Phoenix? Apache phoenix is another Top Level project from Apache Software Foundation. It provides an SQL interface to HBase. It is like an SQL layer on top of HBase architecture. It maps HBase data model to the relational world. Phoenix is developed in java […]

Apache Phoenix – An SQL Layer on HBase

21 Oct, 2014 in HBase / Phoenix tagged Apache Phoenix An SQL Driver for HBase / apache phoenix example queries on hbase tables / apache phoenix features strengths and limitations / apache phoenix hbase tutorials / apache phoenix installation configuration in linux / apache phoenix installation on ubuntu hadoop / apache phoenix performance / can we run sql queries on hbase tables / How to run SQL Commands on HBase / Pheonix HBase Integration / SQL Commands for HBase / SQL Interface for HBase / SQL on HBase with Apache Phoenix / Taming HBase with Apache Phoenix / what are the benefits or advantages of apache phoenix / what is apache phoenix / what is the need for apache phoenix / why is apache phoenix so fast by Siva

hive tez2 — In this post, we will discuss about Hive integration with Tez framework or Enabling Tez for Hive Queries. And we will also run sample hive queries both on Mapreduce and Tez frameworks and we will evaluate the performance difference between Tez and MR Frameworks. Tez Advantages: Tez offers a customizable […]

Hive on Tez – Hive Integration with Tez

19 Oct, 2014 in Hive / Tez tagged apache tez installation on ubuntu hadoop / differences between tez and mapreduce / Enable Tez for Hive queries / enable tez on hive queries / Hive integration with Tez / hive on tez configuration / Hive on Tez framework / hive tez configuration tutorial / Hive with Tez / run hive Queries on Tez examples / tez benefits or advantages over mapreduce / tez hive performance comparison with mapreduce / Tez Integration with Hive / tez vs mapreduce performances by Siva

Tez Dag job output — Apache Tez Overview What is Apache Tez? Apache Tez is another execution framework project from Apache Software Foundation and it is built on top of Hadoop YARN. It is considered as a more flexible and powerful successor of the mapreduce framework. Apache Tez Features: Tez provides, Performance gain over Map Reduce […]

Apache Tez – Successor of Mapreduce Framework

17 Oct, 2014 in Hadoop / Hive / Tez tagged apache hadoop tez tutorial / apache tez architecture / Apache Tez Features / apache tez hadoop / apache tez hive / apache tez installation / Apache Tez Installation on Hadoop / apache tez installation on ubuntu hadoop / apache tez tutorial / hadoop tez configuration / hadoop tez example / hadoop tez setup / hive on Tez / Installing and Configuring Apache Tez / setup of apache tez on hadoop cluster / Tez Data Processing model / Tez example job / tez framework / Tez Installation on Ubuntu / Tez layer on Hadoop 2 Architecture / Tez on Ubuntu Machine / tez on yarn / What is Apache Tez by Siva

Hive Table Mapping with HBase — In this post, we will discuss about the setup needed for HBase Integration with Hive and we will test this integration with the creation of some test hbase tables from hive shell and populate the contents of it from another hive table and finally verify these contents in hbase table. […]

HBase Integration with Hive

16 Oct, 2014 in HBase / Hive tagged Accessing/Querying Hbase tables via hive shell/commands / bulk load csv into hbase / bulk load into hbase example / bulk loading data in hbase / create hive external table on hbase / hbase bulk load example hive / HBase via Hive / HBaseIntegration with Apache Hive / hbasestoragehandler hive example / Hive and HBase Integration / Hive External Table Pointing to HBASE / hive hbase bulk load / hive hbase columns mapping / hive hbase integration performance / hive hbase integration tutorial / hive hbase table mapping / hive hbase.columns.mapping examples / Hive on Hbase / hive on hbase table examples / hive queries on hbase / Hive with HBase / How To Create External Hive Table on HBase / How to map Hive table and HBase table / Integration of HIve and HBase / Interacting with Hbase via hive / Mapping Existing HBase Tables to Hive / Register Hbase table in Hive / Setup for HBase Integration with Hive / Using Hive to interact with HBase by Siva

JSON View of avro file — This post is a continuation for previous post on working with small files issue. In previous we have merged huge number of small files on HDFS directory into sequencefile and in this post we will merge huge number of small files on local file system into avro file on HDFS […]

Merging Small Files Into Avro File

15 Oct, 2014 in Avro / Hadoop / Map Reduce tagged dealing with small file problem in hadoop / how to combine small files in hadoop / How to manage huge numbers of small files in hadoop / how to merging small files in hadoop / how to work with large number of small files in hadoop / importing small files into hdfs / join small files in hdfs / process small files on hadoop example / Processing large set of small files with Hadoop / solution for small files problem in hadoop by Siva

SequenceFile Key Extractor — In this post, we will discuss one of the famous use case of SequenceFiles, where we will merge large number of small files into SequenceFile. We will get to this requirement mainly due to the lack efficient processing of large number of small files in hadoop or mapreduce. Need For […]

Merging Small Files into SequenceFile

14 Oct, 2014 in Hadoop / Map Reduce tagged creating custom input format in hadoop / creating custom inputformat and recordreader example / creating custom record reader in hadoop / Dealing with Hadoop's small files problem / full fileinputformat example hadoop sequence file input format / hadoop custom inputformat example / hadoop custom recordreader example / Hadoop MapReduce with small files example / how to combine small files in hadoop / How to manage huge numbers of small files in hadoop mapreduce / how to work with large number of small files in hadoop / identityreducer hadoop example / importing small files into hdfs / joining small files into single file in hadoop hdfs mapreduce java / merging small files in hadoop / Merging Small Files into Avro File / merging small files into sequence file hadoop example / Processing large set of small files with Hadoop / small files problem in hadoop / solution for small files problem in hadoop by Siva

Error Scenario: java.io.IOException: Cannot create an instance of InputFormat class We will get this error message when we try to execute simple hadoop fs commands or running any hive queries. Below is the complete error message. [crayon-5cebaf98e3dc2237653270/] Root Cause: This error message will be received when there are any spaces […]