Data Science

Hive is a software framework
It provides a SQL-like interface to various databases
It was built for HBase and HDFS, but can be used for others
Hive is sometimes referred to as Hive Hadoop
This is because it easily integrates with HBase and HDFS
Hive is used for:
- Simplifying the need for writing complex MapReduce jobs
- Tracks data that is critical
- Indexing to provide fast queries

In Hive, data is stored in HDFS
Hive creates definitions for the following:
- Tables
- Databases
- Schemas
- HQL operations
These definitions are stored in a metastore
The metastore is separate from the data
It could be any RDBMS database
The HQL operations are:
- SQL-like operations
- Translated to pre-implemented MapReduce jobs

The metastore consists of relational tables
These tables contain metadata for objects created in Hive
The metadata could store the following:
- Column names
- Data types
- Indexes
- Comments

Consider two types of tools used in Hadoop:
- Execution engines
- Query optimizers
These are both individual software frameworks
An execution engine is required in Hadoop
A query optimizer is optional
An execution engine processes jobs related to the data
A query optimizer optimizes queries before they are processed
- Sometimes, they optimize queries during processing too

Hive is mostly referred to as a query optimizer
- This is because Hive is essentially a metastore
Spark is mostly referred to as an execution engine
Hive uses MapReduce as its execution engine by default
Spark uses its own execution engine
- It is an alternative of MapReduce
Spark offers query optimizers as well
- It uses catalyst optimizers
- This uses rule-based and cost-based optimization
Spark and Hive can be used together
Specifically, we can include:
- HDFS as our storage layer
- Hive's metastore for query optimization
- Spark's query optimization
- Either Spark or MapReduce as an execution engine

YARN

Sqoop