What does Apache Hive do?

The Apache Hive™ data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage and queried using SQL syntax.

Built on top of Apache Hadoop™, Hive provides the following features:

• Tools to enable easy access to data via SQL, thus enabling data warehousing tasks such as extract/transform/load (ETL), reporting, and data analysis.
• A mechanism to impose structure on a variety of data formats
• Access to files stored either directly in Apache HDFS™ or in other data storage systems such as Apache HBase™
• Query execution via Apache Tez™, Apache Spark™, or MapReduce
• Procedural language with HPL-SQL
• Sub-second query retrieval via Hive LLAP, Apache YARN and Apache Slider.Hive provides standard SQL functionality, including many of the later SQL:2003 and SQL:2011 features for analytics.
  Hive’s SQL can also be extended with user code via user defined functions (UDFs), user defined aggregates (UDAFs), and user defined table functions (UDTFs).There is not a single “Hive format” in which data must be stored. Hive comes with built in connectors for comma and tab-separated values (CSV/TSV) text files, Apache Parquet™, Apache ORC™, and other formats.
  Users can extend Hive with connectors for other formats. Please see File Formats and Hive SerDe in the Developer Guide for details.Hive is not designed for online transaction processing (OLTP) workloads. It is best used for traditional data warehousing tasks.
  Hive is designed to maximize scalability (scale out with more machines added dynamically to the Hadoop cluster), performance, extensibility, fault-tolerance, and loose-coupling with its input formats.Components of Hive include HCatalog and WebHCat.
  • HCatalog is a component of Hive. It is a table and storage management layer for Hadoop that enables users with different data processing tools — including Pig and MapReduce — to more easily read and write data on the grid.
  • WebHCat provides a service that you can use to run Hadoop MapReduce (or YARN), Pig, Hive jobs or perform Hive metadata operations using an HTTP (REST style) interface.
    

    What Hive Is NOT

    Hive is not designed for online transaction processing.  It is best used for traditional data warehousing tasks.

    Data Units

    In the order of granularity – Hive data is organized into:

    • Databases: Namespaces function to avoid naming conflicts for tables, views, partitions, columns, and so on.  Databases can also be used to enforce security for a user or group of users.
    • Tables: Homogeneous units of data which have the same schema. An example of a table could be page_views table, where each row could comprise of the following columns (schema):
      • timestamp—which is of INT type that corresponds to a UNIX timestamp of when the page was viewed.
      • userid —which is of BIGINT type that identifies the user who viewed the page.
      • page_url—which is of STRING type that captures the location of the page.
      • referer_url—which is of STRING that captures the location of the page from where the user arrived at the current page.
      • IP—which is of STRING type that captures the IP address from where the page request was made.
    • Partitions: Each Table can have one or more partition Keys which determines how the data is stored. Partitions—apart from being storage units—also allow the user to efficiently identify the rows that satisfy a specified criteria; for example, a date_partition of type STRING and country_partition of type STRING. Each unique value of the partition keys defines a partition of the Table. For example, all “US” data from “2009-12-23” is a partition of the page_views table. Therefore, if you run analysis on only the “US” data for 2009-12-23, you can run that query only on the relevant partition of the table, thereby speeding up the analysis significantly. Note however, that just because a partition is named 2009-12-23 does not mean that it contains all or only data from that date; partitions are named after dates for convenience; it is the user’s job to guarantee the relationship between partition name and data content! Partition columns are virtual columns, they are not part of the data itself but are derived on load.
    • Buckets (or Clusters): Data in each partition may in turn be divided into Buckets based on the value of a hash function of some column of the Table. For example the page_views table may be bucketed by userid, which is one of the columns, other than the partitions columns, of the page_view table. These can be used to efficiently sample the data.

    Note that it is not necessary for tables to be partitioned or bucketed, but these abstractions allow the system to prune large quantities of data during query processing, resulting in faster query execution.

Source: Home – Apache Hive – Apache Software Foundation