数据仓库| 数据湖| 木地板 (Data Warehousing | Data Lake | Parquet)

Apache Parquet is a columnar storage format available to any project […], regardless of the choice of data processing framework, data model or programming language. — https://parquet.apache.org/

Apache Parquet是一种列式存储格式，可用于任何项目[…]，而与数据处理框架，数据模型或编程语言的选择无关。 — https://parquet.apache.org/

This description is a good summary of this format. This post will talk about the features of the format and why it is beneficial to analytical data queries in the data warehouse or lake.

此描述是此格式的很好总结。 这篇文章将讨论该格式的功能，以及为什么它有助于数据仓库或湖泊中的分析数据查询。

The first feature is the columnar storage nature of the format. This simply means that data is encoded and stored by columns instead of by rows. This pattern allows for analytical queries to select a subset of columns for all rows. Parquet stores columns as chunks and can further split files within each chunk too. This allows restricting the disk i/o operations to a minimum.

第一个功能是格式的柱状存储性质。 这仅表示数据是按列而不是按行编码和存储的。 此模式允许分析查询为所有行选择列的子集。 Parquet将列存储为大块，并且还可以在每个大块中进一步拆分文件。 这样可以将磁盘的I / O操作限制在最低限度。

The second feature to mention is data schema and types. Parquet is a binary format and allows encoded data types. Unlike some formats, it is possible to store data with a specific type of boolean, numeric( int32, int64, int96, float, double) and byte array. This allows clients to easily and efficiently serialise and deserialise the data when reading and writing to parquet format.

要提到的第二个功能是数据模式和类型。 Parquet是二进制格式，并允许编码的数据类型。 与某些格式不同，可以使用特定类型的布尔，数字(int32，int64，int96，float，double)和字节数组存储数据。 这样，客户端在读取和写入拼花格式时可以轻松有效地对数据进行序列化和反序列化。

In addition to the data types, Parquet specification also stores metadata which records the schema at three levels; file, chunk(column) and page header. The footer for each file contains the file metadata. This records the following:

除数据类型外，Parquet规范还存储元数据，该元数据在三个级别上记录了模式。 文件，块(列)和页面标题。 每个文件的页脚包含文件元数据。 记录以下内容：

• Version (of Parquet format)版本(拼花格式)
• Data Schema数据架构
• Column metadata (type, number of values, location, encoding)列元数据(类型，值的数量，位置，编码)
• Number of row groups行组数
• Additional key-value pairs其他键值对

The metadata is always written in the footer of the file as this allows a single pass write. In plain English, the data is written first, then the metadata can be accurately written knowing all the locations, size and encoding of the written data. Many formats write their metadata in the header. However, this requires multiple passes as data is written after the header. Parquet makes this efficient to read metadata and the data itself. Another advantage is that the file is splittable in any desirable size. For example, if you are using this with Spark or Amazon Redshift, you can specify writing files of 1GB size for efficient loading.

元数据总是写在文件的页脚中，因为这允许单次写入。 用简单的英语来说，首先要写入数据，然后可以在知道写入数据的所有位置，大小和编码的情况下准确地写入元数据。 许多格式将其元数据写入标头中。 但是，这需要多次通过，因为将数据写入标头之后。 Parquet可以有效地读取元数据和数据本身。 另一个优点是文件可拆分为任何所需的大小。 例如，如果将它与Spark或Amazon Redshift一起使用，则可以指定写入1GB大小的文件以进行有效加载。

A separate metadata file is part of the specification allowing, multiple parquet files to be referenced. So the dataset can be arbitrarily large or small to fit in a single file or multiple files. This is particularly advantageous when working with gigabytes or terabytes of data in the data lake. Most of today’s applications and warehouses allow parallel reads. Multiple files mean data can be read in parallel to speed up execution.

一个单独的元数据文件是该规范的一部分，允许引用多个实木复合地板文件。 因此，数据集可以任意大小，以适合单个文件或多个文件。 当在数据湖中处理千兆字节或兆兆字节的数据时，这特别有利。 当今大多数应用程序和仓库都允许并行读取。 多个文件意味着可以并行读取数据以加快执行速度。

Combining the schema and metadata with splittable files makes Parquet a flexible format. The schema can evolve over time. An example is if a field/column is added to the dataset, this is simply encoded within the new chunks and files. the metadata file is updated to record that only certain files and row groups include the new chunk. Thus schema evolution and merging can easily occur. Where files do not contain the new field, they simply result in the field not existing. If multiple files are read where some contain the field and others don’t, null values are used to denote missing column values.

将模式和元数据与可拆分文件结合在一起，可使Parquet成为灵活的格式。 该架构可以随着时间而发展。 例如，如果将字段/列添加到数据集中，则只需在新的块和文件中对其进行编码。 更新元数据文件以记录仅某些文件和行组包括新块。 因此，架构演化和合并很容易发生。 如果文件不包含新字段，它们只会导致该字段不存在。 如果读取了多个文件，其中一些文件包含该字段，而另一些文件不包含该字段，则使用null值表示缺少的列值。

Lastly, the format supports compression natively within its files. This means that data can be efficiently compressed when it is low-medium cardinality by column. When data is high cardinality, compression can be performed on these columns separately in a different file. This allows variability in encoding and compression of different fields and data types. This is another advantage when reading the data efficiently with high throughput.

最后，该格式本身支持其文件内的压缩。 这意味着当数据为低基数列时，可以有效地压缩数据。 当数据具有高基数时，可以在不同文件中分别对这些列执行压缩。 这允许不同字段和数据类型的编码和压缩中的可变性。 当以高吞吐量高效读取数据时，这是另一个优势。

Overall, Parquet’s features of storing data in columnar format together with schema and typed data allow efficient use for analytical purposes. It provides further benefits through compression, encoding and splittable format for parallel and high throughput reads. The metadata enables all this while providing the flexibility of storage and schema evolution. Parquet has been a de-facto format for analytical data lakes and warehouses. Many tools and frameworks support this such as Hadoop, Spark, AWS Redshift, and Databricks platform. Databricks also provide a new flavour to Parquet that allows data versioning and “time travel” with their Delta Lake format. (Teaser: there may be an upcoming post on Delta format.)

总体而言，Parquet以列格式存储数据以及模式和类型化数据的功能可以有效地用于分析目的。 它通过压缩，编码和可拆分格式为并行和高吞吐量读取提供了更多好处。 元数据可以实现所有这些功能，同时提供存储和架构演进的灵活性。 对于分析数据湖泊和仓库，实木复合地板已成为事实上的格式。 许多工具和框架都支持此功能，例如Hadoop，Spark，AWS Redshift和Databricks平台。 Databricks还为Parquet提供了一种新的风格，允许使用Delta Lake格式进行数据版本控制和“时间旅行”。 (Teaser：可能会有即将发布的有关Delta格式的帖子。)