Skip to content

index.md

Latest commit

 

History

History
80 lines (65 loc) · 5.24 KB

index.md

File metadata and controls

80 lines (65 loc) · 5.24 KB
id index
title Ingestion overview
sidebar_label Overview

Loading data in Druid is called ingestion or indexing. When you ingest data into Druid, Druid reads the data from your source system and stores it in data files called segments. In general, segment files contain a few million rows each.

For most ingestion methods, the Druid Middle Manager processes or the Indexer processes load your source data.

During ingestion, Druid creates segments and stores them in deep storage. Historical nodes load the segments into memory to respond to queries. For streaming ingestion, the Middle Managers and indexers can respond to queries in real-time with arriving data. For more information, see Storage overview.

This topic introduces streaming and batch ingestion methods. The following topics describe ingestion concepts and information that apply to all ingestion methods:

  • Druid schema model introduces concepts of datasources, primary timestamp, dimensions, and metrics.
  • Data rollup describes rollup as a concept and provides suggestions to maximize the benefits of rollup.
  • Partitioning describes time chunk and secondary partitioning in Druid.
  • Ingestion spec reference provides a reference for the configuration options in the ingestion spec.

For additional information about concepts and configurations that are unique to each ingestion method, see the topic for the ingestion method.

Ingestion methods

The tables below list Druid's most common data ingestion methods, along with comparisons to help you choose the best one for your situation. Each ingestion method supports its own set of source systems to pull from. For details about how each method works, as well as configuration properties specific to that method, check out its documentation page.

Streaming

There are two available options for streaming ingestion. Streaming ingestion is controlled by a continuously-running supervisor.

Method Kafka Kinesis
Supervisor type kafka kinesis
How it works Druid reads directly from Apache Kafka. Druid reads directly from Amazon Kinesis.
Can ingest late data? Yes. Yes.
Exactly-once guarantees? Yes. Yes.

Batch

There are two available options for batch ingestion. Batch ingestion jobs are associated with a controller task that runs for the duration of the job.

Method Native batch SQL
Controller task type index_parallel query_controller
How you submit it Send an index_parallel spec to the Tasks API. Send an INSERT or REPLACE statement to the SQL task API.
Parallelism Using subtasks, if maxNumConcurrentSubTasks is greater than 1. Using query_worker subtasks.
Fault tolerance Workers automatically relaunched upon failure. Controller task failure leads to job failure. Controller or worker task failure leads to job failure.
Can append? Yes. Yes (INSERT).
Can overwrite? Yes. Yes (REPLACE).
External dependencies None. None.
Input sources Any inputSource. Any inputSource (using EXTERN) or Druid datasource (using FROM).
Input formats Any inputFormat. Any inputFormat.
Secondary partitioning options Dynamic, hash-based, and range-based partitioning methods are available. See partitionsSpec for details. Range partitioning (CLUSTERED BY).
Rollup modes Perfect if forceGuaranteedRollup = true in the tuningConfig. Always perfect.