Skip to content

Latest commit

 

History

History
184 lines (140 loc) · 5.21 KB

index.md

File metadata and controls

184 lines (140 loc) · 5.21 KB

First steps

This guide describes how to deploy and run CLP on a single host.

:::{tip} For deployments that scale across multiple machines (e.g., for higher throughput), see:

The rest of the guide is organized as follows:

:::{note} CLP supports deployment using Docker Compose or Kubernetes. Throughout the guide, some steps will differ depending on the chosen orchestration framework. These steps will have a tabbed interface to choose your framework. :::

System Requirements

::::{tab-set} :::{tab-item} Docker Compose :sync: docker

  • Docker
    • containerd.io >= 1.7.18
    • docker-ce >= 27.0.3
    • docker-ce-cli >= 27.0.3
    • docker-compose-plugin >= 2.28.1

To check whether the required tools are installed on your system, run:

containerd --version
docker version --format '{{.Server.Version}}'
docker compose version --short
* If you're not running as root, ensure Docker can be run
  [without superuser privileges][docker-non-root].
* If you're using Docker Desktop, ensure version 4.34 or higher is installed.

:::

:::{tab-item} Kubernetes (kind) :sync: kind

kind (Kubernetes in Docker) runs a Kubernetes cluster inside Docker containers, making it ideal for local Kubernetes testing and development.

  • Docker (required for kind)
    • containerd.io >= 1.7.18
    • docker-ce >= 27.0.3
    • docker-ce-cli >= 27.0.3
  • kind >= 0.23
  • kubectl >= 1.30
  • Helm >= 4.0

To check whether the required tools are installed on your system, run:

containerd --version
docker version --format '{{.Server.Version}}'
kind version
kubectl version --client --output=yaml | grep gitVersion
helm version --short

::: ::::

Choosing a flavor

There are two flavors of CLP:

  • clp-json for compressing and searching JSON logs.
  • clp-text for compressing and searching unstructured text logs.

:::{note} Both flavors contain the same binaries but are configured with different values for the package.storage_engine key in the package's config file (etc/clp-config.yaml). :::

clp-json

The JSON flavor of CLP is appropriate for JSON logs, where each log event is an independent JSON object. For example:

{
  "t": {
    "$date": "2023-03-21T23:46:37.392"
  },
  "ctx": "conn11",
  "msg": "Waiting for write concern."
}
{
  "t": {
    "$date": "2023-03-21T23:46:37.392"
  },
  "msg": "Set last op to system time"
}

The log file above contains two log events represented by two JSON objects printed one after the other. Whitespace is ignored, so the log events could also appear with no newlines and indentation.

If you're using JSON logs, download and extract the clp-json release from the Releases page, then proceed to the clp-json quick-start guide.

clp-text

The text flavor of CLP is appropriate for unstructured text logs, where each log event contains a timestamp and may span one or more lines.

:::{note} If your logs don't contain timestamps or CLP can't automatically parse the timestamps in your logs, it will treat each line as an independent log event. :::

For example:

2015-03-23T15:50:17.926Z INFO container_1 Transitioned from ALLOCATED to ACQUIRED
2015-03-23T15:50:17.927Z ERROR Scheduler: Error trying to assign container token
java.lang.IllegalArgumentException: java.net.UnknownHostException: i-e5d112ea
    at org.apache.hadoop.security.buildTokenService(SecurityUtil.java:374)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
Caused by: java.net.UnknownHostException: i-e5d112ea
    ... 17 more

The log file above contains two log events, both beginning with a timestamp. The first is a single line, while the second contains multiple lines.

If you're using unstructured text logs, download and extract the clp-text release from the Releases page, then proceed to the clp-text quick-start guide.

Using CLP

Once you've installed CLP's requirements and downloaded a CLP release, proceed to the quick-start guide for your chosen flavor by clicking the corresponding link below.

::::{grid} 1 1 2 2 :gutter: 2

:::{grid-item-card} :link: clp-json CLP for JSON logs ^^^ How to compress and search JSON logs. :::

:::{grid-item-card} :link: clp-text CLP for unstructured text logs ^^^ How to compress and search unstructured text logs. ::: ::::