Skip to content

Commit cbd46a6

Browse files
sravottosravotto
committed
log scanner: adding cli to managing scans.
This change adds a subcommand to the visus cli to manage scans, allowing users to add/delete/view/test scan configurations in the database. The configuration is represented in a yaml file, like in the following example: name: cockroach_log enabled: true format: crdb-v2 path: /var/logs/cockroach.log patterns: - name : events regex: help : number of events The README has been updated to describe the new functionality.
1 parent 5622d89 commit cbd46a6

File tree

6 files changed

+603
-1
lines changed

6 files changed

+603
-1
lines changed

‎README.md‎

Lines changed: 65 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# visus
22

3-
Visus (latin for "the action of looking") enables users to collect metrics using arbitrary SQL queries and expose them in a Prometheus format. Optionally, it can be configured to filter the metrics CockroachDB available in the `/_status/vars` endpoint.
3+
Visus (latin for "the action of looking") enables users to collect metrics using arbitrary SQL queries and expose them in a Prometheus format. Optionally, it can be configured to extracts metrics from logs and filter the metrics CockroachDB available in the `/_status/vars` endpoint.
44

55
Visus runs as a sidecar on each node of a CockroachDB cluster, as shown in the diagram below.
66
It can also be used for any database that is compatible with the Postgres wire protocol.
@@ -172,6 +172,43 @@ Start the server to enable collection of metrics from Prometheus.
172172
$VISUS_USER start --bind-addr :8888 --insecure --endpoint "/_status/custom"
173173
```
174174

175+
## Scanning logs
176+
177+
Visus can also be used to scan cockroachdb log files: it will produce metrics
178+
for the events in the logs, adding labels for the level and the source of the
179+
event. For instance to collect all the events in the cockroack.log file, define
180+
a scan configuration `log.yaml` as follows:
181+
182+
```yaml
183+
name: cockroach_log
184+
enabled: true
185+
format: crdb-v2
186+
path: /var/log/cockroach.log
187+
patterns:
188+
- name : events
189+
regex:
190+
help : number of events
191+
```
192+
193+
Insert the scan configuration in the database:
194+
195+
```bash
196+
$VISUS_ADMIN scan put --yaml - < log.yaml
197+
```
198+
199+
Example of the metrics produced:
200+
201+
```text
202+
# HELP cockroach_log_events number of events
203+
# TYPE cockroach_log_events counter
204+
cockroach_log_events{level="I",source="cli/log_flags.go"} 6
205+
cockroach_log_events{level="I",source="cli/start.go"} 102
206+
cockroach_log_events{level="I",source="gossip/client.go"} 9
207+
cockroach_log_events{level="I",source="gossip/gossip.go"} 6
208+
cockroach_log_events{level="I",source="jobs/job_scheduler.go"} 3
209+
cockroach_log_events{level="I",source="jobs/registry.go"} 12
210+
```
211+
175212
## Histogram rewriting
176213

177214
Visus can also act as a proxy to filter and rewrite CockroachDB histograms (v22.1 and earlier) from a log-2 linear format (HDR histograms) to a log-10 linear format.
@@ -251,6 +288,33 @@ Global Flags:
251288
Use "visus collection [command] --help" for more information about a command.
252289
```
253290

291+
### Scan management commands
292+
293+
Use the `visus scan` command to manage the log scans in the database.
294+
295+
```text
296+
Usage:
297+
visus scan [command]
298+
299+
Available Commands:
300+
delete
301+
get
302+
list
303+
put
304+
test
305+
306+
Flags:
307+
-h, --help help for scan
308+
--url string Connection URL, of the form: postgresql://[user[:passwd]@]host[:port]/[db][?parameters...]
309+
310+
Global Flags:
311+
--logDestination string write logs to a file, instead of stdout
312+
--logFormat string choose log output format [ fluent, text ] (default "text")
313+
-v, --verbose count increase logging verbosity to debug; repeat for trace
314+
315+
Use "visus scan [command] --help" for more information about a command.
316+
```
317+
254318
### Histogram filter management commands
255319

256320
Use the `visus histogram` command to manage the collections in the database.

‎internal/cmd/scan/admin.go‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
// Copyright 2024 Cockroach Labs Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
// Package scan defines the sub command to run visus scan utilities.
16+
package scan
17+
18+
import (
19+
"sort"
20+
"strings"
21+
"time"
22+
23+
"github.com/cockroachlabs/visus/internal/cmd/env"
24+
"github.com/cockroachlabs/visus/internal/scanner"
25+
"github.com/cockroachlabs/visus/internal/stopper"
26+
"github.com/pkg/errors"
27+
"github.com/prometheus/client_golang/prometheus"
28+
"github.com/prometheus/common/expfmt"
29+
"github.com/spf13/cobra"
30+
)
31+
32+
var databaseURL = ""
33+
34+
// Command runs the scan tools to view and manage the configuration in the database.
35+
func Command() *cobra.Command {
36+
return command(env.Default())
37+
}
38+
39+
// command runs the tools to view and manage the configuration in the database.
40+
// An environment can be injected for standalone testing.
41+
func command(env *env.Env) *cobra.Command {
42+
c := &cobra.Command{
43+
Use: "scan",
44+
}
45+
f := c.PersistentFlags()
46+
c.AddCommand(
47+
getCmd(env),
48+
listCmd(env),
49+
deleteCmd(env),
50+
putCmd(env),
51+
testCmd(env))
52+
f.StringVar(&databaseURL, "url", "",
53+
"Connection URL, of the form: postgresql://[user[:passwd]@]host[:port]/[db][?parameters...]")
54+
return c
55+
}
56+
57+
// deleteCmd deletes a scan from the database.
58+
func deleteCmd(env *env.Env) *cobra.Command {
59+
return &cobra.Command{
60+
Use: "delete",
61+
Args: cobra.ExactArgs(1),
62+
Example: `./visus scan delete scan_name --url "postgresql://root@localhost:26257/defaultdb?sslmode=disable" `,
63+
RunE: func(cmd *cobra.Command, args []string) error {
64+
ctx := cmd.Context()
65+
scan := args[0]
66+
store, err := env.ProvideStore(ctx, databaseURL)
67+
if err != nil {
68+
return err
69+
}
70+
if err := store.DeleteScan(ctx, scan); err != nil {
71+
return errors.Wrapf(err, "unable to delete scan %s", scan)
72+
}
73+
cmd.Printf("Scan %s deleted.\n", scan)
74+
return nil
75+
},
76+
}
77+
}
78+
79+
// getCmd retrieves a scan configuration from the database.
80+
func getCmd(env *env.Env) *cobra.Command {
81+
return &cobra.Command{
82+
Use: "get",
83+
Args: cobra.ExactArgs(1),
84+
Example: `./visus scan get scan_name --url "postgresql://root@localhost:26257/defaultdb?sslmode=disable" `,
85+
RunE: func(cmd *cobra.Command, args []string) error {
86+
ctx := cmd.Context()
87+
scanName := args[0]
88+
store, err := env.ProvideStore(ctx, databaseURL)
89+
if err != nil {
90+
return err
91+
}
92+
scan, err := store.GetScan(ctx, scanName)
93+
if err != nil {
94+
return errors.Wrapf(err, "unable to retrieve scan %s", scanName)
95+
}
96+
if scan == nil {
97+
cmd.Printf("Scan %s not found\n", scanName)
98+
return nil
99+
}
100+
res, err := marshal(scan)
101+
if err != nil {
102+
return errors.Wrapf(err, "unable to retrieve scan %s", scanName)
103+
}
104+
cmd.Print(string(res))
105+
return nil
106+
},
107+
}
108+
}
109+
110+
// listCmd list all the log scans in the database
111+
func listCmd(env *env.Env) *cobra.Command {
112+
return &cobra.Command{
113+
Use: "list",
114+
Example: `./visus scan list --url "postgresql://root@localhost:26257/defaultdb?sslmode=disable" `,
115+
RunE: func(cmd *cobra.Command, args []string) error {
116+
ctx := cmd.Context()
117+
store, err := env.ProvideStore(ctx, databaseURL)
118+
if err != nil {
119+
return errors.Wrap(err, "unable to retrieve scans")
120+
}
121+
scans, err := store.GetScanNames(ctx)
122+
if err != nil {
123+
return errors.Wrap(err, "unable to retrieve scans")
124+
}
125+
sort.Strings(scans)
126+
for _, scan := range scans {
127+
cmd.Printf("%s\n", scan)
128+
}
129+
return nil
130+
},
131+
}
132+
}
133+
134+
// putCmd inserts a new scan in the database using the specified yaml configuration.
135+
func putCmd(env *env.Env) *cobra.Command {
136+
var file string
137+
c := &cobra.Command{
138+
Use: "put",
139+
Args: cobra.ExactArgs(0),
140+
Example: `./visus scan put --yaml config.yaml --url "postgresql://root@localhost:26257/defaultdb?sslmode=disable" `,
141+
RunE: func(cmd *cobra.Command, args []string) error {
142+
ctx := cmd.Context()
143+
if file == "" {
144+
return errors.New("yaml configuration required")
145+
}
146+
store, err := env.ProvideStore(ctx, databaseURL)
147+
if err != nil {
148+
return err
149+
}
150+
data, err := env.ReadFile(file)
151+
if err != nil {
152+
return errors.Wrap(err, "unable to read read configuration")
153+
}
154+
scan, err := unmarshal(data)
155+
if err != nil {
156+
return errors.Wrap(err, "unable to read scan configuration")
157+
}
158+
if err := store.PutScan(ctx, scan); err != nil {
159+
return errors.Wrapf(err, "unable to insert scan %s", scan.Name)
160+
}
161+
cmd.Printf("Scan %s inserted.\n", scan.Name)
162+
return nil
163+
},
164+
}
165+
f := c.Flags()
166+
f.StringVar(&file, "yaml", "", "file containing the configuration")
167+
return c
168+
}
169+
170+
// testCmd retrieves a scan configuration and execute it, returning the metrics extracted
171+
// from the log file in the scan.
172+
func testCmd(env *env.Env) *cobra.Command {
173+
var interval time.Duration
174+
var count int
175+
c := &cobra.Command{
176+
Use: "test",
177+
Args: cobra.ExactArgs(1),
178+
Example: `./visus scan test scan_name --url "postgresql://root@localhost:26257/defaultdb?sslmode=disable" `,
179+
RunE: func(cmd *cobra.Command, args []string) error {
180+
ctx := stopper.WithContext(cmd.Context())
181+
182+
scanName := args[0]
183+
store, err := env.ProvideStore(ctx, databaseURL)
184+
if err != nil {
185+
return err
186+
}
187+
scan, err := store.GetScan(ctx, scanName)
188+
if err != nil {
189+
return errors.Wrapf(err, "unable to retrieve scan %s", scanName)
190+
}
191+
if scan == nil {
192+
cmd.Printf("Scan %s not found\n", scanName)
193+
return nil
194+
}
195+
196+
scanner, err := scanner.FromConfig(scan,
197+
&scanner.Config{
198+
FromBeginning: true,
199+
Poll: true,
200+
Follow: false,
201+
},
202+
prometheus.DefaultRegisterer)
203+
if err != nil {
204+
return err
205+
}
206+
scanner.Start(ctx)
207+
for i := 1; i <= count || count == 0; i++ {
208+
select {
209+
case <-ctx.Done():
210+
return ctx.Err()
211+
case <-time.After(interval):
212+
}
213+
gathering, err := prometheus.DefaultGatherer.Gather()
214+
if err != nil {
215+
return errors.Wrap(err, "prometheus failure")
216+
}
217+
cmd.Printf("\n---- %s %s -----\n", time.Now().Format("01-02-2006 15:04:05"), scan.Name)
218+
for _, mf := range gathering {
219+
if strings.HasPrefix(*mf.Name, scan.Name) {
220+
expfmt.MetricFamilyToText(cmd.OutOrStdout(), mf)
221+
222+
}
223+
}
224+
}
225+
return scanner.Stop()
226+
},
227+
}
228+
f := c.Flags()
229+
f.DurationVar(&interval, "interval", 10*time.Second, "interval of scan")
230+
f.IntVar(&count, "count", 1, "number of times to run the scan. Specify 0 for continuos scan")
231+
return c
232+
}

0 commit comments

Comments
 (0)