-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathindex.html
More file actions
2073 lines (1763 loc) · 80.7 KB
/
Copy pathindex.html
File metadata and controls
2073 lines (1763 loc) · 80.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en-US" prefix="og: http://opg.me/ns#">
<head>
<meta charset="UTF-8" />
<meta name="title" property="og:title" content="Crato" />
<meta
name="description"
property="og:description"
content="Crato is an open source framework for small web applications to easily deploy a centralized logging solution that maintains ownership of data"
/>
<meta name="type" property="og:type" content="website" />
<meta
name="url"
property="og:url"
content="https://crato-logging.github.io/"
/>
<meta
name="image"
property="og:image"
content="images/logos/crato-logo.png"
/>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta
name="author"
content="Faazil Shaikh, Kurth O'Connor, Alex Soloviev"
/>
<title>CRATO</title>
<link
rel="icon"
type="image/png"
sizes="16x16"
href="images/icons/favicon_package_v0.16/favicon-16x16.png"
/>
<link
href="https://fonts.googleapis.com/css?family=Hind|Hind:700|Open+Sans:700|Teko:700&display=swap"
rel="stylesheet"
/>
<!-- <style>reset</style> -->
<link rel="stylesheet" href="stylesheets/reset.css" />
<link
rel="stylesheet"
href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.13.1/styles/gruvbox-dark.min.css"
charset="utf-8"
/>
<!-- <style></style> -->
<link rel="stylesheet" href="stylesheets/main.css" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.13.1/highlight.min.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<!-- <script></script> -->
<script src="javascripts/application.js"></script>
</head>
<body>
<div class="logo-links">
<p id="crato-logo">CRATO</p>
<a href="https://github.com/crato-logging/crato" target="_blank">
<img
src="images/logos/github_black.png"
alt="github logo"
id="github-logo"
/>
</a>
</div>
<a id="toTop-link" href="#" target="_blank">
<img
src="images/logos/back-to-top.png"
alt="Back to top"
id="toTop-logo"
/>
</a>
<nav id="site-navigation">
<ul>
<li>
<a href="#home" id="home-link">HOME</a>
</li>
<li>
<a href="#case-study" id="case-study-link">CASE STUDY</a>
<nav id="case-study-mobile">
<ul></ul>
</nav>
</li>
<li>
<a href="#our-team" id="our-team-link">OUR TEAM</a>
</li>
</ul>
</nav>
<header id="home">
<h1>
<img src="images/logos/crato-logo.png" alt="Crato logo" />
<p>easy log management for small applications</p>
</h1>
</header>
<section class="integration">
<div class="box">
<img src="images/crato_horizontal.png" alt="best practices" />
</div>
<article class="box">
<div class="text-box">
<h1>Centralize your logs</h1>
<p>
Crato is an open-source framework for small applications to easily
deploy a centralized logging solution that maintains ownership of
data.
</p>
<!-- <a class="button" href="#case-study">Learn More</a> -->
</div>
</article>
</section>
<section class="integration">
<article class="box">
<div class="text-box">
<h1>Easy To Deploy</h1>
<p>
Get started with a few simple commands.
</p>
<!-- <a class="button" href="#case-study">Learn More</a> -->
</div>
</article>
<div class="box">
<img
id="banner-deploy"
src="images/diagrams/gifs/crato_docker_deploy.gif"
class="softened"
alt="Gif of crato's install-kafka and deploy commands"
/>
</div>
</section>
<main>
<section id="case-study">
<h1>Case Study</h1>
<div id="side-nav">
<img src="images/logos/crato-logo.png" alt="Crato logo" />
</div>
<nav>
<ul></ul>
</nav>
<h2 id="introduction">1 Introduction</h2>
<h3>1.1 What is Crato?</h3>
<p>
Crato is an open source framework for small applications to easily
deploy centralized logging.
</p>
<p>
Applications and system services record their events to local log
files, but by default these log messages remain on local systems and
are eventually either archived away never to be seen or permanently
deleted. Without intervening then, developers and administrators lose
valuable insight into the history, performance, and security of their
systems and applications.
</p>
<p>
Crato allows developers to quickly and easily deploy a centralized
logging infrastructure to address this problem. Using Crato, each
machine sends its logs to a central collection server, where
developers can gain insight into their entire system by analyzing the
consolidated logs across the entire deployed fleet of systems and
applications. Besides log data consolidation and analysis (via
InfluxDB), Crato also provides smart compression and archiving (via
Amazon S3 and S3 Glacier). Crato makes configuring, deploying, and
maintaining all these pieces easy and fast.
</p>
<p>
Before we get into Crato, let's take a step back and discuss what logs
are.
</p>
<h2 id="logging">2 Logging</h2>
<h3>2.1 What is a Log?</h3>
<p>
Developers rely on logs of all sorts on a daily basis. In the simplest
case, we can think of logs as distinct events that get written to
files<sup><a href="#footnote-1">1</a></sup
>. Log data is often the primary way to determine application health,
debug a problem, or derive aggregate metrics. It's the first place any
developer looks to try to piece together what happened in an event of
an error or exception. Logs are critical and every piece of software,
from infrastructure to operating system services, are continuously
logging events as soon as the system is up.
</p>
<h3>2.2 Logs as Files</h3>
<p>
In most UNIX systems, these events are written to log files. For the
most common applications and services, these files can be found in the
<code>var/log</code> directory. Within that directory, some services
will create sub-directories to organize their logs. For example, MySQL
generates its own log file and the <code>httpd</code> sub directory
contains log files for the Apache web server
<code>access_log</code> and <code>error_log</code>. There are also
system process logs that can be found in files like
<code>kern.log</code> or <code>auth.log</code>
that contain information logged by the kernel and system authorization
information, respectively.
</p>
<div class="img-wrapper">
<img src="images/diagrams/logs_as_files.png" alt="Logs as files" />
</div>
<p>
Because log messages are generated by some process and get written to
files on our local file system, it's natural to think of logs simply
as files. If logs are just files, then we use commands like
<code>grep</code> to match against some string or regular expression
in the log file.
</p>
<div class="img-wrapper">
<img
src="images/diagrams/gifs/grep.gif"
class="softened"
id="grep"
alt="Logs as files"
/>
<p>
We can use <code>grep</code> to search a file for logs generated in
the last 10 minutes.
</p>
</div>
<p>
But in order to better understand the structure of application logs,
let's consider another description of what a log is. We know that a
log is a file but that's not the full story.
</p>
<h3>2.3 Logs as Append-Only Data Structures</h3>
<blockquote>
<p>
"A log is perhaps the simplest possible storage abstraction. It is
an append-only, totally-ordered sequence of records ordered by time"
</p>
<p>- Jay Kreps</p>
</blockquote>
<p>
In the simplest case, we have an application that runs on a single
server, and our life is relatively easy. When the application is
contained on one server, so is its log data - when something goes
wrong we know where to look and it's fairly straightforward to paint
the entire picture of the application state. However, as applications
continue to grow in complexity so does the volume of their log data -
the many components, devices and microservices that produce log data
are often distributed on multiple servers and even with a relatively
small number of users, this can quickly become a big problem of
incomplete information. For this reason there has been a greater need
to unify log data, both by location as well as format. On top of data
consolidation, there's also been a growing need to synthesize the
proliferation of logs for business insights. This also implies a
growing need for machines to process log data.
</p>
<p>
In order for a machine to parse and aggregate log data to then be
consumed by humans, a text file may no longer be sufficient. But how
do we store log data, if not in a file?
</p>
<p>
The key is to recognize that logs as files are just one form of a more
general concept of a log as a data structure. That is, if we regard a
log as a time-ordered, append only sequence of records, we can see log
files as just one example. In this light, a log and its basic
properties can be represented by the following diagram.
</p>
<div class="img-wrapper">
<img src="images/diagrams/log_data_structure.png" alt="Log" />
</div>
<p>
Each entry in a log is added sequentially making this an append-only
data structure. We start on the left and each subsequent entry is then
added to the right. The entries on the left are then by definition
older than entries on the right, giving this data structure a built-in
ordering<sup><a href="#footnote-2">2</a></sup
>. Since reads also proceed from left to write, in the context of
building an application, if each entry in a log represents some event,
we can know the order in which the events occurred. In this way, logs
provide a historical context for our applications.
</p>
<h3>2.4 Logs as Streams</h3>
<p>
The "Twelve-Factor App" is a set of best practices for building robust
and maintainable web applications. One of the guidelines directly
address logging, suggesting that we should treat logs not as files but
as event streams, and that a web application shouldn’t concern itself
with the storage of this information. Instead, these logs should be
treated as a continuous stream that is captured and stored by a
separate service<sup><a href="#footnote-3">3</a></sup
>.
</p>
<p>
Of course it's natural to think of logs as files since it's a common
way to interact with logs and because log file generation is typically
an automatic process that happens behind the scenes. But log messages
get written to files mainly because it's a convenient method of
storing data. As we have seen, outside of the context of a file,
individual log messages can be viewed as time ordered streams of
events that <em>can</em>
be written to files but that can also be processed in other ways. To
borrow another quote from Jay Kreps, "Log is another word for stream
and logs are at the heart of stream processing". Streams of data can
be directed or split into multiple streams and sent to different
locations for further processing.
</p>
<p>
If we use a command line utility like <code>tail</code> it's possible
to see a live stream of logs that are coming from an application or
any other process we'd like to monitor. So while messages are being
written to files, from this view it's easier to think of logs as a
collection of events that we can view in real time as they occur -
something that's ongoing, with no beginning or an end.
</p>
<div class="img-wrapper">
<img
src="images/diagrams/gifs/tail.gif"
class="softened"
id="tail"
alt="Logs as streams"
/>
<p>
Ongoing, collated collection of events viewed in real time with the
<code>tail</code> utility.
</p>
</div>
<h3>2.5 Logs Capture Change</h3>
<p>
To review, a log at its core is an append-only sequence of records
that has a natural notion of "time" because newer records are appended
to the end of the log. Taken together, these properties mean that a
log effectively captures changes over time, thereby
<i>creating a full history of the application</i>.
</p>
<p>
We most often think of our application's database, where we keep the
business logic of our application, as the source of truth. If some
data on the UI doesn't look right, we consult the database to
determine whether it's correct. But even a database can't tell you how
the data got to that state. In some situations, knowing the historical
context for how we got to a particular state can be even more
important. As espoused by Martin Kleppman, Jay Kreps, and others, this
idea is known as the "Stream-Table Duality".
</p>
<h3>2.6 Stream-Table Duality</h3>
<p>
Stream-Table duality is another way of saying that there are distinct
but related ways to represent data. The familiar way is of course with
a table which represents a snapshot of current state. Let's consider a
simple example below using relational data in tables. There are two
users and they are transferring funds between one another. As changes
occur over time, the data in a table is updated to reflect a new
current state.
</p>
<div class="buttons">
<button id="resetButton" class="cratoButton">Reset</button>
<button id="nextButton" class="cratoButton">→</button>
</div>
<div class="img-wrapper">
<img
src="images/diagrams/table_states/state1.png"
id="table-state"
data-id="1"
alt="Table State1"
/>
<p>
As changes occur over time, the data in a table is updated to
reflect a new current state. But how did we get there?
</p>
</div>
<p>
From this example, we can see that data in the table is mutable since
the old values are replaced by new values. When we isolate a table and
look at it at any point in time, it's possible to answer questions
about the current state - for example, what <em>is</em> the current
balance of a customer with username big_ed. However, looking only at a
database record at one point in time doesn't paint a full picture -
there are many ways you could have arrived from one state to another
and it is only the log that can describe that journey. The log is the
database's changelog and it's there that we can find the full story of
how the current data arrived at its current state.
</p>
<p>
Representing the same history that we saw above but this time as a
stream of changes would look like this:
</p>
<div class="img-wrapper">
<img src="images/diagrams/gifs/stream.gif" alt="Stream" />
<p>
A stream captures the changes between the table's states as distinct
events.
</p>
</div>
<p>
From a log-as-stream perspective, the data is immutable since the
values are not updated. Instead, distinct and immutable events are
appended to the end of the log. What is captured in the blue boxes are
the changes between the table's states. This stream of changes can
supplement the core business logic that lives in tables and helps
answer not only <em>what</em> changed but also <em>why</em> and
<em>how</em> it changed<sup><a href="#footnote-4">4</a></sup
>.
</p>
<div class="img-wrapper">
<img
src="images/diagrams/stream_table_duality.png"
alt="Stream-table duality"
/>
<p>
Tables show us current state and streams capture how we got there.
</p>
</div>
<p>
And a log is useful in another way besides providing historical
context for existing data. Because a log captures the changes
themselves, it is possible to rebuild the current state of any
database from the database logs alone. In fact, it is possible to
rebuild the "current state" of any point in time of the database from
logs.
</p>
<div class="img-wrapper">
<img
src="images/diagrams/stream_to_table.png"
alt="Stream to a table"
/>
<p></p>
</div>
<p>
In this way, it's possible to think of a log as the real source of
truth since we can recreate a snapshot of the database at any point in
time from logs. That is, we can aggregate streams to arrive at tables,
but we cannot understand the historical context from only tables.
</p>
<h3>2.7 Why Does Logging Matter for Small Applications?</h3>
<p>
There are obvious benefits to logging that are important in day to day
development. Developers use logs every day for analysis and debugging,
infrastructure monitoring and for deriving useful metrics. Some types
of applications are also required to keep logs long term for audit
purposes. However, besides these uses, carefully curated logs related
to your application can provide valuable historical context that
unlocks other benefits:
</p>
<ul>
<li>We can see precisely how our applications are being used</li>
<li>Data can be used to find trends, patterns, and behaviors</li>
<li>
With historical event logs we can rebuild a snapshot of our
application state at any point in time
</li>
</ul>
<p>
For a small but growing application, there are many options to get
started with logging. Despite the myriad of options, many companies
opt to deprioritize logging from the outset. Yet, the power of logs
can only be realized later; by the time an application owner decides
she wants to look at historical data, it's too late to get started as
previous data is already lost. By default, many systems delete log
data; in some cases, it's archived into various machines and unusable
for meaningful analysis. Most small application owners do not think
about a logging strategy until significant data is lost.
</p>
<p>
In the next section, we'll discuss how to get started with Syslog, the
standard log recording format on UNIX, and the various architectural
deployment options that small application owners have for centralizing
and storing their log data.
</p>
<h2 id="syslog">3 Syslog</h2>
<h3>3.1 Single Host Architecture</h3>
<p>
Previously, we talked about how log files are automatically generated
by a logging system on the operating system. Syslog is the most widely
used logging system. In fact, syslog is a UNIX standard that defines a
message format and allows for standardization of how logs are
generated and processed.
</p>
<p>
Since the original inception of Syslog in the 1980's, there are now
various implementations of the syslog daemon, with rsyslog and
syslog-ng being the most notable examples. For the purposes of our
discussion here, we'll use the generic syslog daemon in our diagrams,
however any of the more modern implementations can be used in its
place.
</p>
<div class="img-wrapper">
<img src="images/diagrams/syslog_local.png" alt="Syslog local" />
<p></p>
</div>
<p>
First let's take a look at what syslog looks like locally. In a simple
case, for example on a single node system, the log daemon will receive
log messages and then write them to the local file system as we saw
earlier - typically the files are written to the
<code>var/log</code> directory.
</p>
<p>
There is a separate process called "log rotation" that is responsible
for compressing, archiving and eventually removing old log files. Log
rotation occurs daily, weekly or when a file reaches a certain size -
this depends on the configuration<sup><a href="#footnote-5">5</a></sup
>. For example, you can choose to keep 3 days worth of logs, in which
case the log rotation process will rename the log file to make room
for the next file, eventually leaving the most recent three files:
<code>example.log.1</code>, <code>example.log.2</code> and
<code>example.log.3</code>
</p>
<p>
During the next iteration, the <code>example.log.3</code> file will
get rotated out and <code>example.log.2</code> file takes its place
and is renamed <code>example.log.3</code>.
</p>
<div class="img-wrapper">
<img
src="images/diagrams/syslog_local2.png"
alt="Syslog local on client"
/>
<p></p>
</div>
<p>
This process occurs behind the scenes in a localized system. But
applications are rarely deployed on a single node. There are typically
many moving parts and various components across different servers.
Each of these components is generating its own log data in real time,
so the question is how do you as a developer sift through this data
and correlate events from each component?
</p>
<p>
Imagine that you have an error and you need to refer to your logs to
see what went wrong. In this case you would have to individually
connect to every single one of the servers one at a time, locate logs
on each machine and try to find the information that you are looking
for, and then try to correlate different events across different
machines, perhaps by their timestamp, to understand what went wrong.
</p>
<div class="img-wrapper">
<img
src="images/diagrams/gifs/local_logging.gif"
alt="Localized logging"
/>
<p></p>
</div>
<p>
This is not what we want. Having all these different logs hosted in
different locations makes the correlation and analysis difficult and
on top of that, through the process of log rotation, it might be
possible that the log file you're looking for has been rotated out and
deleted forever. Ideally we would bring together all these logs into
one central place, where we can analyze the logs in aggregate and come
up with a uniform back-up strategy for them.
</p>
<h3>3.2 Forwarding Messages to a Central Server</h3>
<p>
Syslog can also help with forwarding log events across machines. It
does this by having facilities for both sending and receiving log
events: a syslog daemon for each log-sending machine and a syslog
daemon on the log-receiving machine where all log data is to be
centralized.
</p>
<div class="img-wrapper">
<img
src="images/diagrams/syslog_centralized.png"
alt="Centralized syslog"
/>
<p></p>
</div>
<p>
This time, the syslog daemon on each of your components captures log
messages but instead of simply writing them to a file, it forwards
messages to a remote machine where another syslog daemon accepts them.
Here, it's possible to view our logs as a consolidated whole. Already
this is a big improvement from the disjointed scenario we had above.
</p>
<p>
Now you can inspect all your logs from a single place and you don’t
have to rely on connecting to every machine individually; you can
directly see the logs from the same filesystem. This also makes for a
far more reliable way to access logs because even if one of these
clients becomes inaccessible, all logs up to the point of the client's
failure are already in the central server.
</p>
<div class="img-wrapper">
<img
src="images/diagrams/centralized_logging.png"
alt="Centralized logging"
/>
<p></p>
</div>
<p>
Now we have a much more desirable and more manageable situation and
this is the basis for a centralized logging architecture. Once all
logs that are relevant to your application are consolidated in one
place, their value in aggregate starts to become more apparent.
</p>
<p>
Whether you want to use simple command line utilities like
<code>grep</code>, ship your logs somewhere else, or visualize them
using the many monitoring and visualization tools that exist, it can
only be possible once the logs are centralized. In the next section,
we'll enumerate over some solutions to help developers set up a
centralized logging architecture.
</p>
<h2 id="current-solutions">4 Current Solutions</h2>
<p>
What solutions exist for a small and growing application that wants to
manage its own logs?
</p>
<h3>4.1 Three Main Options</h3>
<div class="img-wrapper">
<img
src="images/three_options.png"
alt="options for centralized logging"
/>
<p>
Three main types of log management solutions
</p>
</div>
<p>
To quickly survey the available options, let's distinguish between
three main categories of log management solutions.
<em>Logging as a Service</em> (LaaS) consists of paid, proprietary,
typically cloud-based systems. The <em>Do it Yourself</em> (DIY) path
involves designing and building a custom, self-hosted solution from a
variety of components. Finally, the
<em>Open-Source Software</em> (OSS) solutions are open-source,
pre-designed centralized logging systems that you deploy on your own
machines.
</p>
<h3>
4.2 LaaS: Full Service Log Management
</h3>
<p>
Full-service Logging as a Service (LaaS) products provide their
customers with the full range of options, including most - if not all
- of the following log management functions: Log Collection, Ingestion
& Storage, Search & Analysis, Monitoring & Alerting,
Visualization & Reporting.
</p>
<p>
LaaS solutions are generally cloud-based. They're designed to be
easy-to-use and they provide a rich feature-set along with technical
support. The most difficult part in working with a LaaS may be
configuring your machine(s) to forward logs into the service. Once the
data is ingested, the LaaS handles the rest. The major advantage of
buying a LaaS product is convenience and ease of setup.
</p>
<p>
The drawbacks however are also apparent. The services aren't free, and
costs can fluctuate based on a number of factors, including the volume
of data being ingested, the number of users, and storage or retention
plans. Nor is it simply an issue of costs. Whether cloud-based or
on-premise, proprietary solutions are marked by a lack of ownership:
users cede control of their data to a third party, and they are unable
to inspect, modify, or audit proprietary code.
</p>
<p>
For some, giving up control of sensitive data is prohibitive, so
another option would be to build your own.
</p>
<h3 id="diy">4.3 DIY</h3>
<p>
Another approach to managing logs is to Do It Yourself (DIY). The DIY
approach allows developers to design and build a system out of
pre-existing components. The key components for a centralized log
management infrastructure are below.
</p>
<ol>
<li>Log Collection</li>
<li>Processing & Aggregation</li>
<li>Ingestion & Storage</li>
<li>Monitoring & Visualization</li>
</ol>
<div class="img-wrapper">
<img
src="images/diagrams/components/components.png"
alt="key components in log management"
/>
<p>
Key components in a log management system
</p>
</div>
<h4 id="collection">4.3.1 Collection</h4>
<p>
Log collection occurs on each machine with Syslog daemons, also known
as agents, listening for messages generated by applications and system
processes. When new events are generated, the agent(s) collects them.
There are alternatives for <code>Syslogd</code>, the standard Syslog
daemon which can collect, process, and transport logs<sup
><a href="#footnote-7">7</a></sup
>.
</p>
<div class="buttons">
<button id="resetSystemBtn" class="cratoButton">Reset</button>
<button id="nextSystemBtn" class="cratoButton">→</button>
</div>
<div class="img-wrapper">
<img
src="images/diagrams/log_collection/log_collection.png"
id="log-collection"
data-id="1"
alt="Log Collection1"
/>
<p>
Logs can be collected and transported from each client machine and
forwarded to a central log server using different systems built on
top of the Syslog daemon.
</p>
</div>
<h5 id="syslog-protocol-its-implementations">
4.3.1.1 Syslog Protocol & its Implementations
</h5>
<p>
Syslog, and its more recent implementations rsyslog and syslog-ng, are
commonly found on <code>*nix</code> systems, and are even available on
Windows. Because of this ubiquity, DIY systems are often able to
directly work with one or another of these syslog implementations for
log collection. Other options exist as well but syslog implementation
is the standard.
</p>
<h4 id="processing-aggregration">4.3.2 Processing & Aggregation</h4>
<div class="img-wrapper">
<img
src="images/diagrams/components/collection_aggregation.png"
alt="log collection and aggregation"
/>
<p>
Once collected from each client, logs are aggregated in a central
location
</p>
</div>
<p>
After collecting logs from each client, the next step is to aggregate
these logs on a single machine. To accomplish this, a logging system
must 'ship' or forward the logs to a centralized server. Here, the
logs may be parsed and transformed into formats like JSON, enriched
with context and stored. There is a healthy amount of options from
which to choose here, and choices will depend largely on application
need.
</p>
<h4 id="ingestion-storage">4.3.3 Ingestion & Storage</h4>
<p></p>
<div class="img-wrapper">
<img
src="images/diagrams/components/zoomed_in/ingestion_storage2.png"
alt="log ingestion"
/>
<p>
Logs ingested into storage
</p>
</div>
<p>
Once logs are collected, the goal is to transport them to a storage
service from which they can be searched and analyzed. If the write
rate of the chosen data store is compatible with potentially bursty
influx of logs into the system, then the logs could be sent directly
to storage. However, not all applications will be able to rely on this
throughput parity. Applications that need to scale, or that expect
wide fluctuations in data flow, would likely do better by decoupling
log aggregation and storage. This can be handled in a number of ways,
and a common option is to use a messaging system. Doing so would
provide a more robust, fault-tolerant system, along with the ability
to handle higher throughput. But it also introduces additional
complexity.
</p>
<p>
Here, as with the other components, there are a number of options to
choose from. The design and implementation choices will depend on a
number of factors. Relevant considerations include the following:
</p>
<ul>
<li>How much data is being ingested?</li>
<li>Who is going to be using the data?</li>
<li>What is the structure of the log?</li>
<li>What search, analysis, and indexing capabilities are needed?</li>
</ul>
<h4 id="monitoring-visualization">
4.3.4 Monitoring & Visualization
</h4>
<p>
Finally, once data is in a database, we can connect it to a
visualization tool for more user friendly analysis and reporting. Some
of these tools also provide alerting and system configuration
capabilities. The visual representation of data allows users to
transform and aggregate in more effective ways. There are many open
source options here, with Kibana and Grafana as the leaders.
</p>
<div class="img-wrapper">
<img
src="images/diagrams/components/visualization.png"
alt="system visualization"
/>
<p>
Visualization tools provide convenient insight into a system's
performance
</p>
</div>
<p>
Ultimately, if you decide to DIY a central log management system, at
each layer you will have several options available.
</p>
<p>
Overall, doing it yourself (DIY) and building a custom system allows
for greater control and flexibility at the cost of complexity. Custom
systems can address very precise needs, and they can provide high
control of data and security. But it requires a dedicated team to
setup, configure, and maintain the system.
</p>
<h3 id="oss">4.4 OSS</h3>
<p>
The final category of log management solutions consists of open source
pre-designed systems. These tools allow a developer to maintain higher
levels of data control with an ease-of-use closer to LaaS tools.
However, they typically require self-management and hosting. The two
main solutions here are the Elastic Stack and Graylog.
</p>
<h4 id="elastic-stack">4.4.1 Elastic Stack</h4>
<div class="img-wrapper">
<img
src="images/diagrams/elastic_stack_diagram.png"
alt="Elastic Stack"
/>
<p>Elastic Stack Architecture</p>
</div>
<p>
Elastic develops a collection of open-source log management products
designed to work well together. Altogether, they have products that
cover each of the four key phases of centralized log management. In
fact, 'ELK' refers to a frequently-used combination of their tools,
comprised of the text-search storage system ElasticSearch, the log
collection and storage service Logstash, and the
visualization/monitoring tool Kibana.
</p>
<h4 id="graylog">4.4.2 Graylog</h4>
<div class="img-wrapper">
<img
src="images/diagrams/graylog_diagram.png"
alt="Graylog architecture"
/>
<p>Graylog architecture</p>
</div>
<p>
Graylog is a company and open-source centralized log management
server. It is built on top of ElasticSearch and MongoDB and offers a
web interface connected to the server. In addition to their OSS
self-hosted option, Graylog provides a LaaS as well.
</p>
<p>
The Elastic Stack and Graylog are very popular open-source self-hosted
solutions. Each offers the expected ease of use, along with the
potential for data ownership and security assurances that come from
self-hosting. There is a rich ecosystem of plugins for each tool, thus
offering extended flexibility. However, these systems are more complex
than LaaS. You'll need to configure, deploy, maintain, and potentially