-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathChangelog
145 lines (137 loc) · 13.2 KB
/
Changelog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
v1.3.2 named "named "Katherine Johnson" (Dec 18, 2023)
This is a minor version upgrade release:
Imrpovements/upgrades/compatibility issues:
- Upgrade of POFR Perl release from version 5.38.0 to version 5.38.2 addressing all security and bug issues of PERL.
- Upgrade of scponly for RHEL 9 and derivaties from version 4.8.29 to version 4.8.32.
- Upgrade of the IP2LOCATION LITE DB to December 2023 version.
- Added the getsshfailedattempts.pl tool to help admins/devops handle SSH brute force attacks and cross reference data from the OS journal with data collected from the relational layer.
v1.3.1 named "Grace Hopper" (Aug 20, 2023):
This is a minor version upgrade release:
-Bug fixes:
- newdeltaparseproc.pl: Generating the file shasum before or after quoting creates inconsistencies in the shasum (same file access can have two shasum digests). The correct behavior is to generate the shasum BEFORE quoting.
So, the correct sequence of actions is sanitize the filename, then generate the sha_hex and then quote the filename, prior SQL inserting it. All differences/inconsistencies have been corrected now.
- scanproc.pl and tuneperf.pl: Fix a bug where whitespace in a filename breaks the filename into multiple RDBMS entries.(the client code generates the bug).
-Improvements/upgrades/compatibility issues:
- newdeltaparseproc.pl and POFR.pm: Implemented and improved the delta parsing for network endpoint data (processnetfile subroutine) to achieve delta parse performance improvements on systems with high network endpoint activity.
- Upgraded POFR Perl release (v5.36.0->5.38.0)
- Updated the IP2LOCATION LITE DB to the August 2023 release.
v1.2.3 named "Mary Jackson" (Apr 8, 2023):
This is a bug fix release:
-Bug fixes:
- newdeltaparseproc.pl: Bug fix/enhancement: If the server IP is not DNS resolvable, get it via Sys::Hostname.
Backported from the makenetworkdelta branch.
- newdeltaparseproc.pl: Fix the thread timing with thread number 10. It is supposed to sleep for 225 seconds, instead it was sleeping for 22.5 seconds,
so it was firing up early. This could skew data processing.
- pofrclientderegister.pl: Corrected regular expression to ensure we do not delete the pofr client utilities when we remove a system.
-Impovements/upgrades/compatibility issues:
- Updated the IP2LOCATION LITE DB to the April 2023 release.
Due to the severity of some of these bugs, all users are advised to immediately update to v.1.2.3.
v1.2.2 named "Mary Jackson" (Mar 17, 2023):
This is a bug fix and enhancement release:
-Bug fixes:
- newdeltaparseproc.pl: Corrected runtime errors from unmodified statements as a result of the modifications to include both real and effective uid/gids.
- scanproc.pl: Removed all the kernel version dependency code that was parsing on the basis of the order of fields in /proc/[pid]/status.
Instead we read the entire file to a hash now, and we lookup the fields as hash keys. This makes the parsing independent from the order of fields,
so any changes in procfs are less likely to affect us in the future.
- mergearchive.pl: Fixed an SQL INSERT statement field that was preventing the tool from functioning properly.
- mergearchive.pl: Fixed the dispusage subroutine. The switch is --tspec, no longer --datespec.
- sendproc.pl: Made the detectandsendleftovers() subroutine to ensure that we send leftover tarballs from bad connections as part of the send loop and not only
when we start sendproc.pl. This fixes the issue of sending tarballs out of sequence and leaving them over hanging there until scripts are restarted.
- mergetables.pl and mergearchive.pl: Important fix that produced a runtime crash, due to lack of quoting. We were quoting the wrong field. The field that
needs quooting is the arguments field. That is now fixed and tested.
-Impovements/upgrades/compatibility issues:
- RHEL/AlmaLinux/RockyLinux version 9 is now a supported OS option for a POFR server.
- Incoprorated the scponly package as part of POFR: https://github.com/gmagklaras/POFR/pull/4
- Replaced all luarm -> pofr file extensions.
- newdeltaparseproc.pl and scanproc.pl: Change the no open files field flag from LUARMv2NOOPENFILES to POFRv1NOOPENFILES.
- tuneperf.pl: Added the tuneperf.pl script for tuning the procfs sampling frequency on the client target systems.
- Improved the POFR manual by adding section F5 on target system tuning techniques.
Because of these changes, all users are advised to unregister and re-register clients to the server after upgrading to version v1.2.2.
v1.2.1 named "Hedy Lamarr" (Oct 23 2022):
This is a bug fix release:
-Bug fixes:
https://github.com/gmagklaras/POFR/pull/2
-Fix timing issues by:
-Re-introducing into the code subroutine timestamp() which times the data from the POFR tarball epoch and tz data.
-Differentiating between the timestamp() and dbtimestamp(). The latter just takes a timestamp from the RDBMS, for example to time a registration
-Re-ordering the data into "ORDER BY" SQL statements to avoid a bug where the psinfo/fileinfo/netinfo tables contain data that spans multiple days (say two consecutive days), so the "ORDER BY chour,cmin,csec,cmsec" will place the data of the latter day first. Consequently, these SQL ORDER BY statements should instead read "ORDER BY cyear,cmonth,cday,chour,cmin,csec,cmsec" and
"ORDER BY cyear DESC,cmonth DESC,cday DESC,chour DESC,cmin DESC,csec DESC,cmsec DESC" to find the latest data instead.
Due to the severity of this bug that affects idle/less loaded systems, all users are advised to switch immediately to release v1.2.1.
v1.2.0 named "Hedy Lamarr" (Oct 1 2022):
- Bug fixes:
- mergetables.pl: Improved the data date range sampling loop (speed and accuracy) of the architables subroutine, by ordering the fields with the ORDER BY
statement. psentity is NOT unique in the psinfo table. (info fed from MERGE tables), so it can never be used to sort out tables. The same in ordering
the fields before we extract them to the CSV file. (https://github.com/gmagklaras/POFR/commit/701e67a816cc2971be2c08f5c7968bef944aa9d2)
- POFR.pm: Corrections on the fact we were not returning array references from the get_requested_data_from_time_range subroutine and that was creating
problems when we wanted to call the function. Fixed now. (https://github.com/gmagklaras/POFR/commit/c853cef81d73e671dc0a038d887fa49e843683e2)
- Security issues:
- All security bugs addressed in the Perl Delta for release 5.36.0 Delta. (https://perldoc.perl.org/5.36.0/perldelta) (see also
Improvements/upgrades/compatibility issues)
- Impovements/upgrades/compatibility issues:
- sendproc.pl: Increased the persistence of the client on failed connections from 3 min to 4 min. (https://github.com/gmagklaras/POFR/commit/9f46f0862407a34e772832a1a5de07671e4ac32b)
- producepofrperl: Added the DateTime::Format::Duration CPAN module, necessary for date conversions. (https://github.com/gmagklaras/POFR/commit/1eb31cbd78de9863193f53971522c2729d0ee82f)
- Added the utility scripts:
findrange.pl: A collection of routines to create subsets of the archived data. Not yet in use in production, only for testing:
(https://github.com/gmagklaras/POFR/commit/33edeb53f2e07c3d138d29bd7d7be2f9f098cfaf)
(https://github.com/gmagklaras/POFR/commit/692f5c2068940c2325ab53f1890e0ee8e011e5f3)
(https://github.com/gmagklaras/POFR/commit/6de984cfdba0520b786231d44c294de9073afa05)
(https://github.com/gmagklaras/POFR/commit/5b8e17bc2e3da4ae0a5608537706dcfd0c4553be)
- Big refactoring of code by moving often used subroutines to the POFR.pm:
(https://github.com/gmagklaras/POFR/commit/ce0c8f0b39e16862a550114c9d712a68c9d961fd)
(https://github.com/gmagklaras/POFR/commit/d3323ddfc196dbb7c8502aca78da5e264a39aedd)
(https://github.com/gmagklaras/POFR/commit/4ffbbd752ee040cda5530c06ee0f2d74c47a51c4)
(https://github.com/gmagklaras/POFR/commit/c3bb8f4a9c2c4c9983660258775308738e4fb72c)
(https://github.com/gmagklaras/POFR/commit/6af7155550850103143591b1d37f4a73644c1809)
(https://github.com/gmagklaras/POFR/commit/adc4c9406e06530886d1faa400e03daf00284ab7)
(https://github.com/gmagklaras/POFR/commit/7552c4e34a0347d4c199c7370aa21f6833bf11c5)
(https://github.com/gmagklaras/POFR/commit/519a043fd51eb1a26f7198d4f443144544ef3d35)
(https://github.com/gmagklaras/POFR/commit/f8e2fb805c7405bee169cb6672a76a4b8a479acb)
(https://github.com/gmagklaras/POFR/commit/24a0d72b6df58df7b8119ab0ddc22d373ffd3c84)
(https://github.com/gmagklaras/POFR/commit/bac50f260b9899b985292e6ae7a85fc2ef8cc56f)
(https://github.com/gmagklaras/POFR/commit/a9f415163300152ec70a599bce8d701156c9bd75)
(https://github.com/gmagklaras/POFR/commit/88586c847ffaaab334a7c347d3d16f9ff55c55e4)
- mergearchive.pl: Re-write to produce time subsets with the get_requested_data_from_time_range from POFR.pm. Also changed the command line argument
structure to specify date and time ranges. (https://github.com/gmagklaras/POFR/commit/cbcd655f34e549aa19728a23a90a4c3a34cd54cf)
- mergearchive.pl: Corrected the dispusage subroutine. (https://github.com/gmagklaras/POFR/commit/d8cf97fe562dd4a3fdaf6f469af1a5c8e7a8cb3a)
- Upgrade of the POFR Perl release from Perl 5.34.0/5.34.1 to Perl 5.36.0:
(https://github.com/gmagklaras/POFR/commit/73919c24346422b7a91b3a39a963ca649a90e41c)
(https://github.com/gmagklaras/POFR/commit/d473be29044833791f86e1ba32336cc3854e3356)
(https://github.com/gmagklaras/POFR/commit/8c45139dc0d28074df8cca65e9509d5c0dc95871)
(https://github.com/gmagklaras/POFR/commit/9f1b7d0303a3873e45fbdfdad00fe3c3ecd2ebde)
(https://github.com/gmagklaras/POFR/commit/c909b9da87e8c2a0cc8c9177a8b2bbe915c31d3e)
(https://github.com/gmagklaras/POFR/commit/f7f21dec06b48b6bb144e3c20688d7f798fa62e6)
(https://github.com/gmagklaras/POFR/commit/9157254bf4f63137bcec9d64cb9eb38ad70be7a9)
- Added the POFR Perl Release 5.36.0 for RHEL9 and derivatives. (https://github.com/gmagklaras/POFR/commit/253afe118231d08fe12ee5df7c1fb2cb7b2537fa)
- POFRmanual.pdf: Updated versions of distros in the compatibility section, added suggection of client logrotation config, revised the temporal assembly
and data browsing text of section F4, added section G to outline the ethical considerations for POFR usage.
(https://github.com/gmagklaras/POFR/commit/e7f59cb45e9794091f5c95ed7092fb2093607b61)
- Updated the IP2LOCATION-LITE DB to the October 2022 release. (https://github.com/gmagklaras/POFR/commit/61984e4afc72b2a17f92b7335479ad8395bd8aff)
v1.1.0 named "Lorinda Cherry" (Feb 27 2022):
- Bug fixes:
- mergetables.pl: Removed a loop where we waiting from thread 1 for the other threads to finish to call mergetables.pl. Mergetables.pl is called at the beginnig of the
processing cycle. (https://github.com/gmagklaras/POFR/commit/aa7691f4fd6169571fa832e6cfef04c6cbbddb32)
- pofrclientunregister.pl: Fixed issue with the script erroneously checking for server user registration directories by removing left over sanity check code from the
server part. (https://github.com/gmagklaras/POFR/commit/8a8233d61d66939e0a0bef204302df6783e157be)
- Security issues:
- pofrsreg.pl: Possible security issue: Remove the display from status messages of the variable. Not safe and not necessary.
(https://github.com/gmagklaras/POFR/commit/86306a72de8438bea213d96da8a54b791a58c171
- Impovements/upgrades/compatibility issues:
- Added the utility scripts:
pofrclientderegister.pl (client side: https://github.com/gmagklaras/POFR/commit/b560ca51d1841a386a055715e464538df0611f64)
pofrcleanreg.pl (server side: https://github.com/gmagklaras/POFR/commit/a24c6e0f8dfce99760b38f31bec3460b335437bc)
to automate/aid the deregistration of systems from the POFR solution
- Various improvements of the POFR Technical Operations and User Manual (https://github.com/gmagklaras/POFR/commit/69d422288f78ad2d9ac7106892ece60620dfa340)
- pofrsreg.pl: Improved/refactored the STATUS, Warning and Error messages. (pofrsreg.pl: Improved/refactored the STATUS, Warning and Error messages.)
- pofrperlrhelcentos8x86_64.tar.gz: Added a POFR Perl release 5.34.0 for the RHEL/ALMA/Rocky/CentOS 8 ecosystem.
(https://github.com/gmagklaras/POFR/commit/75cbce1b26c4ede08ba55277ec21a89835427a53)
- LHLT.sql: Added a client hostname field. However, this creates incompatibility of the 'lhlt.lhltable' table with previous versions/releases.
To avoid re-registering clients on the server side and constructing the table from scratch, the easiest thing to do is to:
-a)Wait for all POFR server threads to finish processing.
-b)Connect to the POFR server MariaDB as administrator as described in section E of the POFR Technical Operations and User Manual and issue the following:
use lhlt;
ALTER TABLE lhltable ADD hostname varchar(130) NOT NULL AFTER lastip;
These commands should allow you to alter the table on the fly. You can then continue processing data with currently registered systems with no issues.
v1.0.0 named "Nicole-Reine Lepaute" (Feb 05 2022):
- Bug fixes: Race errors halting execution of parse cycles when calling the mergetables.pl and mergearchives.pl scripts are now fixed.
- Feature implementation: GeoIP2 locating IPv4 and IPV6 destination IP addresses has been implemented by incorporating the IP2Location Lite databases.
- Rebase of PERL to the latest stable version 5.34.0