v2.1.6

• [2.1.6] repository name change - ApacheLogs2MySQL to apache-logs-to-mysql
• [2.1.6] rename files - apachelogs2MySQL.py to logs2mysql.py, apachelogs2MySQL.sql to apache_logs_schema.sql
• [2.1.6] modify logs2mysql.py line if useragent_process == 1: to if useragent_process >= 1:
• [2.1.6] modify all files with refers to repository name. Changed ApacheLogs2MySQL to apache-logs-to-mysql
• [2.1.6] "application name" is still referred to as ApacheLogs2MySQL in README.md, CITATION.md, logs2mysql.py, watch4logs.py, apache_logs_schema.sql, INSTALL.md and settings.env

v2.1.5

• [2.1.5] move platformNode COLUMN from import_client TABLE to import_device TABLE. This attribute should not change once set. import_client are attributes which may change.
• [2.1.5] modify U_import_file_name INDEX on import_file TABLE from UNIQUE (name) to UNIQUE (importdeviceid, name). This was missed in last release and main reason for release.

v2.1.4

• [2.1.4] add import_device TABLE to separate import_client TABLE columns to resolve software login, IP address and software version change information creating new record.
• [2.1.4] add importdeviceid COLUMN to import_file TABLE to resolve Client Modules with identical FILE NAMES and PATHS. Application detected file already imported.
• [2.1.4] modify import_client and import_device COLUMN widths and created UNIQUE INDEX for combined columns for each table.
• [2.1.4] modify importfileexists and importfileID FUNCTIONS with new parameter for importdeviceid.

v2.1.3

• [2.1.3] modify access_log and error_log TABLES reordering COLUMNS to improve database design readability.
• [2.1.3] modify normalize_useragent to removed first parameter restriction. Any string 8 characters are more can be passed.
• [2.1.3] modify apacheLogs2MySQL.py add completed COLUMN to UPDATE statement to fix processing process_access_import and process_error_import with ALL parameter.
• [2.1.3] modify call_processes.sql adding more comments to better describe options and parameters and overall processing.

v2.1.2

• [2.1.2] modify TABLE access_log ADD CONSTRAINT F_access_requestlogid FOREIGN KEY (requestlogid) REFERENCES log_requestlogid(id)
• [2.1.2] modify TABLE error_log ADD index CONSTRAINT F_error_requestlogid FOREIGN KEY (requestlogid) REFERENCES log_requestlogid(id)
• [2.1.2] add TABLE import_format for reporting and filtering features. Allows adjusted File Format names in tables rather than hardcoding. Also enables addtional formats,
• [2.1.2] add COLUMN importformatid to TABLE import_file for reporting and filtering features. This is FOREIGN KEY to import_format TABLE.
• [2.1.2] modify process_access_parse and process_error_parse - WHERE CLAUSE for parameter ALL to select ONLY completed LOAD processes.
• [2.1.2] modify importFileID STORED FUNCTION to add fileformat PARAMETER.
• [2.1.2] modify apacheLog2MySQL.py to pass importFileID parameters 2=common and combined OR 5=error_default and error_vhost. Does not determine format difference here.
• [2.1.2] modify process_access_parse and process_error_parse - to SET 'Import File Format - 1=common,2=combined,3=vhost,4=csv2mysql,5=error_default,6=error_vhost'
• [2.1.2] modify process_access_parse and process_error_parse - WHERE CLAUSE for parameter ALL to select ONLY completed LOAD processes.
• [2.1.2] reformatted SQL statements in all 66 schema views for code standardization in SQL files used to create apacheLogs2MySQL.sql
• [2.1.2] modify all 11 access_ua_ views SQL statements FROM apache_logs.access_log_ua ln INNER JOIN apache_logs.access_log_useragent lua INNER JOIN apache_logs.access_log
• [2.1.2] created new entity_relationship_diagram.png to reflect database changes.

2.1.1

• [2.1.1] rename COLUMN timeStamp to logged in TABLES access_log and error_log.
• [2.1.1] add access_log INDEXES I_access_log_logged and I_access_log_servernameid_logged.
• [2.1.1] add error_log INDEXES I_error_log_logged and I_error_log_servernameid_logged.
• [2.1.1] modify process_access_import and process_error_imort for COLUMN rename timeStamp to logged in TABLES access_log and error_log.
• [2.1.1] add access_log views access_period_year_list, access_period_month_list, access_period_week_list, access_period_day_list, access_period_hour_list.
• [2.1.1] add error_log views error_period_year_list, error_period_month_list, error_period_week_list, error_period_day_list, error_period_hour_list.

Request log ID functionality

Format Strings added to LogFormat and ErrorLogFormat for Reporting to definitively associate Error record to Access record.

ErrorLogFormat adds %L - Log ID of the request
LogFormat adds %L - The request log ID from the error log (or '-' if nothing has been logged to the error log for this request). Look for the matching error log line to see what request caused what error.

Requires Apache mod_unique_id module be loaded. Log ID only generated when error occurs. Not when request occurs. Copied from link:

/*
* There are two ways the generation of a unique id can be triggered:
 *
 * - from the post_read_request hook which calls set_unique_id()
 * - from error logging via the generate_log_id hook which calls
 *   generate_log_id(). This may happen before or after set_unique_id()
 *   has been called, or not at all.
 */ 

https://github.com/omnigroup/Apache/blob/master/httpd/modules/metadata/mod_unique_id.c

Comprehensive Update

Version 2.0.0 fixes issues encountered running 4 weeks on 7 VPS with 2 to 6 VirtualHosts on each VPS. Each VPS is running watch4logs.py in PM2 connecting to one MySQL server apache_logs schema. Application currently consolidates Access and Error logs from 32 domains.

Extended to Csv2MySQL

The keyword 'extended' was used at project inception to represent a custom LogFormat and was propagated throughout the application. Concern was raised the use of this keyword could cause confusion with 'Extended Log Format'. Extended Log Format (ELF) is a standardized text file format that is used by web servers when generating log files. https://en.wikipedia.org/wiki/Extended_Log_Format

The decision was made to make this change now since table and column names are affected. The new keyword 'csv2mysql' is a more appropriate representation of the LogFormat since the format uses commas. CSV (comma separated values) 2 MySQL.

Extended Normalization

Created additional tables to normalize UserAgent data into 11 tables and 11 views. Added req_query column to separate URI and Query String data. Cleaned up and added a bunch of improvements in last 5 days after being completely away from project for week. Also renamed LOAD DATA tables to create cleaner readability of schema.