Implementing SQLite based request queue

* the husk

* setPath fix

* Co-authored-by: ArtursKadikis <[email protected]>

* Reworking storage module API and creating implementation for the memory variant (#111)

* - polymorphous behavior added
- sqlite storge class added
- code is building

* storage modules integration
stable state

* - log added

* storage unit tests added

* unit tests updated

* - virtual destructor
- sqlite storage tests

* files and class names updated

* - storage unit tests improved
- PR Changes
- Code formatting

* Log print updated while adding event.

* cmakelist file updated

* - Abstract methods signatures updated
- Stable state

* - new methods added
- Unit test improved
- stable state

* - Unit tests add/updated
- methods signatures improved
- Fixed potential bugs

* - logic updated

* - Fixed issue with storage moduel
- Unit tests

* - request remove front code refactored

* - Moved to new structure

* -Unit tests fixed
-Updated logic

* memory deallocated

* - Pointer to smart pointer

* - RQpeekFront would return DataENtry with id = -1 and data = ""
- Incremental Id Unit tests added

* - make_shard used
- peekAll and clearAll

* removed logs

* storage module unit tests updated

* - unit test changes,

* - Only unit tests added
Note: Code isn't compiling

* - unit test comments added
- compile errors fixed
- All unit tests are doing fine.

* Code refactor

* PR Changes,

* test changes,

* - missed requested chagnes

* - RQ removed on the basis of id, not ref

* Storage module methods documentation added.

* request processing is functional

* Method comments updated

* - fixed unit tests

* - fix bug
- views test running.

* - fixe a potential bug,

---------

Co-authored-by: Zahid Zafar <>

* View changes added (#113)

* views

* comments

* extracted things (#114)

* bits

* Co-authored-by: ArtursKadikis <[email protected]>

* Adding request queue db variant (#112)

* - Request sql calls added

* - code compile checks
- storage unit tests logic updated

* storage tests reworked

* - sqlite storage unit tests fixed

* - clearSDK
- debugEQState
- Views tests

* - finishing test

* - PR Requested changes
- raw pointer to smart pointer

* Requested changes

* - crash tests fixed
- sessions tests fixed

* - set request queue size
- safe check on configuration values
- new configuration tests

* Request module request threshold unit tests added.

* - safety check for storage modules

* - unit tests added in the storage module
- PR changes
- fixed event schema creation

* Changelog added
SDK Version updated

* PR Changes

* Update views.cpp

* comments and name changes

* Co-authored-by: ArtursKadikis <[email protected]>

* new vacuum

* its a me function

---------

Co-authored-by: Zahid Zafar <>
Co-authored-by: ArtursK <[email protected]>
Co-authored-by: turtledreams <[email protected]>

* Update CHANGELOG.md

* Update CHANGELOG.md

* database

---------

Co-authored-by: Zahid Zafar <[email protected]>
Co-authored-by: ArtursK <[email protected]>

Author: 3 people

CHANGELOG.md

@@ -1,3 +1,8 @@
+22.09.0
+* ! Minor breaking change ! SDK configuration now can't be changed after initialization/start
+* Added a persistent requests queue when building the SDK with the 'COUNTLY_USE_SQLITE' flag
+* Fixed a bug where view's name was being overriden by segmentation provided.
+
 22.06.4
 * Fixed a bug where the SDK 'mutex' was being locked twice when built with the 'COUNTLY_USE_SQLITE' flag.
 

CMakeLists.txt

@@ -36,8 +36,6 @@ set(COUNTLY_PUBLIC_HEADERS
 
   ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/logger_module.hpp
   ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/crash_module.hpp
-  ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/request_module.hpp
-  ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/request_builder.hpp
   ${CMAKE_CURRENT_SOURCE_DIR}/include/countly/views_module.hpp)
 
 add_library(countly
@@ -48,6 +46,8 @@ add_library(countly
   ${CMAKE_CURRENT_SOURCE_DIR}/src/request_module.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/src/crash_module.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/src/request_builder.cpp
+  ${CMAKE_CURRENT_SOURCE_DIR}/src/storage_module_db.cpp
+  ${CMAKE_CURRENT_SOURCE_DIR}/src/storage_module_memory.cpp
   ${CMAKE_CURRENT_SOURCE_DIR}/src/event.cpp)
 
 target_include_directories(countly
@@ -107,8 +107,10 @@ if(COUNTLY_BUILD_TESTS)
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/main.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/views.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/session.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/tests/storage.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/event.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/crash.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/tests/request.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/tests/config.cpp)
 
     target_compile_definitions(countly-tests PRIVATE COUNTLY_BUILD_TESTS)

bin/README.md

@@ -0,0 +1,25 @@
+## Build Scripts
+
+Scripts provided here offers a convenient way to build sample app and the tests at different platforms.
+Still SQLite setup must be done separately at the "/vendor/sqlite" directory if SQLite is needed.
+Current cmake options set in the script that differs from the default are:
+
+- COUNTLY_BUILD_SAMPLE = ON
+- COUNTLY_BUILD_TESTS = ON
+
+To change any other options you can manually modify the scripts.
+
+### Run
+
+You can run the scripts simply by executing them from the root directory.
+
+### Flow
+
+When run the script would:
+
+- Check if a build folder is present at the root and erase it if it exists
+- Run Cmake with mentioned options
+- Build countly-sample app
+- Build countly-test
+
+After this point you can just go to the "/build" directory and execute those build files

bin/mac_build_script.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+if [ "${PWD##*/}" = "build" ]
+then
+    echo "In build folder. Changing directory..."
+    cd ..
+    echo "Current directory: $PWD"
+else
+    echo "Not in build folder. Current directory: $PWD"
+fi
+
+echo "Deleting folder and its subfolders..."
+rm -rf build
+echo "Folder and subfolders deleted."
+
+cmake -DCOUNTLY_BUILD_SAMPLE=1 -DCOUNTLY_BUILD_TESTS=1 -DCOUNTLY_USE_CUSTOM_SHA256=OFF -DCOUNTLY_USE_SQLITE=OFF -B build .
+
+cd build
+make ./countly-sample
+make ./countly-tests

bin/windows_build_script.bat

@@ -0,0 +1,15 @@
+@echo off
+if /i "%CD:~-5%"=="build" (
+    echo In build folder. Changing directory...
+    cd ..
+    echo Current directory: %CD%
+) else (
+    echo Not in build folder. Current directory: %CD%
+)
+echo Deleting folder and its subfolders...
+rmdir /s /q build
+echo Folder and subfolders deleted.
+cmake -DCOUNTLY_BUILD_SAMPLE=1 -DCOUNTLY_BUILD_TESTS=1 -DCOUNTLY_USE_CUSTOM_SHA256=OFF -DCOUNTLY_USE_SQLITE=OFF -B build . -G "Unix Makefiles"
+cd build
+make ./countly-sample
+make ./countly-tests

examples/example_integration.cpp

@@ -34,10 +34,10 @@ int main() {
   cout << "Sample App" << endl;
   Countly &ct = Countly::getInstance();
   ct.alwaysUsePost(true);
+  ct.setLogger(printLog);
+  ct.SetPath("databaseFileName.db"); // this will be only built into account if the correct configurations are set
   ct.setDeviceID("test-device-id");
   // ct.setSalt("test-salt");
-
-  ct.setLogger(printLog);
   // OS, OS_version, device, resolution, carrier, app_version);
   ct.SetMetrics("Windows 10", "10.22", "Mac", "800x600", "Carrier", "1.0");
   // Server and port

include/countly.hpp

@@ -22,17 +22,20 @@
 #endif
 #include "countly/event.hpp"
 #include "countly/logger_module.hpp"
+#include "countly/storage_module_base.hpp"
 #include "countly/views_module.hpp"
+#include <countly/crash_module.hpp>
 #include <countly/request_builder.hpp>
 #include <countly/request_module.hpp>
-#include <countly/crash_module.hpp>
 
 namespace cly {
 class Countly : public cly::CountlyDelegates {
 public:
   Countly();
 
   ~Countly();
+
+  // Returns the singleton instance of Countly
   static Countly &getInstance();
 
   // Do not implicitly generate the copy constructor, this is a singleton.
@@ -43,6 +46,8 @@ class Countly : public cly::CountlyDelegates {
 
   void alwaysUsePost(bool value);
 
+  void setMaxRequestQueueSize(unsigned int requestQueueSize);
+
   void setSalt(const std::string &value);
 
   void setLogger(void (*fun)(LogLevel level, const std::string &message));
@@ -93,6 +98,8 @@ class Countly : public cly::CountlyDelegates {
 
   void addEvent(const cly::Event &event);
 
+  void addEventToSqlite(const cly::Event &event);
+
   void setMaxEvents(size_t value);
 
   void flushEvents(std::chrono::seconds timeout = std::chrono::seconds(30));
@@ -179,7 +186,14 @@ class Countly : public cly::CountlyDelegates {
   }
 
   /* Provide 'updateInterval' in seconds. */
-  inline void setAutomaticSessionUpdateInterval(unsigned short updateInterval) { configuration->sessionDuration = updateInterval; }
+  inline void setAutomaticSessionUpdateInterval(unsigned short updateInterval) {
+    if (is_sdk_initialized) {
+      log(LogLevel::WARNING, "[Countly][setAutomaticSessionUpdateInterval] You can not set the session duration after SDK initialization.");
+      return;
+    }
+
+    configuration->sessionDuration = updateInterval;
+  }
 
 #ifdef COUNTLY_BUILD_TESTS
   /**
@@ -188,15 +202,7 @@ class Countly : public cly::CountlyDelegates {
    * You should not be using this method.
    * @return a vector object containing events.
    */
-  const std::vector<std::string> debugReturnStateOfEQ() {
-
-#ifdef COUNTLY_USE_SQLITE
-    return {};
-#else
-    std::vector<std::string> v(event_queue.begin(), event_queue.end());
-    return v;
-#endif
-  }
+  std::vector<std::string> debugReturnStateOfEQ();
 
   /**
    * This function should not be used as it will be removed in a future release.
@@ -229,6 +235,9 @@ class Countly : public cly::CountlyDelegates {
   void _deleteThread();
   void _sendIndependantLocationRequest();
   void log(LogLevel level, const std::string &message);
+#ifdef COUNTLY_USE_SQLITE
+  bool createEventTableSchema();
+#endif
 
   /**
    * Helper methods to fetch remote config from the server.
@@ -257,11 +266,12 @@ class Countly : public cly::CountlyDelegates {
   std::unique_ptr<std::thread> thread;
   std::unique_ptr<cly::CrashModule> crash_module;
   std::unique_ptr<cly::ViewsModule> views_module;
+
   std::shared_ptr<cly::CountlyConfiguration> configuration;
   std::shared_ptr<cly::LoggerModule> logger;
-
   std::shared_ptr<cly::RequestBuilder> requestBuilder;
   std::shared_ptr<cly::RequestModule> requestModule;
+  std::shared_ptr<cly::StorageModuleBase> storageModule;
   std::shared_ptr<std::mutex> mutex = std::make_shared<std::mutex>();
 
   bool is_queue_being_processed = false;

include/countly/constants.hpp

@@ -1,19 +1,19 @@
 #ifndef COUNTLY_CONSTANTS_HPP_
 #define COUNTLY_CONSTANTS_HPP_
 
+#include "nlohmann/json.hpp"
 #include <cassert>
 #include <chrono>
+#include <climits>
 #include <functional>
 #include <map>
 #include <memory>
 #include <random>
 #include <sstream>
 #include <string>
-#include <climits>
-#include "nlohmann/json.hpp"
 
 #define COUNTLY_SDK_NAME "cpp-native-unknown"
-#define COUNTLY_SDK_VERSION "22.06.4"
+#define COUNTLY_SDK_VERSION "22.09.0"
 #define COUNTLY_POST_THRESHOLD 2000
 #define COUNTLY_KEEPALIVE_INTERVAL 3000
 #define COUNTLY_MAX_EVENTS_DEFAULT 200
@@ -31,7 +31,7 @@ const std::default_random_engine generator(std::chrono::system_clock::now().time
 const std::uniform_int_distribution<int> distribution(1, INT_MAX);
 
 /**
- * Convert map into a string.
+ * Formats the given arguments into a string buffer.
  *
  * @param format formatting string
  * @param args arguments to be formatted
@@ -49,7 +49,7 @@ template <typename... Args> static std::string format_string(const std::string &
 }
 
 /**
- * Convert map into a string.
+ * Gives a string representation of the size of a map.
  *
  * @param m a map containing key-value pairs
  * @return a string object holding size of map.

include/countly/countly_configuration.hpp

@@ -27,6 +27,13 @@ struct CountlyConfiguration {
    */
   std::string salt;
 
+  /**
+   * Path to the database.
+   */
+#ifdef COUNTLY_USE_SQLITE
+  std::string databasePath;
+#endif
+
   /**
    * Sets the interval for the automatic update calls
    * min value 1 (1 second), max value 600 (10 minutes)

include/countly/request_module.hpp

@@ -8,13 +8,14 @@
 #include "countly/countly_configuration.hpp"
 #include "countly/logger_module.hpp"
 #include "countly/request_builder.hpp"
+#include "countly/storage_module_base.hpp"
 
 namespace cly {
 class RequestModule {
 
 public:
   ~RequestModule();
-  RequestModule(std::shared_ptr<CountlyConfiguration> config, std::shared_ptr<LoggerModule> logger, std::shared_ptr<RequestBuilder> requestBuilder);
+  RequestModule(std::shared_ptr<CountlyConfiguration> config, std::shared_ptr<LoggerModule> logger, std::shared_ptr<RequestBuilder> requestBuilder, std::shared_ptr<StorageModuleBase> storageModule);
 
   HTTPResponse sendHTTP(std::string path, std::string data);