xAPI 2.0 Support (#477)

Author: milt

Makefile

@@ -2,7 +2,7 @@
 
 # Version of LRS Admin UI to use
 
-LRS_ADMIN_UI_VERSION ?= v0.2.5
+LRS_ADMIN_UI_VERSION ?= v0.2.6
 LRS_ADMIN_UI_LOCATION ?= https://github.com/yetanalytics/lrs-admin-ui/releases/download/${LRS_ADMIN_UI_VERSION}/lrs-admin-ui.zip
 LRS_ADMIN_ZIPFILE ?= lrs-admin-ui-${LRS_ADMIN_UI_VERSION}.zip
 

README.md

@@ -52,6 +52,7 @@ For releases and release notes, see the [Releases](https://github.com/yetanalyti
 - [Reactions](doc/reactions.md)
 - [JSON Spec](doc/reactions/spec.md)
 - [Sending xAPI statement(s) with Postman](doc/postman.md)
+- [xAPI Versioning](doc/xapi_versioning.md)
 
 ### Demos
 

deps.edn

@@ -53,12 +53,12 @@
   xyz.capybara/clamav-client                    {:mvn/version "2.1.2"}
   ;; Yet Analytics deps
   com.yetanalytics/lrs
-  {:mvn/version "1.3.1"
+  {:mvn/version "1.4.0"
    :exclusions  [org.clojure/clojure
                  org.clojure/clojurescript
                  com.yetanalytics/xapi-schema]}
   com.yetanalytics/xapi-schema
-  {:mvn/version "1.4.0"
+  {:mvn/version "1.4.1"
    :exclusions  [org.clojure/clojure
                  org.clojure/clojurescript]}
   com.yetanalytics/colossal-squuid
@@ -72,7 +72,7 @@
    :exclusions  [org.clojure/clojure
                  buddy/buddy-sign]}
   com.yetanalytics/lrs-reactions
-  {:mvn/version "0.0.1"
+  {:mvn/version "0.0.3"
    :exclusions  [org.clojure/clojure]}
   com.yetanalytics/gen-openapi
   {:mvn/version "0.0.4"
@@ -136,7 +136,7 @@
                   :git/sha "3f288f1f16d167723ad87cc35b1dfee3c1681e10"}
                  com.yetanalytics.lrs/test-runner
                  {:git/url    "https://github.com/yetanalytics/lrs-test-runner.git"
-                  :git/sha    "8bd5be7816288e85f5c07fc11bf8cf53667e72da"
+                  :git/sha    "1891bb5c8802a397ffaee86d336ee95d71cc1a4d"
                   :exclusions [org.clojure/data.json]}
                  com.yetanalytics/datasim
                  {:mvn/version "0.4.4"

doc/env_vars.md

@@ -110,6 +110,9 @@ The following options are used for advanced database performance tuning and may
 | `LRSQL_STMT_RETRY_BUDGET`       | `stmtRetryBudget`       | The max amount of time allowed for statement POST transaction retries before failing (ms).                                                                                                                                               | `1000`                                                                                |
 | `LRSQL_ENABLE_REACTIONS`        | `enableReactions`       | Whether or not to enable statement reactions.                                                                                                                                                                                            | `false`                                                                               |
 | `LRSQL_REACTION_BUFFER_SIZE`    | `reactionBufferSize`    | Number of pending reactions to allow. Additional reactions will be dropped with a warning message.                                                                                                                                       | `10000`                                                                               |
+| `LRSQL_REACTION_VERSION`        | `reactionVersion`       | The xAPI version used when the reactor emits statements. Must be either `1.0.3` or `2.0.0`.                                                                                                                                             | `1.0.3`                                                                               |
+| `LRSQL_SUPPORTED_VERSIONS`      | `supportedVersions`     | A comma-separated list of supported xAPI versions. This is used to validate incoming requests and ensure compatibility.                                                                                                                  | `1.0.3,2.0.0`                                                                          |
+| `LRSQL_ENABLE_STRICT_VERSION`   | `enableStrictVersion`   | Whether or not to enable strict versioning. When enabled, GET requests for statements using a 1.0.3 header will convert 2.0.0 statements by removing or normalizing fields.                                                                 | `false`                                                                               |
 
 _NOTE:_ `LRSQL_STMT_RETRY_LIMIT` and `LRSQL_STMT_RETRY_BUDGET` are used to mitigate a rare scenario where specific Actors or Activities are updated many times in large concurrent batches. In this situation the DBMS can encounter locking and these settings are used to allow retries that eventually write all the conflicting transactions, but may incur performance degradation. If you are experiencing this situation the first step would be to look at why your data needs to rewrite specific Actors or Activities rapidly with different values, which could potentially solve it at the source. If the issue cannot be avoided by data design alone, another possible solution is reducing batch sizes to decrease or eliminate locks. As a last resort, increasing these settings will at least ensure the statements get written but as mentioned may incur a slowdown in concurrent throughput.
 

doc/index.md

@@ -32,6 +32,7 @@
 - [Example AWS Deployment](aws.md)
 - [Reactions](reactions.md)
 - [Sending xAPI statement(s) with Postman](postman.md)
+- [xAPI Versioning](xapi_versioning.md)
 
 ### Demos
 

doc/xapi_versioning.md

@@ -0,0 +1,15 @@
+[<- Back to Index](index.md)
+
+# xAPI Versioning
+
+SQL LRS supports xAPI versions 1.0.3 and 2.0.0. The version used is determined by the `X-Experience-API-Version` header in the request. Use the LRSQL_SUPPORTED_VERSIONS environment variable or the `supportedVersions` LRS configuration property to set which versions are supported. By default, both versions are supported. For more information see [the configuration documentation](env_vars.md).
+
+## Strict Versioning
+
+Note that by default, responses to requests with an `X-Experience-API-Version` header set to `1.0.3` may contain statements in `2.0.0` format. To ensure that `2.0.0` statements are downgraded to `1.0.3` format use the `LRSQL_ENABLE_STRICT_VERSION` environment variable or the `enableStrictVersion` LRS configuration property. For more information see [the configuration documentation](env_vars.md).
+
+## Reaction Versioning
+
+Reactions generate version `1.0.3` statements by default but you can configure this by setting the `LRSQL_REACTION_VERSION` environment variable or the `reactionVersion` LRS configuration property. For more information see [the configuration documentation](env_vars.md). Note that creating reactions under `2.0.0` and then restricting the LRS to version `1.0.3` only will lead to errors in the LRS Admin UI frontend. To recover from this situation reenable `2.0.0` support and delete any incompatible reactions before returning to `1.0.3` only.
+
+[<- Back to Index](index.md)

resources/lrsql/config/prod/default/lrs.edn

@@ -13,4 +13,7 @@
  :stmt-retry-limit        #or [#env LRSQL_STMT_RETRY_LIMIT 10]
  :stmt-retry-budget       #or [#env LRSQL_STMT_RETRY_BUDGET 1000]
  :enable-reactions        #boolean #or [#env LRSQL_ENABLE_REACTIONS false]
- :reaction-buffer-size    #long #or [#env LRSQL_REACTION_BUFFER_SIZE 10000]}
+ :reaction-buffer-size    #long #or [#env LRSQL_REACTION_BUFFER_SIZE 10000]
+ :reaction-version        #or [#env LRSQL_REACTION_VERSION "1.0.3"]
+ :supported-versions      #or [#env LRSQL_SUPPORTED_VERSIONS "1.0.3,2.0.0"]
+ :enable-strict-version   #boolean #or [#env LRSQL_ENABLE_STRICT_VERSION false]}

resources/lrsql/config/test/default/lrs.edn

@@ -13,4 +13,7 @@
  :stmt-retry-limit        20
  :stmt-retry-budget       10000
  :enable-reactions        true
- :reaction-buffer-size    10000}
+ :reaction-buffer-size    10000
+ :reaction-version        "1.0.3"
+ :supported-versions      "1.0.3,2.0.0"
+ :enable-strict-version   false}

src/db/mariadb/lrsql/mariadb/record.clj

@@ -68,11 +68,10 @@
     (create-credential-table! tx)
     (create-credential-to-scope-table! tx)
     (create-blocked-jwt-table! tx))
-  (-update-all! [_ _tx]
-    ; MariaDB is the latest database to get a SQl-LRS implementation, so no migrations are needed.
-    ; In the future migrations will go here.
-    )
-  
+  (-update-all! [_ tx]
+    (when-not (= 1 (:has_context_agent (query-statement-to-actor-usage-enum-has-context-actors tx)))
+      (alter-statement-to-actor-usage-enum-add-context-actors! tx)))
+
   bp/BackendUtil
   (-txn-retry? [_ ex]
     (and (instance? java.sql.SQLException ex)
@@ -201,7 +200,7 @@
   ;;;; So we don't actually store the JWT; we just store a (shorter) hash of it.  
   ;;;; Operations that don't interact with the jwt obviously don't need to hash it.
   ;;;;
-  
+
   bp/JWTBlocklistBackend
   (-insert-blocked-jwt! [_ tx input]
     (insert-blocked-jwt! tx (update input :jwt md/sha256-base64)))

src/db/mariadb/lrsql/mariadb/sql/ddl.sql

@@ -273,3 +273,34 @@ CREATE TABLE IF NOT EXISTS blocked_jwt (
   one_time_id CHAR(36) UNIQUE,
   KEY blocked_jwt_evict_time_idx (evict_time)
 );
+
+/* Migration 2025-09-26 - Add ContextAgent, ContextGroup, SubContextAgent, SubContextGroup to statement_to_actor usage enum */
+
+-- :name query-statement-to-actor-usage-enum-has-context-actors
+-- :command :query
+-- :result :one
+-- :doc Query to see if the `statement_to_actor.usage` column has ContextAgent in its enum.
+SELECT EXISTS (
+    SELECT 1
+    FROM INFORMATION_SCHEMA.COLUMNS
+    WHERE TABLE_NAME = 'statement_to_actor'
+      AND COLUMN_NAME = 'usage'
+      AND FIND_IN_SET('ContextAgent', REPLACE(REPLACE(COLUMN_TYPE, 'enum(', ''), ')', '')) > 0
+) AS has_context_agent;
+
+-- :name alter-statement-to-actor-usage-enum-add-context-actors!
+-- :command :execute
+-- :doc Change the enum datatype of the `statement_to_actor.usage` column to add ContextAgent, ContextGroup, SubContextAgent, and SubContextGroup.
+ALTER TABLE statement_to_actor MODIFY COLUMN `usage` ENUM ('Actor',
+                     'Object',
+                     'Authority',
+                     'Instructor',
+                     'Team',
+                     'SubActor',
+                     'SubObject',
+                     'SubInstructor',
+                     'SubTeam',
+                     'ContextAgent',
+                     'ContextGroup',
+                     'SubContextAgent',
+                     'SubContextGroup') NOT NULL;