WeBWorK 2.20 Release Candidate #2721
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For LTI 1.3 only consider roles in the context for automatically created users.
…se (due) date for the LTI critical date
Some features of this change are improved syntax highlighting and the addition of autocompletion. To see available autocompletions at the current cursor location, you can type Ctrl-Space. In many cases, autocompletions are offered automatically when certain things have been typed. CodeMirror 6 can not be used via script tags. So a PGProblemEditor package has been created for this. The repository for that is https://github.com/openwebwork/pg-problem-editor. An initial package has been published in npm at https://www.npmjs.com/package/@openwebwork/pg-codemirror-editor. The real work is done in the codemirror-lang-pg package that adds support for the PG language in CodeMirror 6. The repository for that package is https://github.com/openwebwork/codemirror-lang-pg, and the npm package at https://www.npmjs.com/package/@openwebwork/codemirror-lang-pg. The pg-problem-editor package is designed so that it can be used by webwork2, the standalone renderer, and possible others that want a PG problem editor. It may need a few tweaks to make it work for webwork3 in the future, but should also work for that. The codemirror-lang-pg package consists of a base PG parser that is largely derived from the codemirror-lang-perl package (at https://github.com/drgrice1/codemirror-lang-perl and https://www.npmjs.com/package/codemirror-lang-perl) with a modifications specific to the perl available in a PG problem. The main differences are that its escape sequence is ~~, much of what can not be used in the safe compartment has been stripped out, and there are special blocks for PGML, PG text, and LaTeX image code. Then there are two other parsers. Those are the PGML and PG text parsers. Those parsers run on the contents of the PGML and PG text blocks found by the base PG parser. Note that the PGML parser is actually a javascript implementation of the parser in PGML.pl with some modifications to track position and things needed to associate the parsed tree with the original code. Note that the configuration of theme, keymaps, and such are now part of the pg-codemirror-editor, and are shown in a panel of the editor instead of below as before. Emacs and Vim keymaps are available, but at this point there is not a Sublime keymap for CodeMirror 6. The themes that are available include a basic default theme (that is part of the pg-problem-editor package and just adds some syntax highlighting that is missing from the CodeMirror 6 default), the OndDark theme that is the only other theme provided by the CodeMirror 6 developers directly, the themes in the thememirror package, and the themes from https://github.com/craftzdog/cm6-themes which are in several separate npm packages. Note that both thememirror and craftzdog/cm6-themes provide a solarized light theme, and that is why there are two of those. They are slightly different though. I am sure that there is a lot more work to do on this, and there are other things available with CodeMirror 6 that could probably be implemented. In any case, this is a good start.
…ievement notifications. The new codemirror-lang-mt package is used for this, and support for that language has been added to the pg-codemirror-editor.
Restructure LTI grade pass back so that there are no additional database
queries with the new restrictions on when grades are passed back to the
LMS. With this restructuring the only database queries are made while
grading the sets the same as before.
The grade_set and grade_gateway methods return the things they pull from
the database so that the database records can be culled for the
additional information needed to determine if grade pass back should
occur. The usage of these methods elsewhere is the same. The new data
that is passed back will simply be ignored in those cases.
The `lib/WeBWorK/Authen/LTI/MassUpdate.pm` file has been renamed to
`lib/WeBWorK/Authen/LTI/GradePassback.pm` and contains the previous
`mass_update` method as well as all methods needed for the new
processing.
The `grade_all_sets` method that was in `lib/WeBWorK/Utils/Sets.pm` now
takes `$getSetGradeConditionally` as an optional last argument. That
argument should be a reference to a subroutine that takes the arguments
$db, $ce, $studentName, and $userSet, and either returns a reference to
a hash containing the keys totalRight and total with the grade for the
set, or returns `undef`. If it returns `undef` then the set will not be
included in the grade computation. Otherwise the totalRight and total
will be added into the grade for all sets. If the optional last
arugment is not provided, then a default method will be used that does
the same thing as before.
Note that the `$LTI{v1p1}{grader}` option the `authen_LTI_1_1.conf.dist`
and `authen_LTI_1_3.conf.dist` files has been removed. Instead the
correct module is determined directly from the `$LTIVersion` as was
previously done in the `lib/Mojolicious/WeBWorK/Tasks/LTIMassUpdate.pm`
file. This way a system adiminstrator cannot accidentally delete that
or modify it in some way and mess things up. Since this is only done in
these two places and not needed elsewhere the convenience of the course
environment variable containing the module name is not needed.
Rework of LTI grade pass back.
Upgrade to CodeMirror 6.
When adding a course on the course admin page, list all users in the admin course and let the admin select which users to copy over to the new course (by default all admin users are selected). This way an admin can select an existing user to copy over which can include any password or OTP secrets setup for that use (currently the admin will have to copy that over to the admin course manually). This also allows only copying a subset of admins to a newly created course instead of all. When creating a new user to add to a course, be consistent with adding a user on the accounts management page, which doesn't require a password (useful when using external auth), email, etc. Now the only required field is the new user ID. Flag the new user password input as 'new-password', so webbrowsers don't try to auto fill it in. Add the student ID as a field they can fill out instead of making it equal to the user ID. Since users can be copied from the admin course, add an option to add the new user to the admin course (as a dropped user so they can't login) so admins can select the same user to add to future courses as needed. Remove the empty list of sets to assign users to when adding new users on the accounts management page in the admin course. Make it so all users are shown on the Email page in the admin course, since most instructors are "Dropped" to still allow sending them emails.
This way it is harder to unselect the admin users, to avoid accidentally unselected them if ctrl isn't being held down.
Use the course_admin_id instead of 'admin' when talking about the
admin course name on this page and in its help.
Mention setting $permissionLevels{login}='admin' in course.conf
in the help as a way to control access to the admin course.
Use `overflow-auto` and `max-height` in the select which admin users
to add to the new course div.
When adding a new course, and someone unchecks all admin users to be copied to the new course, ensure they stay unchecked if an error occurs and the page is reloaded.
Co-authored-by: Glenn Rice <[email protected]>
… email feedback template
A little code clean up.
Fix the import form date shift input on the sets manager page.
Currently the user achievements are obtained from the database in the user loop. This takes a long time when there are lots of users. So instead, this gets all user achievements for all users before the loop and references them by user id and then achievement id. Testing this on a course for 5000 users shows this gives a significant speed up on load time for the page. With the develop branch it takes around 25 seconds, and with this branch it takes around 3 seconds. Note that there were also two redundant queries (one that listed all achievements and then one that fetched all achievements on lines 49 and 50 in the develop branch) for which the data from those queries was not even used. Those were removed.
Instead of listing user achievements and then deleting them one by one in a loop, just delete them with one query using a `where` statement.
It turns out that none of the ContentGenerator controller objects are
being destroyed when a request finishes. So each hypnotoad process (or
morbo in development) has every ContentGenerator controller object for
every request it renders saved in memory until the process ends. That
means that everything it had a reference to is also saved in memory.
That includes a `WeBWorK::CourseEnvironment` instance, a
`WeBWorK::Authen` instance, a `WeBWorK::Authz` instance, and a
`WeBWorK::DB` instance (and everything it has a reference to).
Furthermore, even if the controller objects are destroyed and the
`WeBWorK::DB` instance with it, none of the `WeBWorK::DB::Schema`
instances (one for each table) are ever destroyed.
There are two things that cause these references to be kept when they
shouldn't be.
The first is the more obvious circular reference..
A `WeBWorK::Controller` object (from with the `WeBWorK::ContentGenerator`
modules derive) keeps a reference to a `WeBWorK::Authz` instance, and
that instance keeps a reference back to the controller. However, the
`WeBWorK::Authz` doesn't weaken the reference back to the controller.
That was my fault in the conversion to Mojolicious I commented out the
`weaken` statement that prevented this circular reference. That was
because in the initial conversion the controller didn't have a reference
to the `WeBWorK::Authz` instance, and so it was going out of scope and
causing problems. However, when the reference to that instance was
added that should have been uncommented.
Another case of this is that `WeBWorK::Authen::LTIAdvanced` and
`WeBWorK::Authen::LTIAdvantage` packages were keeping a circular
reference to the controller as well. The new methods in those packages
was just deleted so that they use the `WeBWorK::Authen` new method
which already does the right thing.
A third case occurs with the `WeBWorK::DB` instance and the
`WeBWorK::DB::Schema` instances both of which hold references to each
other.
The other thing that causes an extra reference to be kept is an
anonymous subroutine (or closure) using an instance. In this case Perl
forces the instance to be kept in scope for usage in the closure.
The global `$SIG{__WARN__}` handler defined in `Mojolicious::WeBWorK`
uses the `$c` controller instance, and that is what prevents the
`WeBWorK::ContentGenerator` modules from going out of scope. So that
instance in the `around_action` hook needs to be weakened.
For the `WeBWorK::DB::Schema::NewSQL::Std` and `WeBWorK::DB::Schema::NewSQL::Merge`
objects the issue is the `transform_table` and `transform_all` closures
for the sql abstract instances. Those prevent the schema objects from
going out of scope and so the `$self` in the `sql_init` methods where
those closures are defined needs to be weakened as well.
that the $SIG{__WARN__} handler is reset in the after_dispatch hook so
that the reference to the controller is released.
Fix memory leaks.
Improve loading time for the achievements leaderboard.
…-speed-tweak Speed up deleting of global user achievements.
The `exportUsersToCSV` method of `WeBWorK::ContentGenerator::Instructor::UserList` assumes that all users have password and permission records in the database. Since that is no longer the case, that code is not working right. In fact if some users do not have password records and others do, then the resulting classlist that is exported will have many passwords in associated to the wrong users. Some of the users that don't have passwords will have passwords, and vice versa. Note that the code had `defined` statements checking if the permission and password records exist, which mean that the code was technically incorrect to begin with. It only worked because usually all users had password and permission records. The general issue with this is noted in #2704 with regards to the deletion of the code that auto creates password or permission records when `getPasswords` or `getPermissionLevels` is called. The general issue is that the `getPasswords` and `getPermissionLevels` methods call the `WeBWorK::DB::Schema::NewSQL::Std::gets` method which does not even include non-existent records. So for example, if `getPasswords` is called with the list of user IDs `'user1', 'user2', 'user3'` and `user2` does not have a password, but the others do, then `getPasswords` will return an array with two elements which are the password records for `user1` and `user3`. So when iterating by index on the original list the password record for `user1` will correctly go to `user1`, but the password record for `user3` will be paired with `user2`, and `user3` will not have a password record. Also, the `exportUsersToCSV` dies if the provided filename contains a forward slash. So now the JavaScript form validation checks for this, and prevents those filenames from being submitted to the server. Thus preventing the `die` statement from occurring.
Use a batch insertion and update for users, permissions, and passwords when importing a class list. This improves the speed of imporing large class lists quite a bit. For example, importing a classlist containing 5000 users improves from around 22 seconds to around 7 seconds in my testing.
…achment.
Email::Stuffer incorrectly adds an attachment together with the text and
html body parts all at the same level with content type multipart/mixed.
Structurally this is as follows:
[
(content type multipart/mixed)
text body,
html body,
attachment
]
As a result, when an email has a text body, an html body, and an
attachment, both the text and html are shown in most email clients.
The text and html body should be parts of a separate part that has
content type 'multipart/alternative'. So the email has two parts, and
the first part has two parts in that which are the text and html body.
The second part is the attachment. The following shows how it should be
structurally:
[
(content type multipart/mixed)
[
(content type multipart/alternative)
text body,
html body
],
attachment
]
To fix this, before adding an attachment the text and html body are
moved into a separate part with content type multipart/alternative.
Load all user sets for the given set and all users before rendering the template instead of getting one set at a time in the for loop inside the template. This speeds up loading of this page a lot with a large number of users. With 5000 users it decreases the load time from 3.4 seconds to 0.14 seconds.
Fix an issue with feedback emails with text and html body, and an attachment.
…ed-up Speed up loading of the "Users Assigned to Set" page.
Speed up class list importing.
Make WebworkWebservice honor showEvaluatedAnswers
LTI section identification
…swords Fix classlist exporting with users that do not have passwords.
There is no need for the license to be in all files. That creates an unnecessary maintenance hassle with keeping the copyright years up to date, and it is sufficient to have the license (with the copyright years) in the `LICENSE` file alone. Furthermore, this is done quite inconsistently. Sometimes files are added without the license, and potentially worse is a file being added with the license but not in the exact format needed for the `bin/dev_scripts/update-copyright` script to detect and properly update the copyright years. The `bin/dev_scripts/update-copyright` script is thus no longer needed and has been removed. Also, the fallback copyright years for the footer now is just "unknown" instead of being the correct copyright years. If the version and copyright years are not correctly loaded in the course environment from the `VERSION` file then that is what you will see. That should never happen. So from now on the only files that need to be updated are the `LICENSE`, `README.md`, and `VERSION` files. In addition the `docker-config/docker-compose.dist.yml` and `DockerfileStage2` files need to have `forWW...` tag name updated to the current version (for the two stage build).
Change the version to 2.20 and remove the license from files.
changes to email template for feedback
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It is time for this. We have talked about just using the develop branch, but I think it is nice to have a separate branch for the release candidate. It just keeps things separate in a nice way.
So as usual, re-target pull requests for this branch that you want to get into the release, and I will handle synchronizing those to develop as they are merged.