Formgrader doesn't work, command line do

[bpfrd]

Hi
I deployed jupyterhub & nbgrader in kubernetes. I'm using this version of nbgrader https://github.com/CERIT-SC/nbgrader-k8s
please find the issue and details below:

Operating system

Centos

nbgrader --version

nbgrader version 0.9.2

jupyterhub --version (if used with JupyterHub)

jupyterhub version 3.0.3

jupyter notebook --version

jupyterhub notebook version 7.1.2

Actual behavior

Formgrader stopped working recently and none of its functionalities, e.g., autograde, release feedback, manual grading, etc work! It doesn't give any specific errors. it shows something like "there is an error in autograding...." However, the command line works. but manual grading is not possible with the command line.

best regards

[bpfrd]

I should add that, when doing manual grading the comments and grades are not saved and have no effect.

[brichet]

@bpfrd did you update nbgrader or jupyterlab/notebook recently ?
Otherwise, I would suspect an issue with configuration, maybe related to https://github.com/CERIT-SC/nbgrader-k8s (I don't know about that).

[bpfrd]

Thanks. I didn't update and I don't think teachers did too. How can I debug?
We do autograding, releasing assignment, etc. with command line. Is it possible to do manual grading with command line as well?

[brichet]

For reference CERIT-SC/nbgrader-k8s#11

[perllaghu]

Is it possible to do manual grading with command line as well?

Manual grading from the command-line would need an alternative UI to allow comments & grade-changes - which seems a lot of work when there's already a UI available
[YMMV]

[bpfrd]

Just an update about this issue. (formgrader GUI not working). Here is the console log when I click on any key in GUI (such as generate assignment, etc.) It seems it's not authorized.

console:

POST https://redacted/services/python_fundamentals_fs24/formgrader/api/assignment/Musterpruefung/assign 403 (Forbidden)

network tab:

Request URL:
https://redacted/services/python_fundamentals_fs24/formgrader/api/assignment/Musterpruefung/assign
Request Method:
POST
Status Code:
403 Forbidden
Remote Address:
redacted:443
Referrer Policy:
same-origin

request headers:

:authority:
redacted
:method:
POST
:path:
/services/python_fundamentals_fs24/formgrader/api/assignment/Musterpruefung/assign
:scheme:
https
accept:
*/*
accept-encoding:
gzip, deflate, br, zstd
accept-language:
en-US,en;q=0.9,de-DE;q=0.8,de;q=0.7
cache-control:
no-cache
content-length:
0
cookie:
_xsrf=2|fae28604|4873e5d65a82822f21e6eee8fc724c9b|1733150397; service-python_fundamentals_fs24=2|1:0|10:1733151656|32:service-python_fundamentals_fs24|40:NzlxdGtFbDE2T2dZam82T2Z5Rk9GVERNOVB6Vnk2|94d6527732567b342ee0d91cb8b59c1400e1a9f949d3fad9ef4dcad779d31ddf; jupyterhub-session-id=233cbc7e758643029d35685ef48d0623
origin:
redacted
pragma:
no-cache
priority:
u=1, i
referer:
https://redacted/services/python_fundamentals_fs24/formgrader
sec-ch-ua:
"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"
sec-ch-ua-mobile:
?0
sec-ch-ua-platform:
"Windows"
sec-fetch-dest:
empty
sec-fetch-mode:
cors
sec-fetch-site:
same-origin
user-agent:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
x-csrftoken:
undefined
x-requested-with:
XMLHttpRequest

response headers:

content-length:
5957
content-security-policy:
frame-ancestors 'self'; report-uri /services/python_fundamentals_fs24/api/security/csp-report
content-type:
text/html; charset=UTF-8
date:
Mon, 02 Dec 2024 15:08:19 GMT
permissions-policy:
microphone=(), geolocation=(), camera=()
referrer-policy:
same-origin
strict-transport-security:
max-age=31536000; includeSubDomains
x-content-type-options:
nosniff
x-content-type-options:
nosniff
x-download-options:
noopen
x-frame-options:
SAMEORIGIN
x-jupyterhub-version:
X
x-permitted-cross-domain-policies:
none
x-robots-tag:
noindex, nofollow
x-xss-protection:
1; mode=block

network preview:

Sorry, you are not authorized to access the formgrader.

The formgrader CLI works. Do you need more information? Do you have any clues about how the problem is caused?

best

[bpfrd]

and here seems to be the hub logs on kubernetes for the same request:

[D 2024-12-02 16:28:29.035 ServerApp] Received request from Hub user {'kind': 'user', 'name': 'redacted', 'groups': ['formgrade-python_fundamentals_fs24'], 'admin': True, 'session_id': '233cbc7e758643029d35685ef48d0623', 'scopes': ['access:services!service=python_fundamentals_fs24', 'read:users:groups!user=redacted', 'read:users:name!user=redacted']}
[D 2024-12-02 16:28:29.035 ServerApp] Checking user redacted with scopes ['access:services!service=python_fundamentals_fs24', 'read:users:groups!user=redacted', 'read:users:name!user=redacted'] against {'access:services!service=python_fundamentals_fs24', 'access:services'}
[D 2024-12-02 16:28:29.035 ServerApp] Allowing user redacted with scopes {'access:services!service=python_fundamentals_fs24'}
[I 2024-12-02 16:28:29.036 ServerApp] 101 GET /services/python_fundamentals_fs24/api/events/subscribe?token=[secret] (redacted@10.42.0.1) 19.87ms
[D 2024-12-02 16:28:30.924 ServerApp] Checking user redacted with scopes ['access:services!service=python_fundamentals_fs24', 'read:users:groups!user=redacted', 'read:users:name!user=redacted'] against {'access:services!service=python_fundamentals_fs24', 'access:services'}
[D 2024-12-02 16:28:30.924 ServerApp] Allowing user redacted with scopes {'access:services!service=python_fundamentals_fs24'}
[W 2024-12-02 16:28:30.925 ServerApp] 403 POST /services/python_fundamentals_fs24/formgrader/api/assignment/Musterpruefung/assign (10.42.0.1): XSRF cookie does not match POST argument
[W 2024-12-02 16:28:30.925 ServerApp] 403 POST /services/python_fundamentals_fs24/formgrader/api/assignment/Musterpruefung/assign (redacted@10.42.0.1) 1.60ms

where my username is redacted.

best

[bpfrd]

another update:
as you see the x-csrftoken in the request header was undefined
this was because of the following configuration in my values.yaml

jupyterhub:
  ingress:
    annotations:
      nginx.ingress.kubernetes.io/configuration-snippet: |
        proxy_cookie_flags ~ Secure HttpOnly;

if I comment this line, formgarder GUI works well. but the xsrf token is no longer secure and httponly.

The question is: what is happening exactly? How can I maintain these security flags while ensuring the formgrader GUI continues to function correctly?

if I understand correctly this is caused because the nbgrader frontend extension tries to get the xsrf token in js which is in conflict with httponly flag. is that right? is there any fix?

best