feat(managesieve): add XOAUTH2 authentication mechanism #2773

felixauringer · 2025-07-23T07:23:17Z

As the managesieve server only supports plain authentication, here is a first implementation of XOAUTH2 as an additional authentication mechanism for managesieve.

I would be happy about feedback :)

chibenwa

This looks great.

Can we try to write a test for OIDC support in managedSieve like this is done for IMAP and SMTP?

Also, out of curiosity do we have a managedSive compatible client>

We are missing:

ManageSiee exemple in https://github.com/apache/james-project/tree/master/examples/oidc
Documentation on how to configure this feature

quantranhong1999

Read it.

Thank you for your contribution.

This lacks some tests indeed.

felixauringer · 2025-08-15T11:49:22Z

I added OAUTHBEARER authentication and some tests.

The authentication logic I wrote is very similar to james-project/server/protocols/protocols-smtp/src/main/java/org/apache/james/smtpserver/UsersRepositoryAuthHook.java or james-project/protocols/imap/src/main/java/org/apache/james/imap/processor/AuthenticateProcessor.java, I assume that those implement correct SASL XOAUTH2, but still decided to test many cases here.

I was not sure about using MPT and decided to implement a ManageSieveTestSystem like already existing for SMTP. As there is no managesieve client by Apache, I also had to write a small client.

I added tests for the whole authentication logic and found some bugs / non-standard-conforming behavior and tried to fix them.
There is one test I did not get working so far: After sending the logout command, I would like to check that the server really closes the connection. This happens asynchronously but even when waiting multiple seconds, isConnected still returns true. The RFC says the server MUST close the connection and in my understanding of the code, it also does so (channel.writeAndFlush(Unpooled.EMPTY_BUFFER).addListener(ChannelFutureListener.CLOSE)).

Also, out of curiosity do we have a managedSive compatible client>

Roundcube supports using managesieve with XOAUTH2/OAUTHBEARER.

I am working on the example and documentation, but if you have feedback to the tests before that, I would appreciate it 🙂

felixauringer · 2025-09-08T13:20:57Z

I have updated documentation and the oidc example. I added some CLI commands to test oidc authentication and made test.sh succeed. I did not test with thunderbird but my changes should not break the existing documentation on that.

chibenwa · 2025-09-14T20:27:07Z

Failing tests seems to be related. Can we investigate this?

felixauringer · 2025-10-09T07:34:30Z

The MPT tests are working again now.

The two logout tests which I added are still failing. As described above, I would like to keep those tests but do not know how to fix them.

There is also one test which I marked as disabled. It checks for malformed authentication data and currently fails but would also have failed with the original implementation. The current code is definitive too lenient when looking at the RFC but I didn't know whether there was a specific reason to allow this case
(You also allow spaces as separators in the authentication data instead of null bytes. This is also not correct, but I left it as is and didn't write a test for it.)
Are there any clients that rely on these non-RFC-compliant behaviors?

...l/managesieve/core/src/main/resources/org/apache/james/managesieve/scripts/authenticate.test

chibenwa · 2025-10-26T14:47:31Z

...mpl/managesieve/core/src/main/resources/org/apache/james/managesieve/scripts/capability.test

 C: AUTHENTICATE "PLAIN"
-S: \+ ""
-C:  user password
+S: ""
+S: OK
+C: "user password"


I am not fully sure that this code is compliant with eg Thunderbird extension

From my memories of SASL we either have

C: AUTHENTICATE "PLAIN" "AGFsaWNlAHNlY3JldA=="\r\n S: OK "Logged in"

Or

C: AUTHENTICATE "PLAIN"\r\n S: +\r\n C: AGFsaWNlAHNlY3JldA==\r\n S: OK "Logged in"\r\n

(with base64 being \0LOGIN\0password)

So seing not base64 encoded, qutoed password seems indeed weird to me...

From the above exemple the initial server response is + \r\n

After reading the code below I think the client payload handling is retro-compatible.

So I would welcome tests that demonstrate previous behaviours are still supported.

Yes, I am pretty sure that having spaces as separator is not compliant. If that's okay for you, I am fine with removing them.

The RFC actually also does not mention base64 but I also have never seen it without it.

The current behavior implementation allows the following for PLAIN:

null as delimiter, then base64 encoded, then quoted

null as delimiter, then quoted

space as delimiter, then quoted

(strings always need to be quoted except if the length is given beforehand)

Which of those behaviors would you like to be supported? I implemented all of them because the other protocols allow these options.

Because it was supported before, I added an additional test ensuring that a space still works as delimiter here.

chibenwa · 2025-10-26T14:52:49Z

...anagesieve/src/main/java/org/apache/james/managesieve/core/PlainAuthenticationProcessor.java

    @Override
    public String initialServerResponse(Session session) {
-        return "+ \"\"";
+        return "\"\"\r\nOK";


Why do we need changing the initial server response here?

I do not understand where this + "" comes from but I didn't find it in any of the RFCs relating to ManageSieve SASL authentication.
There is one mention in the SASL RFC but only for the EXTERNAL mechanism which we do not use here.
The in my opinion relevant RFC here is RFC 5804: A Protocol for Remotely Managing Sieve Scripts. The formal syntax there defines the response to an authenticate command as follows:

response-authenticate = *(string CRLF) ((response-ok [response-capability]) / response-nobye) ;; <response-capability> is REQUIRED if a ;; SASL security layer was negotiated and ;; MUST be omitted otherwise.

If we want continuation, we want to have a response-ok and because we do not negotiate a SASL security layer, we must omit response-capability.
It then simplifies to response-authenticate = *(string CRLF) response-ok. Assuming we do not want to have a response code or description, it further simplifies to response-authenticate = *(string CRLF) "OK" CRLF.
The description says:

Note that an empty challenge/response is sent as an empty string.

With all our supported authentication mechanisms, the challenge sent by the server is empty, so I would argue that the server should send the empty string. Sadly, the examples in the RFC do not show this case. Using the formal syntax from above, this leads to response-authenticate = DQUOTE DQUOTE CRLF "OK" CRLF. (One could probably also use the literal string variant without quotes but by supplying the exact string length. However, I would argue that it does not make much sense with an empty string.)

That's the reason why I changed it. In the RFC, I do not see any possibility how an (unquoted!) + sign could be the response to an authenticate command. I also didn't find why this server response shouldn't end with an OK, but I'm less sure about that (the examples seem to omit it in the challenge-response exchanges but I don't think the formal syntax allows it). What's the reason why you use the + here? Is there another RFC that I do not know?

Okay, I found the PLAIN SASL RFC from where your previous server response seems to come from. I really do think that it conflicts with the managesieve RFC (5804) but I am fine with keeping the old syntax then.

Hello @felixauringer

Thanks for having a look

I think as an implementer I just sticked to RFC-4616 exemples, yes (but that's10 years back.... I'm a bit rusty on the topic)

Note that the "" is likely none compulsory and not returned eg by the IMAP layer which also implement RFC-4616...

While what comes after is subject to discussion the fact of doing a continuation is mandatory.

I changed my code and all tests to behave like before and send + "".

chibenwa · 2025-10-26T14:58:25Z

...protocols-managesieve/src/test/java/org/apache/james/managesieveserver/AuthenticateTest.java

+    // The SASL PLAIN standard (https://datatracker.ietf.org/doc/html/rfc4616) defines the following message:
+    // message = [authzid] UTF8NUL authcid UTF8NUL passwd
+    // The current code is more lenient and accepts the message without the first null byte.
+    @Disabled
+    @Test


I we think it is a good thing to be more relaxed than the RFC, I see no issue:

Keeping the valuable comment

Testing the existing behaviour with a passing test.

I enabled the test and kept the comment.

chibenwa · 2025-10-26T15:02:28Z

Otherall agree with proposed changes but I would need clarifications on AUTHENTICATE "PLAIN" handling.

felixauringer · 2025-11-24T08:38:28Z

Otherall agree with proposed changes but I would need clarifications on AUTHENTICATE "PLAIN" handling.

Now, the behavior is the same except that the code is more strict when it comes to quoting. SASL mechanisms and the values sent by the client must be quoted according to the RFC. (In the RFC, there is the possibility of sending unquoted data which must be preceded by the length in the form {20+} but that was not supported before and is not supported now.)

felixauringer · 2025-11-24T11:57:17Z

I do not understand the failing test, I have changed nothing related to the Postgres repository, I think.

chibenwa

Thanks for the extensive test suite it's pleasan to read.

chibenwa · 2025-11-24T15:06:00Z

...mpl/managesieve/core/src/main/resources/org/apache/james/managesieve/scripts/capability.test

 C: AUTHENTICATE "PLAIN"
 S: \+ ""
-C:  user password
+C: "user password"


When authentication happens as part of continuation I am rather sure that the crediential must nt be quoted

Suggested change

C: "user password"

C: user password

CF https://datatracker.ietf.org/doc/html/rfc4616#section-4

I have added a commit that ensures continuation must be sent without quotes and adapts the tests accordingly.

However, I am not sure that your reference supports this behavior. All examples using continuation use the syntax where the unquoted message is preceded by the exact length of the client message. With that syntax, it is of course not necessary to quote the client message.

chibenwa · 2025-11-24T15:10:30Z

The test failure is unrelated.

I rescheduled a new build.

chibenwa · 2025-12-01T21:47:17Z

https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2773/23

org.apache.james.managesieveserver.AuthenticateTest.authenticatedStateUnlocksNewCommands

Failing for the past 1 build (Since Failed#23 )
Took 0.63 sec.
Add description
Error Message
expected: OK
 but was: NO
Stacktrace
org.opentest4j.AssertionFailedError: 

expected: OK
 but was: NO
	at org.apache.james.managesieveserver.AuthenticateTest.authenticatePlain(AuthenticateTest.java:227)
	at org.apache.james.managesieveserver.AuthenticateTest.authenticatedStateUnlocksNewCommands(AuthenticateTest.java:182)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

This seems related. Could you please have a look?

chibenwa

This work looks nice to me.
I can give a hand with the picky test suite if of any use.
Just ask.
Best regards,
Benoit

chibenwa · 2025-12-02T09:47:13Z

...mpl/managesieve/core/src/main/resources/org/apache/james/managesieve/scripts/capability.test

 C: AUTHENTICATE "PLAIN"
 S: \+ ""
-C:  user password
+C: user password


CF exemples in https://datatracker.ietf.org/doc/html/rfc4616

Example mentions <NUL>tim<NUL>tanstaaftanstaaf

We did test those inputs by using (space) and <NUL> in an inter-changeable way hence I explexpect we are still able to handle C: user password`

Is it still the case?
Can we have a tet making sure this is still the case?

I would argue that the line before the one you referenced is also important here: C: {21}
The clients announces the exact length of its message. This syntax is defined in the managesieve standard but currently not implemented in James. I do not know whether there are any clients using it.

The exact line C: user password appears multiple times in the managesieve MPL tests that were there before my contribution.
I think in the MPL tests only use plain authentication with continuation, use spaces instead of NUL and do not quote the client message during continuation.
The Java Tests I wrote usually use NUL (including the leading NUL) and encode it with base64. However, there is also one test for accepting it without the leading NUL and one that accepts the credentials with one space between username and password (not base64 encoded). There are no Java tests that do test those variants also with continuation.

I am not exactly sure what additional tests you would like to have.

I think there are currently no tests for a leading space when the credentials are not base64 encoded, do you mean that?

I added an additional test.

felixauringer · 2025-12-02T10:26:34Z

This seems related. Could you please have a look?

Yes, should be fixed now.

I can give a hand with the picky test suite if of any use.

Thanks for the offer! But this time it was not the testsuite but just a plain programming error by me. I thought I could make this small change without running the tests locally but that didn't work apparently 😅
However, there are still 2 disabled tests where I wanted to check whether the server really closes the connection but I couldn't get them to work so far.

chibenwa · 2025-12-02T11:36:39Z

I can give a hand with the picky test suite if of any use.

Good job it's 🍏 :-)

XOAUTH2 is described here: https://developers.google.com/workspace/gmail/imap/xoauth2-protocol OAUTHBERER is described here: https://datatracker.ietf.org/doc/html/rfc5801#section-4 There is a small difference in how the user argument is called and OAUTHBEARER also includes a GS2 header.

- Authentication is now aborted when session is already authenticated - Authentication is now aborted when client sends "*" - Arguments by the client are now consistently unquoted - If unquotation fails, authentication is aborted to prevent later NullPointerExceptions - Fixed NullPointerException on AuthenticationException - Server now sends correctly formatted response (including code) when waiting for additional authentication arguments - When client sends authentication credentials in second message, the first argument (command) is now interpreted as argument - When choosing mechanism failed, a proper error is returned to the client - When argument is not quoted, authentication is aborted I think with these changes, the server now behaves more like described in the RFC: https://datatracker.ietf.org/doc/html/rfc5804#section-4

… in oidc example

- More modern syntax in compose file. - Remove non-working links from readme. - Use consistent container names (always ending in .example.com). - Reduce output of test script.

…ation error While trying to get the MPT tests for managesieve running, I realized that a client is trapped in the state "AuthenticationInProgress" when using continuation. This made longer test scripts impossible. To consistently reset the session, I introduced more errors that are handled in one place instead of returning a "NO" answer directly during authentication. Those changes also required some small changes to the MPT and the normal tests. Additionally, there were a parsing bug when sending space-separated username and password in the authentication continuation which is fixed now.

…ll byte

…uring continuation

…ers in SASL PLAIN

chibenwa · 2025-12-17T10:46:59Z

Recent changes regarding SASL improvement impact this PR

10:42:32,414 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.14.0:compile (default-compile) on project protocols-managesieve: Compilation failure: Compilation failure: 
more_vert
9765	11:42:32 AM	
10:42:32,414 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/api/Session.java:[27,38] cannot find symbol
more_vert
9766	11:42:32 AM	
10:42:32,414 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9767	11:42:32 AM	
10:42:32,414 [ERROR]   location: package org.apache.james.protocols.api
more_vert
9768	11:42:32 AM	
10:42:32,414 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/api/Session.java:[57,14] cannot find symbol
more_vert
9769	11:42:32 AM	
10:42:32,414 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9770	11:42:32 AM	
10:42:32,414 [ERROR]   location: interface org.apache.james.managesieve.api.Session
more_vert
9771	11:42:32 AM	
10:42:32,414 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/api/Session.java:[59,44] cannot find symbol
more_vert
9772	11:42:32 AM	
10:42:32,414 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9773	11:42:32 AM	
10:42:32,414 [ERROR]   location: interface org.apache.james.managesieve.api.Session
more_vert
9774	11:42:32 AM	
10:42:32,414 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/core/OAUTHAuthenticationProcessor.java:[34,38] cannot find symbol
more_vert
9775	11:42:32 AM	
10:42:32,414 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9776	11:42:32 AM	
10:42:32,414 [ERROR]   location: package org.apache.james.protocols.api
more_vert
9777	11:42:32 AM	
10:42:32,414 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/core/OAUTHAuthenticationProcessor.java:[40,19] cannot find symbol
more_vert
9778	11:42:32 AM	
10:42:32,414 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9779	11:42:32 AM	
10:42:32,414 [ERROR]   location: class org.apache.james.managesieve.core.OAUTHAuthenticationProcessor
more_vert
9780	11:42:32 AM	
10:42:32,414 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/core/OAUTHAuthenticationProcessor.java:[42,41] cannot find symbol
more_vert
9781	11:42:32 AM	
10:42:32,414 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9782	11:42:32 AM	
10:42:32,415 [ERROR]   location: class org.apache.james.managesieve.core.OAUTHAuthenticationProcessor
more_vert
9783	11:42:32 AM	
10:42:32,415 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/util/SettableSession.java:[28,38] cannot find symbol
more_vert
9784	11:42:32 AM	
10:42:32,415 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9785	11:42:32 AM	
10:42:32,415 [ERROR]   location: package org.apache.james.protocols.api
more_vert
9786	11:42:32 AM	
10:42:32,415 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/util/SettableSession.java:[36,22] cannot find symbol
more_vert
9787	11:42:32 AM	
10:42:32,415 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9788	11:42:32 AM	
10:42:32,415 [ERROR]   location: class org.apache.james.managesieve.util.SettableSession
more_vert
9789	11:42:32 AM	
10:42:32,415 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/util/SettableSession.java:[89,21] cannot find symbol
more_vert
9790	11:42:32 AM	
10:42:32,415 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9791	11:42:32 AM	
10:42:32,415 [ERROR]   location: class org.apache.james.managesieve.util.SettableSession
more_vert
9792	11:42:32 AM	
10:42:32,415 [ERROR] /home/jenkins/workspace/james_ApacheJames_PR-2773/protocols/managesieve/src/main/java/org/apache/james/managesieve/util/SettableSession.java:[94,51] cannot find symbol
more_vert
9793	11:42:32 AM	
10:42:32,415 [ERROR]   symbol:   class OidcSASLConfiguration
more_vert
9794	11:42:32 AM	
10:42:32,415 [ERROR]   location: class org.apache.james.managesieve.util.SettableSession
more_vert
9795	11:42:32 AM	
10:42:32,415 [ERROR] -> [Help 1]

A rebase is needed. If needed I can do it.

chibenwa reviewed Jul 23, 2025

View reviewed changes

quantranhong1999 reviewed Jul 23, 2025

View reviewed changes

felixauringer force-pushed the feature-managesieve-xoauth2 branch from e17b138 to 6b61443 Compare August 15, 2025 11:47

felixauringer force-pushed the feature-managesieve-xoauth2 branch from 6b61443 to 49efc9a Compare August 25, 2025 07:34

felixauringer force-pushed the feature-managesieve-xoauth2 branch from 7fc01a0 to 91ac2a3 Compare September 13, 2025 10:02

felixauringer marked this pull request as ready for review September 13, 2025 19:13

felixauringer force-pushed the feature-managesieve-xoauth2 branch 2 times, most recently from da9b214 to c31419e Compare October 9, 2025 07:33

felixauringer force-pushed the feature-managesieve-xoauth2 branch 2 times, most recently from 2387ebe to 05bb331 Compare October 26, 2025 13:54

chibenwa reviewed Oct 26, 2025

View reviewed changes

felixauringer force-pushed the feature-managesieve-xoauth2 branch 3 times, most recently from 8fe1061 to 1a05a2e Compare November 24, 2025 08:31

felixauringer force-pushed the feature-managesieve-xoauth2 branch 3 times, most recently from 91d2815 to e5815f9 Compare November 24, 2025 09:29

chibenwa reviewed Nov 24, 2025

View reviewed changes

felixauringer force-pushed the feature-managesieve-xoauth2 branch 2 times, most recently from 1087d44 to a78424c Compare December 1, 2025 10:36

chibenwa reviewed Dec 1, 2025

View reviewed changes

felixauringer force-pushed the feature-managesieve-xoauth2 branch from a78424c to 4fca7f8 Compare December 2, 2025 08:05

chibenwa reviewed Dec 2, 2025

View reviewed changes

felixauringer added 16 commits December 8, 2025 12:55

feat(managesieve): add XOAUTH2 authentication mechanism

7084021

feat(managesieve): add OAUTHBEARER authentication mechanism

da6d022

test(managesieve): test authentication of managesieve server

f77fb0b

docs(managesieve): document usage of oidc authentication

eff9cbe

fix(examples): fix introspection mismatching issuer error

e6a5905

feature(examples): configure managesieve and test oidc authentication…

7edf78a

… in oidc example

refactor(examples): small improvements for oidc example

f8055c3

- More modern syntax in compose file. - Remove non-working links from readme. - Use consistent container names (always ending in .example.com). - Reduce output of test script.

fix(managesieve): Implement continuation as described in RFC-4616

c64f6ac

fix(oidc): wrong format of gs2-header

7db89be

refactor(managesieve): accept plain authentication without leading nu…

e5662fc

…ll byte

feat(managesieve): add test for old SASL PLAIN parsing behavior

cb2aa9d

refactor(managesieve): only accept unquoted initial client response d…

22a76eb

…uring continuation

feat(managesieve): add additional test for usage of spaces as delimit…

07ac6a3

…ers in SASL PLAIN

felixauringer force-pushed the feature-managesieve-xoauth2 branch from bfc44d2 to 07ac6a3 Compare December 8, 2025 11:55

feat(managesieve): add XOAUTH2 authentication mechanism #2773

Are you sure you want to change the base?

feat(managesieve): add XOAUTH2 authentication mechanism #2773

Uh oh!

Conversation

felixauringer commented Jul 23, 2025

Uh oh!

chibenwa left a comment

Choose a reason for hiding this comment

Uh oh!

quantranhong1999 left a comment

Choose a reason for hiding this comment

Uh oh!

felixauringer commented Aug 15, 2025

Uh oh!

felixauringer commented Sep 8, 2025

Uh oh!

chibenwa commented Sep 14, 2025

Uh oh!

felixauringer commented Oct 9, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chibenwa Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chibenwa commented Oct 26, 2025

Uh oh!

felixauringer commented Nov 24, 2025

Uh oh!

felixauringer commented Nov 24, 2025

Uh oh!

chibenwa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chibenwa commented Nov 24, 2025

Uh oh!

chibenwa commented Dec 1, 2025

Uh oh!

chibenwa left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixauringer commented Dec 2, 2025

Uh oh!

chibenwa commented Dec 2, 2025

Uh oh!

chibenwa commented Dec 17, 2025

chibenwa Nov 12, 2025 •

edited

Loading

chibenwa left a comment •

edited

Loading