Skip to content

Commit 40a4adc

Browse files
pcapriottismattinglepsaakshaymankarmdimjasevic
authored
Merge develop into mls (#3597)
* Fix bug: federatorInternal host not set for background-worker (#3516) * WPB-3916: Filtering out duplicate members when sending defederation notifications (#3515) * integration: Add test to verify behaviour with offline backends (#3501) * background-worker: Make push backoff times configurable * brig/getFederationStatus: Always return NonConnectedBackends as empty when fed policy is AllowAll * integration: Use separate vHosts for backendA and B. * integration/RunServices: Add hack to make federation work * integration: Add test to verify behaviour with offline backends * helm-var-integration: Workaround bug with federation * integration-test.sh: Run new integration test suite first --------- Co-authored-by: Marko Dimjašević <[email protected]> * Distinguish between update and upsert cassandra commands (#3513) * Remove billing-team-member-backfill tool (#3520) * dockerephemeral: Increase nofile ulimits for ES and Fake DynamoDB (#3521) * [WPB 3842] Federation completeness check (#3514) * WPB-3842: Improving checks for adding users to a conversation. Added a check to `ensureAllowed` that checks for full federation connections for domains in a conversation, including the domains for new users. * WPB-3842: Adding the changelog * WPB-3842: Moving where the extra domain checks are being performed. Updating integration tests to reflect the updated semantics of conversation join semantics. Many of them weren't expecting errors relating to unreachable domains, and had to be updated to reflect this. * Fix asserted domains in an integration test * Integration test: assert on non-federating domains * WPB-3842: Changing parallel testing to sequential testing --------- Co-authored-by: Marko Dimjašević <[email protected]> * WPB-3798 incorrect json field names (#3518) * WPB-3798: Updating code and tests after renaming fields * WPB-3798: More updates to names after finding more JSON prefix mangling * WPB-3798: Fixing schema instances for SAML data * WPB-3798: Fixing instances that had errors, found by tests * WPB-3798: Adding changelogs * WPB-3798: PR feedback. * WPB-3798: Fixing an error with a field called `data'` The trailing ' would end up in the JSON representation. I've changed it to use a leading `_` like other structures, and wrote a newtype to handle the minimal prefix stripping. Also cleaning up the diff in regards to imports. * WPB-3798: Cleaning up imports to minimise the diff * nit-picks (#3519) * Remove unneeded -Wwarn (re-enabeling -Werror in those modules). * Makefile: fix hspec_options overloading in .envrc.local. * integration: Fix testAddingUserNonFullyConnectedFederation and testNotificationsForOfflineBackends (#3529) * integration: Fix testAddingUserNonFullyConnectedFederation * integration: Don't allow adding users to conv when one of the pariticipating backends is down * integration: Add retries to get around problem of federation domain sync threads * Introduce API v5 (#3527) * Introduce development version 5 * Specialise API to a specific version * Use versioned swagger for galley * Use version swagger for all other services * Collect all service Swaggers into a typeclass * Fix swagger integration tests * Revert any changes to API versions before 5 * Remove promotion of isDevelopmentVersion * Add CHANGELOG entry * stern: Optimize RAM usage of /i/users/meta-info (#3522) * stern: Fetch only the notifications that are needed * stern: Fetch only the conversations that are needed * Integration tests: use static ports (#3536) * [WPB-3799] cannot fetch conversation details after connection request (#3538) * brig-integration: Fix flaky tests for API.Federation (#3539) * brig-integration: Don't assume only 1 result in search by display name Display names are random strings from 2 to 128 characters. If a 2 string name gets generated it is likely that it matches some name generated in another test. * brig-integration: Mark test not flaky It didn't fail after runnning it 1000 times. * Integration suite: Fix bug in local setup: wrong port for nginz http2 (#3543) * [WPB-662] servantify brig provider bot api (#3540) * Fix broken "we are hiring" link (#3549) * Multi-ingress guest links (#3546) * Check validity of notification IDs (#3550) * Check validity of notification IDs * Add CHANGELOG entry * fixup! Add CHANGELOG entry * fixup! fixup! Add CHANGELOG entry * WPB-633 Servantify Brig/Provider.Service API (#3554) * WPB-1214: Servantify Brig/Provider.Service API - Moving the routes over to servant, and removing the old routing code. - Adding new instances to types that needed them for servant. * WPB-663: Removing a redundant TODO comment, adding changelog * Fix ES migration script. (#3558) * Revert "WPB-633 Servantify Brig/Provider.Service API (#3554)" This reverts commit 3653d56. * Integration tests: delete all rabbitmq queues during dynamic backends setup phase (#3523) * [WPB-4406] federator improve logging (#3556) * Makefile: Avoid executing the hint (#3564) Backticks execute the command even when they are in quotes. * Finalise v4 (#3545) * Remove MLS endpoints from the API They will be reintroduced when merging the mls branch. These endpoints are not currently functional on develop, so removing them from here will reduce the amount of conflicts. * Finalise v4 * Add CHANGELOG entry * Add pregenerated swagger for v4 * Delete MLS tests in brig * Remove more MLS endpoints from v4 * Set default API version to 5 in integration tests * Update the documentation on API versioning --------- Co-authored-by: Marko Dimjašević <[email protected]> * Fix: SCIM user lookup after changing IdP issuer ID (#3473) * doc: document webapp configuration for multi-ingress environments (#3569) --------- Co-authored-by: Sven Tennie <[email protected]> * [WPB-4361] upgrade jwt-tools (#3559) * cassandra: Add column and table names in parsing error messages (#3555) * s/CORS/CSP/ as mentionned by Sven in WPB-2912 * Replace broken integrations with links see WPB-3599 * replace all instances of example.com with wire.example as per wpb-2621, in charts only * change back from wire.example to example.com as this was mistakenly commit to develop instead of to the proper branch * add documentation on creating a first user * reverting previous commit as sent to wrong branch * Update sftd docs: include uri scheme in allowOrigin (#3584) * Update sftd docs: include uri scheme in allowOrigin * fixup * WPB-4629 impossible to add users to a conversation if one of the members is from an offline backend (#3585) * fake-aws-s3 chart: Upgrade to minio 5.0.13 (#3565) * Disable de-federation to avoid running into a scalability issue (#3582) https://wearezeta.atlassian.net/browse/WPB-4668 Co-authored-by: Akshay Mankar <[email protected]> * [WPB-3664] Bug fix: Notify remote backends of their users removed from conversation when reachable again (#3537) * Formatting * Test utilities for changing a conv name * Add a test confirming the bug report * An action to enqueue notifications concurrently * Enqueue member removal notification for remotes * Add a changelog * Test case formatting * Migrate test roleUpdateWithRemotesUnavailable * Migrate test putReceiptModeWithRemotesOk * Migrate test putReceiptModeWithRemotesUnavailable * Migrate test testRoleUpdateWithRemotesOk * Migrate test roleUpdateRemoteMember * Migrate test putQualifiedConvRenameWithRemotesUnavailable This one is already covered by testSynchroniseUserRemovalNotification * Migrate test putQualifiedConvRenameWithRemotesOk * Migrate test deleteLocalMemberConvLocalQualifiedOk * Migrate test deleteRemoteMemberConvLocalQualifiedOk * Migrate test deleteUnavailableRemoteMemberConvLocalQualifiedOk * Add the copyright header to a test module * Move a test utility (allPreds) * Test utility: create a team with members * Migrate test testAccessUpdateGuestRemoved * Migrate test messageTimerChangeWithRemotes * Migrate test messageTimerUnavailableRemotes * Migrate test testAccessUpdateGuestRemovedRemotesUnavailable * Migrate test accessUpdateWithRemotes * Migrate test testAddRemoteMember * Migrate test testDeleteTeamConversationWithRemoteMembers * Migrate test testDeleteTeamConversationWithUnavailableRemoteMembers * Move a test utility (assertLeaveNotification) * Migrate test "POST /federation/leave-conversation : Success" * Migrate test "POST /federation/on-user-deleted-conversations : Remove deleted remote user from local conversations" * Migrate test updateConversationByRemoteAdmin * Tests: support giving a role when adding * Use cannon API for notifications when possible * Use startDynamicBackends when possible * Fix assertion * Migrate test testAddRemoteUsersToLocalConv * Test add member endpoint at version 1 * Add return value to enqueueNotification * Use cannon assertions in offline backends test * Check that remote notifications are received * Test removal of users from unreachable backends * Use correct domains for default backends Taking the domains in the `backendA` and `backendB` resources only works locally. * fixup! Use cannon assertions in offline backends test --------- Co-authored-by: Paolo Capriotti <[email protected]> Co-authored-by: Akshay Mankar <[email protected]> * WPB-4240: Migrate from swagger2 to openapi3 (#3570) --------- Co-authored-by: Igor Ranieri Elland <[email protected]> Co-authored-by: Igor Ranieri <[email protected]> * Remove mocked MLS member add test * Resolve conflict in pregenerated swagger * Remove MLS end2end tests --------- Co-authored-by: Stefan Matting <[email protected]> Co-authored-by: Owen Harvey <[email protected]> Co-authored-by: Akshay Mankar <[email protected]> Co-authored-by: Marko Dimjašević <[email protected]> Co-authored-by: fisx <[email protected]> Co-authored-by: Igor Ranieri Elland <[email protected]> Co-authored-by: Leif Battermann <[email protected]> Co-authored-by: Jappie Klooster <[email protected]> Co-authored-by: Leif Battermann <[email protected]> Co-authored-by: Thomas Belin <[email protected]> Co-authored-by: Sven Tennie <[email protected]> Co-authored-by: Arthur Wolf <[email protected]> Co-authored-by: Igor Ranieri <[email protected]>
1 parent c2b0c81 commit 40a4adc

File tree

272 files changed

+2303
-5004
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

272 files changed

+2303
-5004
lines changed

changelog.d/1-api-changes/WPB-3611

-5
This file was deleted.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Remove de-federation (to avoid a scalability issue).

changelog.d/3-bug-fixes/WPB-4629

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Fixed add user to conversation when one of the other participating backends is offline
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
This fixes a bug where a remote member is removed from a conversation while their backend is unreachable, and the backend does not receive the removal notification once it is reachable again.

changelog.d/4-docs/WPB-4240

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Updating the route documentation from Swagger 2 to OpenAPI 3.

changelog.d/5-internal/WPB-4240

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Updating the route documentation library from swagger2 to openapi3.
2+
3+
This also introduced a breaking change in how we track what federation calls each route makes.
4+
The openapi3 library doesn't support extension fields, and as such tags are being used instead in a similar way.

changelog.d/6-federation/WPB-3611

-1
This file was deleted.

charts/fake-aws-s3/requirements.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
dependencies:
22
- name: minio
3-
version: 3.2.0
3+
version: 5.0.13
44
repository: https://charts.min.io/

charts/fake-aws-s3/values.yaml

+1-5
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,5 @@
1-
# See defaults in https://github.com/helm/charts/tree/master/stable/minio
1+
# See defaults in https://github.com/minio/minio/blob/RELEASE.2023-07-07T07-13-57Z/helm/minio/values.yaml
22
minio:
3-
mcImage:
4-
repository: quay.io/minio/mc
5-
tag: RELEASE.2021-10-07T04-19-58Z
6-
pullPolicy: IfNotPresent
73
fullnameOverride: fake-aws-s3
84
service:
95
port: "9000"

deploy/dockerephemeral/docker-compose.yaml

+1-2
Original file line numberDiff line numberDiff line change
@@ -57,8 +57,7 @@ services:
5757

5858
fake_s3:
5959
container_name: demo_wire_s3
60-
# image: minio/minio:RELEASE.2018-05-25T19-49-13Z
61-
image: julialongtin/minio:0.0.9
60+
image: minio/minio:RELEASE.2023-07-07T07-13-57Z
6261
ports:
6362
- "127.0.0.1:4570:9000"
6463
environment:

docs/src/how-to/install/sft.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ tags:
1818

1919
sftd:
2020
host: sftd.example.com # Replace example.com with your domain
21-
allowOrigin: webapp.example.com # Should be the address you used for the webapp deployment
21+
allowOrigin: https://webapp.example.com # Should be the address you used for the webapp deployment (Note: you must include the uri scheme "https://")
2222
```
2323
2424
In your `secrets.yaml` you should set the TLS keys for sftd domain:

docs/src/understand/configure-federation.md

+2-7
Original file line numberDiff line numberDiff line change
@@ -457,13 +457,8 @@ the sysadmin:
457457

458458
* [`PUT`](https://staging-nginz-https.zinfra.io/api-internal/swagger-ui/brig/#/brig/put_i_federation_remotes__domain_)
459459

460-
* [`DELETE`](https://staging-nginz-https.zinfra.io/api-internal/swagger-ui/brig/#/brig/delete_i_federation_remotes__domain_)
461-
- **WARNING:** If you delete a connection, all users from that
462-
remote will be removed from local conversations, and all
463-
conversations hosted by that remote will be removed from the local
464-
backend. Connections between local and remote users that are
465-
removed will be archived, and can be re-established should you
466-
decide to add the same backend later.
460+
* **NOTE:** De-federating (`DELETE`) has been removed from the API to
461+
avoid a scalability issue. Watch out for a fix in the changelog!
467462

468463
The `remotes` list looks like this:
469464

integration/default.nix

+6
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,11 @@
1515
, Cabal
1616
, case-insensitive
1717
, containers
18+
, cql
19+
, cql-io
1820
, cryptonite
1921
, data-default
22+
, data-timeout
2023
, directory
2124
, errors
2225
, exceptions
@@ -80,8 +83,11 @@ mkDerivation {
8083
bytestring-conversion
8184
case-insensitive
8285
containers
86+
cql
87+
cql-io
8388
cryptonite
8489
data-default
90+
data-timeout
8591
directory
8692
errors
8793
exceptions

integration/integration.cabal

+6-1
Original file line numberDiff line numberDiff line change
@@ -100,21 +100,23 @@ library
100100
Notifications
101101
RunAllTests
102102
SetupHelpers
103+
Test.AccessUpdate
103104
Test.AssetDownload
104105
Test.B2B
105106
Test.Brig
106107
Test.Client
107108
Test.Conversation
108-
Test.Defederation
109109
Test.Demo
110110
Test.Federation
111111
Test.Federator
112+
Test.MessageTimer
112113
Test.MLS
113114
Test.MLS.KeyPackage
114115
Test.MLS.One2One
115116
Test.MLS.SubConversation
116117
Test.Notifications
117118
Test.Presence
119+
Test.Roles
118120
Test.User
119121
Testlib.App
120122
Testlib.Assertions
@@ -147,8 +149,11 @@ library
147149
, bytestring-conversion
148150
, case-insensitive
149151
, containers
152+
, cql
153+
, cql-io
150154
, cryptonite
151155
, data-default
156+
, data-timeout
152157
, directory
153158
, errors
154159
, exceptions

integration/test/API/BrigInternal.hs

-19
Original file line numberDiff line numberDiff line change
@@ -99,25 +99,6 @@ updateFedConn' owndom dom fedConn = do
9999
conn <- make fedConn
100100
submit "PUT" $ addJSON conn req
101101

102-
deleteFedConn :: (HasCallStack, MakesValue owndom) => owndom -> String -> App Response
103-
deleteFedConn owndom dom = do
104-
bindResponse (deleteFedConn' owndom dom) $ \res -> do
105-
res.status `shouldMatchRange` (200, 299)
106-
pure res
107-
108-
deleteFedConn' :: (HasCallStack, MakesValue owndom) => owndom -> String -> App Response
109-
deleteFedConn' owndom dom = do
110-
req <- rawBaseRequest owndom Brig Unversioned ("/i/federation/remotes/" <> dom)
111-
submit "DELETE" req
112-
113-
deleteAllFedConns :: (HasCallStack, MakesValue dom) => dom -> App ()
114-
deleteAllFedConns dom = do
115-
readFedConns dom >>= \resp ->
116-
resp.json %. "remotes"
117-
& asList
118-
>>= traverse (\v -> v %. "domain" & asString)
119-
>>= mapM_ (deleteFedConn dom)
120-
121102
registerOAuthClient :: (HasCallStack, MakesValue user, MakesValue name, MakesValue url) => user -> name -> url -> App Response
122103
registerOAuthClient user name url = do
123104
req <- baseRequest user Brig Unversioned "i/oauth/clients"

integration/test/API/Galley.hs

+134-5
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ module API.Galley where
55
import Control.Lens hiding ((.=))
66
import Control.Monad.Reader
77
import Data.Aeson qualified as Aeson
8+
import Data.Aeson.Types qualified as Aeson
89
import Data.ByteString.Base64 qualified as B64
910
import Data.ByteString.Base64.URL qualified as B64U
1011
import Data.ByteString.Char8 qualified as BS
@@ -83,6 +84,21 @@ postConversation user cc = do
8384
ccv <- make cc
8485
submit "POST" $ req & addJSON ccv
8586

87+
deleteTeamConversation ::
88+
( HasCallStack,
89+
MakesValue user,
90+
MakesValue conv
91+
) =>
92+
String ->
93+
conv ->
94+
user ->
95+
App Response
96+
deleteTeamConversation tid qcnv user = do
97+
cnv <- snd <$> objQid qcnv
98+
let path = joinHttpPath ["teams", tid, "conversations", cnv]
99+
req <- baseRequest user Galley Versioned path
100+
submit "DELETE" req
101+
86102
putConversationProtocol ::
87103
( HasCallStack,
88104
MakesValue user,
@@ -296,12 +312,39 @@ getGroupClients user groupId = do
296312
(joinHttpPath ["i", "group", BS.unpack . B64U.encodeUnpadded . B64.decodeLenient $ BS.pack groupId])
297313
submit "GET" req
298314

299-
addMembers :: (HasCallStack, MakesValue user, MakesValue conv) => user -> conv -> [Value] -> App Response
300-
addMembers usr qcnv newMembers = do
315+
data AddMembers = AddMembers
316+
{ users :: [Value],
317+
role :: Maybe String,
318+
version :: Maybe Int
319+
}
320+
321+
instance Default AddMembers where
322+
def = AddMembers {users = [], role = Nothing, version = Nothing}
323+
324+
addMembers ::
325+
(HasCallStack, MakesValue user, MakesValue conv) =>
326+
user ->
327+
conv ->
328+
AddMembers ->
329+
App Response
330+
addMembers usr qcnv opts = do
301331
(convDomain, convId) <- objQid qcnv
302-
qUsers <- mapM objQidObject newMembers
303-
req <- baseRequest usr Galley Versioned (joinHttpPath ["conversations", convDomain, convId, "members"])
304-
submit "POST" (req & addJSONObject ["qualified_users" .= qUsers])
332+
qUsers <- mapM objQidObject opts.users
333+
let path = case opts.version of
334+
Just v | v <= 1 -> ["conversations", convId, "members", "v2"]
335+
_ -> ["conversations", convDomain, convId, "members"]
336+
req <-
337+
baseRequest
338+
usr
339+
Galley
340+
(maybe Versioned ExplicitVersion opts.version)
341+
(joinHttpPath path)
342+
submit "POST" $
343+
req
344+
& addJSONObject
345+
( ["qualified_users" .= qUsers]
346+
<> ["conversation_role" .= r | r <- toList opts.role]
347+
)
305348

306349
removeMember :: (HasCallStack, MakesValue remover, MakesValue conv, MakesValue removed) => remover -> conv -> removed -> App Response
307350
removeMember remover qcnv removed = do
@@ -342,3 +385,89 @@ getConversationCode user conv mbZHost = do
342385
& addQueryParams [("cnv", convId)]
343386
& maybe id zHost mbZHost
344387
)
388+
389+
changeConversationName ::
390+
(HasCallStack, MakesValue user, MakesValue conv, MakesValue name) =>
391+
user ->
392+
conv ->
393+
name ->
394+
App Response
395+
changeConversationName user qcnv name = do
396+
(convDomain, convId) <- objQid qcnv
397+
let path = joinHttpPath ["conversations", convDomain, convId, "name"]
398+
nameReq <- make name
399+
req <- baseRequest user Galley Versioned path
400+
submit "PUT" (req & addJSONObject ["name" .= nameReq])
401+
402+
updateRole ::
403+
( HasCallStack,
404+
MakesValue callerUser,
405+
MakesValue targetUser,
406+
MakesValue roleUpdate,
407+
MakesValue qcnv
408+
) =>
409+
callerUser ->
410+
targetUser ->
411+
roleUpdate ->
412+
qcnv ->
413+
App Response
414+
updateRole caller target role qcnv = do
415+
(cnvDomain, cnvId) <- objQid qcnv
416+
(tarDomain, tarId) <- objQid target
417+
roleReq <- make role
418+
req <-
419+
baseRequest
420+
caller
421+
Galley
422+
Versioned
423+
( joinHttpPath ["conversations", cnvDomain, cnvId, "members", tarDomain, tarId]
424+
)
425+
submit "PUT" (req & addJSONObject ["conversation_role" .= roleReq])
426+
427+
updateReceiptMode ::
428+
( HasCallStack,
429+
MakesValue user,
430+
MakesValue conv,
431+
MakesValue mode
432+
) =>
433+
user ->
434+
conv ->
435+
mode ->
436+
App Response
437+
updateReceiptMode user qcnv mode = do
438+
(cnvDomain, cnvId) <- objQid qcnv
439+
modeReq <- make mode
440+
let path = joinHttpPath ["conversations", cnvDomain, cnvId, "receipt-mode"]
441+
req <- baseRequest user Galley Versioned path
442+
submit "PUT" (req & addJSONObject ["receipt_mode" .= modeReq])
443+
444+
updateAccess ::
445+
( HasCallStack,
446+
MakesValue user,
447+
MakesValue conv
448+
) =>
449+
user ->
450+
conv ->
451+
[Aeson.Pair] ->
452+
App Response
453+
updateAccess user qcnv update = do
454+
(cnvDomain, cnvId) <- objQid qcnv
455+
let path = joinHttpPath ["conversations", cnvDomain, cnvId, "access"]
456+
req <- baseRequest user Galley Versioned path
457+
submit "PUT" (req & addJSONObject update)
458+
459+
updateMessageTimer ::
460+
( HasCallStack,
461+
MakesValue user,
462+
MakesValue conv
463+
) =>
464+
user ->
465+
conv ->
466+
Word64 ->
467+
App Response
468+
updateMessageTimer user qcnv update = do
469+
(cnvDomain, cnvId) <- objQid qcnv
470+
updateReq <- make update
471+
let path = joinHttpPath ["conversations", cnvDomain, cnvId, "message-timer"]
472+
req <- baseRequest user Galley Versioned path
473+
submit "PUT" (addJSONObject ["message_timer" .= updateReq] req)

integration/test/API/GalleyInternal.hs

-9
Original file line numberDiff line numberDiff line change
@@ -51,12 +51,3 @@ getFederationStatus user domains =
5151
submit
5252
"GET"
5353
$ req & addJSONObject ["domains" .= domainList]
54-
55-
deleteFederationDomain ::
56-
( HasCallStack
57-
) =>
58-
String ->
59-
App Response
60-
deleteFederationDomain domain = do
61-
req <- rawBaseRequest OwnDomain Galley Unversioned $ joinHttpPath ["i", "federation", domain]
62-
submit "DELETE" req

0 commit comments

Comments
 (0)