|
1 | 1 | # Waku's Requirements on Codex |
2 | 2 |
|
3 | | -## Publish Large Messages - Uploader is online |
| 3 | +## Message Archival |
4 | 4 |
|
5 | 5 | To be used for messages archival in Chat SDK, Qaku, opchan, etc. |
6 | | -It assumes that a special user (admin) regularly bundles messages and pushes them to an external system. |
7 | | -It then pushes the CID (or any other reference to retrieve the bundle) over Waku. |
| 6 | +It assumes that a special user (admin, referred to as "original uploader") regularly bundles messages and pushes them to an external system. |
| 7 | +It then pushes the reference to the bundle over Waku. |
8 | 8 |
|
9 | | -New users retrieve and listen to new messages using Waku. |
10 | | -Thanks to SDS, they learn whether they miss messages, and if so, can proceed with retrieval from the latest bundle. |
| 9 | +New users retrieve and listen to new messages using Waku upon start up. |
| 10 | +Then, they may retrieve bundles, likely because the know they are missing message via SDS. |
11 | 11 |
|
12 | | -(clearly, spec is needed). |
| 12 | +The original uploader is the one to determine durability, aka, relevance of data over time. |
| 13 | +It is application specific (eg until a Q&A is completed), and not related to users having downloaded the data. |
13 | 14 |
|
14 | | -### Functionality |
| 15 | +Which means it's a scenario where: |
| 16 | + |
| 17 | +- Time from upload to retrieval is **not** critical (latest messages are available on Waku) |
| 18 | +- Several users can seed and download the bundle. |
| 19 | + |
| 20 | +This is very similar to BitTorrent integration in Status. I need to find specs to be more explicit about difference. |
| 21 | +Some notes on BitTorrent integration in Status (AFAIK, asking @osmaczko for help): |
| 22 | + |
| 23 | +1. Known issue is that the bundle is very large, and hence consumes a lot of bandwidth. I don't know if the bundle is "updated" or overridden. |
| 24 | + On Waku app side, we need to be open to have one large bundle vs a series of manageable bundle. The latter offers more flexibility such as attaching bloom filters, |
| 25 | + for selective download. |
| 26 | +2. The bundle download is indiscriminate, meaning every user will download it at some point, with SDS, we can do something smarter |
15 | 27 |
|
16 | | -1. Ability to transfer a message of 1MB or more between two or more nodes. |
17 | | -2. Message's CID is less than 100kB. |
| 28 | +Also note (more of a personal opinion), usage of BitTorrent/webtorrent could be an acceptable starting point. |
18 | 29 |
|
| 30 | +### Functionality |
| 31 | + |
| 32 | +1. Ability to transfer a bundle of 1MB or more between two or more nodes. |
| 33 | +2. Reference to bundle is 50kB or less. |
19 | 34 |
|
20 | 35 | ### Usability |
21 | 36 |
|
22 | | -1. Developer can implement upload feature with 10 lines of code or less. |
23 | | -2. No configuration is necessary. |
| 37 | +1. Developer can implement feature with 10 lines of code or less. |
24 | 38 |
|
25 | 39 | ### Reliability |
26 | 40 |
|
27 | 41 | 1. Download operation can be resumed. |
28 | 42 | 2. Upload operation can be resumed. |
29 | | -3. Uploader can be expected to be online when user are downloading. |
| 43 | +3. As long as original developer is online, bundle should be retrievable. |
| 44 | +4. As long as N out of M users are online, bundle should be retrievable. |
30 | 45 |
|
31 | 46 | ### Performance |
32 | 47 |
|
33 | | -None |
| 48 | +1. Time between bundle uploaded, and retrieved by users can be in the span of minutes and hours (we assumes messages are available in Waku store for several hours). |
| 49 | +2. The burden of re-upload is shared by users. |
34 | 50 |
|
35 | 51 | ### Supportability |
36 | 52 |
|
37 | 53 | 1. Library for Browser applications. |
38 | 54 | 2. Library for Nim desktop applications. |
39 | 55 | 3. Library for Nim mobile applications. |
| 56 | +4. Most users may be behind NAT routers and other domestic network setup. |
40 | 57 |
|
41 | | -**+ (Privacy, Anonymity, Censorship-Resistance, Deployments)** |
| 58 | +### + (Privacy, Anonymity, Censorship-Resistance, Deployments) |
42 | 59 |
|
43 | 60 | 1. The unavailability of a static host (IP, DNS) does not prevent a user to upload or download (censorship-resistance). |
44 | | -2. TODO (privacy) |
| 61 | +2. A participant cannot determine original uploader's PII (anonymity). |
45 | 62 |
|
46 | | -## Publish Large Messages - Uploader is offline |
| 63 | +## Large File Transfer |
47 | 64 |
|
48 | | -To be used for |
| 65 | +To be used when 2 users or more, are transferring a large payload. This may be a large image or video in a private chat. |
| 66 | +Or it could be a llm prompt that returns a large image or video. |
49 | 67 |
|
50 | | -- large messages transfers (such as images, videos, audio) in Chat SDK, Opchan, etc. |
51 | | -- Enhancement of message archival (uploader does not need to be online for messages to be retrieved). |
| 68 | +Due to the broadcast nature of Waku, it would hog too much bandwidth if every large file sent between users where sent over Waku. |
52 | 69 |
|
53 | | -Builds on [Publish Large Messages - Uploader is online](#publish-large-messages---uploader-is-online) |
| 70 | +In terms of durability, it can be assumed that once all participants have downloaded the payload, it does not need to be retrievable anymore. |
| 71 | +It should also be assumed that the users may not be online at the same time (mobile). |
| 72 | +There is more expectation on timeliness of retrievability, as one would want to be able to download seconds after the upload happened. |
54 | 73 |
|
55 | 74 | ### Functionality |
56 | 75 |
|
57 | | -1. After upload, message is retrievable without sender being online. |
58 | | -2. Best effort in terms of message retention; expectations on restrictions are documented. |
| 76 | +1. Ability to transfer a payload of 1MB or more between two or more peers. |
| 77 | +2. Reference to payload is less than 50kB. |
59 | 78 |
|
60 | 79 | ### Usability |
61 | 80 |
|
62 | | -1. Receiver can download the large message, even if sender is offline, as long as they get the CID out-of-band. |
| 81 | +1. Developer can implement feature with 10 lines of code or less. |
63 | 82 |
|
64 | 83 | ### Reliability |
65 | 84 |
|
66 | | -1. Uploader may be offline when receiver is retrieving the large message. |
| 85 | +1. Download operation can be resumed. |
| 86 | +2. Upload operation can be resumed. |
| 87 | +3. Payload should be retrievable even if original uploader goes offline. |
| 88 | +4. Once all recipients have downloaded the payload, there is no more durability expectations. |
67 | 89 |
|
68 | 90 | ### Performance |
69 | 91 |
|
70 | | -None |
| 92 | +1. Payload download should start within seconds of the upload start. |
71 | 93 |
|
72 | 94 | ### Supportability |
73 | 95 |
|
74 | 96 | 1. Library for Browser applications. |
75 | | -2. Library for Nim applications. |
76 | | - |
77 | | -### + (Privacy, Anonymity, Censorship-Resistance, Deployments)** |
78 | | - |
79 | | -TODO (privacy) |
80 | | - |
81 | | -## Publish Large Messages - Retention is guaranteed |
82 | | - |
83 | | -In this scenario, the uploader wants to ensure the data is persisted and is willing to pay for it. |
84 | | -This may be a Qaku Q&A admin, a opchan cell owner or Status Communities owner. |
85 | | - |
86 | | -Builds on previous requirements. |
87 | | - |
88 | | -### Functionality |
89 | | - |
90 | | -1. Uploader may pay for large message storage to have guaranteed retention. |
91 | | - |
92 | | -### Usability |
93 | | - |
94 | | -1. Receiver can download the large message, even if sender is offline, as long as they get the CID out-of-band. |
95 | | -2. Receiver does not need to pay to retrieve the large message. |
96 | | - |
97 | | -### Reliability |
98 | | - |
99 | | -1. Uploader may be offline when receiver is retrieving the large message. |
100 | | -2. Uploader is guaranteed a period of retention for a given price. |
101 | | - |
102 | | -### Performance |
103 | | - |
104 | | -See previous requirements. |
105 | | - |
106 | | -### Supportability |
107 | | - |
108 | | -See previous requirements. |
| 97 | +2. Library for Nim desktop applications. |
| 98 | +3. Library for Nim mobile applications. |
| 99 | +4. Most users may be behind NAT routers and other domestic network setup. |
109 | 100 |
|
110 | | -### + (Privacy, Anonymity, Censorship-Resistance, Deployments)** |
| 101 | +### + (Privacy, Anonymity, Censorship-Resistance, Deployments) |
111 | 102 |
|
112 | | -See previous requirements. |
| 103 | +1. The unavailability of a static host (IP, DNS) does not prevent a user to upload or download (censorship-resistance). |
| 104 | +2. An external observer cannot tie the PIIs of the uploader and downloaders of one payload; |
| 105 | + it is assumed that the reference to the payload (eg, CID) is not leaked outside the participants. |
0 commit comments