The Saga Software Project
Revisting Android/Mobile support (follow up from #1307) #2975
Replies: 2 comments 8 replies
|
I actually looked into switching to PySide6 when I ported from PyQt5, but there are real differences that matter between the two. novelWriter has some PyQt-specific adaptation, and the app is complex enough that the way memory management and multi-threading is implemented is a bit too closely tied into PyQt6. Also, while PySide6 is maintained by Qt, it gets less maintainer attention than PyQt6 does. So I stuck with PyQt6. If you need PySide6, the app needs to be ported, because I don't want to add an abstraction layer on top of the framework because of the above mentioned reasons. So then it is a matter on how that project would be driven. If it's a maintained fork with synced releases, then I can point the website to it, which is also what I want to do with MacOS releases where there is no longer an official release. An alternative is to finally take the step and move this project into a GitHub org, which allows adding maintainers and multiple projects. But if it's just going to be me in the end doing most of it, I don't really want to commit to more work. A GitHub org gives much more granular control (it's one of the things I do in my day job anyway, so it's easy peasy to manage). |
|
Thank you for the detailed reply, Vkbo, It's also good to know about the PyQt6 specific memory/threading ahead of time, to be clear on directions first: I would commit to a full PySide6 port, not the abstraction layer on top, mainly for the same reasons you gave, that's where I'd start first getting nw running on PySide6. One thing I'd like to ask before hand, ; Do you have a rough sense of whether the memory/threading coupling is concentrated in a specific area (e.g. the project indexer, autosave, a particular worker thread), or whether it's spread more generally through how the app uses Qt objects and signals? If it's concentrated, then it's a normal contained rewrite to PySide6. If it's spread throughout, that would change how I would want to test migration to PySide6. My plan right now is something like; Getting a PySide6 port passing on desktop first, with no Android-specific changes mixed, so any breakage latter is clearly attributable to the swap and not platform code. Once that's stable I would move onto the more Android-specific work (storage, touch UI). As for structure, a fork with synced releases works well for me too, I don't have anyone else lined up yet, it's just me at the moment, so a full GitHub org feels premature. Happy to revisit that down the line if this actually grows a team, but for now the fork model fits what I can realistically commit to, and what you said you're comfortable with. Appreciate you taking the time to lay this out, and being open to it at all. I'll look more into PySide6 and PyQt6 to see where there memory-management and muit-threading differ along side nw's handling of mm, rather than just more questions. :) |
|
The GitHub org part is based on more than just this request. But before I go down that path I want to see if anyone actually sticks around long enough to warrant it. People come and work on this project for a time, and then disappear. I guess that's just the nature of open source, but I have learned the hard way to not promise to take on responsibility for the things contributors leave behind, whether it is translation work, packaging options, or other features that I wouldn't have otherwise added. Of course, sometimes people just help with stuff that needs doing, which is a different matter. In any case, my point is that if you want to drive the Android/iOS support, that's great, and I'm happy to assist where I can. What I learned from my own attempt at PySide was that some of my memory management work did not port over cleanly. I'm happy to go more into detail, and I worked quite a lot with it this summer and fixed many more issues. It's perfectly possible that my recent changes also port cleanly to PySide. But last time I tried, Qt/PySide segfaulted. There are some fundamental challenges with ownership of objects between C++ and Python when working with Qt which basically boils down to two competing garbage collectors with different strategies. The issues are similar for both PyQt and PySide, but the bindings are generated with different tools (sip vs shiboken) so they aren't identical. Another challenge has to do with multi-threading, implemented using the Qt threading module, which poses some problems for Python which doesn't expect to act on objects belonging to multiple threads. There are also some bugs here and there in both Qt, PyQt and PySide. There are some adaptations in novelWriter that handle past issues I've run into, which may not exist in PySide, or there may be different ones there. Some strategies in the code exist to work around known bugs. A common gotcha is that Python strings adapt to ASCII, UCS-2 or UCS-4 encoding depending on content, while Qt always uses UTF-16. This causes problems for 4 byte Unicode characters since string indices no longer match. There have been bugs in the Python bindings on the conversion between these in the past too. When I ported to PyQt6, I had to stop using the Qt regex tooling for instance, which may have been related to this (I never really found out). Good news is that the test suite is extensive, and should be able to catch a lot of these if something breaks. I'm sure there are things it doesn't catch though. |
|
Also worth adding: I have used Claude Code a fair bit to track down and find some of these difficult to handle bugs (it also introduced some in the process so it needs to be used very carefully) so I have quite few markdown files locally that it uses as its knowledge base. I also have it set up to scan the Qt6 source code to figure out how the library works under the hood, which helps a lot to resolve weird bugs. The code base contains well over 2 million lines of code, so navigating it manually is very time consuming (I've done that many times too). Not sure of you use tools like that, but if you do, I can probably share some files that will jump start it in trying to resolve differences between the binding libraries. |
|
Hey folks, I'd like to help out with both topics. I have a couple of hours of capacity I can put into novelWriter dev and my background and most of my time is focused on building apps for Apple platforms and some cross-platform stories.
Regarding macOS distribution of the current app, I'm more than happy to setup a fork that (with a few adjustments to the packaging) regularly builds notarised versions of the app via my Apple Developer program membership. Regarding mobile, a version of novelWriter which works on iPad/iOS/Android would be amazing! My feeling here is that we will have a much happier story if we choose a different path for those platforms. Given that novelWriter is a very text-focused experience, I would go for a solution allowing us to access the platform's native text input components (and try not to involve Qt at all). It will be a bit more work upfront e.g. to port the 'nvw project' framework to something like Kotlin Native (KMP is great these days) or Swift (it supports building libraries for Android) which will enable sharing common features while using each platform's first party UI library. Some help from LLMs can speed up the process. We also give ourselves some flexibility to choose which features make it to mobile and how to translate desktop-focused tools into something suited for small screen/touch-sometimes-keyboard form factors. For example, novelWriter for mobile could start as a companion to the desktop app for "writing on the go" and gradually gain more of the suite's functionality. The "core" and starting point would be implementing an SDK to read and write the project file structure (and links) in a reliable and reversible way. I'd be happy to setup a PoC if that sounds like something you'd be ok with. |
|
This all sounds great! As discussed elsewhere about macOS distribution, I think it may be time to move these repos over to a GitHub org I set up last year for the purpose. It would not only make it easier to add people with elevated access to the project, but also have a suitable place for related repos. It sounds like the mobile app should maybe be a separate code base? Could you email me via this temp email? I would like to discuss a few technical details with you offline. vkbo.alive305@passmail.net. |
|
This does sound great. Especially since Kotlin would mean getting logic to both platforms and then from there it would just be ui on top, honestly I didn't even think about cutting qt out of the picture for mobile... In any reason, it sounds good to me, and it would honestly be a better approach down the road. Rather then a PySide6 port and migration, since that was mainly about getting qt onto android specifically. Otherwise, happy to help. |
|
I'm happy to give you the access to set up a repo and run this project from within the new org, @kkostov. I've sent you an invite. (As you can see, the repo has moved.) Are you thinking to just support the same file format in a parallel writing app? Even if just the writing part is ported, the build and other more complex parts can stay in the desktop app. That's the bulk of the code really, aside from the GUI of course. The index is the other big component, but the index.json file in the project/meta folder isn't actually required to be maintained in a writing session. It has a version number, so it is possible to force a re-index run when the project is opened on a computer again by making sure the version number in the project XML file is bumped on each save. It is in theory also possible to strip down the main Python app and only run the core components in a library. I have tried to keep them separate. But it is tricky to get completely free of the PyQt dependency even in the core because it uses QtCore components also outside of the GUI code. |
That was indeed my thinking. e.g. the following user flow: As a novelWriter user, I would like to be able to make edits to my novel while on the go using my mobile device. At the very least, the mobile app should know how the project must be organised on the file system. What kind of edits can one do on mobile: I suggest to start with something scoped to text and format changes to existing files or creating a new structured file within the project (e.g. a new file in a folder, a new folder, scene, character etc). To enable this flow, folks would either have their project stored on a synced file location (e.g. iCloud, Dropbox, Google Drive, Proton Drive...etc) or are manually syncing the folder (e.g. by connecting their device via USB). A consequence of this is that the project may "sync" while the project is open so some care will have to put so the app doesn't accidentally loose/overwrite a piece of content. Another aspect will be support for the markdown formatting flavour so text can be rendered to and from raw content and look exactly the same as it does on desktop. It feels like folks would be distracted (or feel like something is broken) if the look and feel is too different. I imagine some may copy-paste text from other apps on their device, so the app will need a "paste and match style of the novel" style of integration with system text services. I would keep custom fonts out from the first release. With this approach, we are not going to share code with the existing codebase (apart from assets like images, logos etc). Project structure and format compatibility will be guarded more on unit test level e.g. the mobile app project should pass the same sorts of tests when any of the above functions are used. |
|
That's sensible. The document file format, as of 26.2, is a *.md file with a standard toml header ( The project file format is XML, and is a changing format as new features are added. Preserving compatibility there is trickier. The file format is properly versioned though, with version information headers. Keeping the two parsers in sync may require some more coordination so that new settings aren't silently dropped on write. |
Uh oh!
There was an error while loading. Please reload this page.
@vkbo @HeyMyian Hello :) tagging you both since you were part of this last round way back in 2023, on #1307
Wanted to check whether the situation has changed at all in the last couple years. Back then the summary was:
In principle possible, no capacity or Android knowledge on your end, but open to it if a contributor wants to drive it and maintain it separately.
Curious if that still holds?
I'd like to actually take a serious run at this, starting with Android, iOS later if Android goes well, and wanted to float a rough plan here first and see who's interested before committing any real time to it.
Why now? Well, the Qt/Python side of this has moved a lot since 2023. Qt for Python (PySide6) now ships official Android wheels and a pyside6-android-deploy CLI tool that handles packaging end-to-end (Qt 6.8+). And as of Qt 6.12, PySide6 has landed initial iOS support too, so the same codebase could plausibly target both down the line, once Android is solid.
The catch is, novelWriter is built on PyQt6, not PySide6, and PyQt6 has no real Android/iOS path. So I was thinking about Forking the project, migrate the fork from PyQt6 to PySide6 (or via qtpy for a compatibility layer) rework the UI for touch, menus, toolbars, and shortcuts are all built assuming mouse + keyboard right now.
Handle desktop↔mobile project syncing,
leaning toward not building anything custom here, just making sure novelWriter projects work cleanly over whatever generic sync the user already has (Syncthing, a cloud folder, git, the in-progress zip export) just so, if an existing project on desktop is moved to Android, the user can always update once they are back to desktop, I feel like this should be a key part that I have yet to fully highlight, this way Android is cross compatible with the desktop versions.
iOS is also a later phase, though PySide6 is newer and still has real gaps on ios (no simulator yet, no C extension packages on PyPI yet)
This would live as a separate fork/distribution rather than something that needs to merge into or be maintained by the main repo, to keep it off your plate.
I don't have Android dev experience myself (my background is more systems/low-level), so I'd genuinely welcome anyone who does wanting to weigh in helping, or just tell me where this plan is naive.
Mainly posting this to gauge: Is there still appetite for this, and is anyone interested in helping before I start fully looking into it?
P.S.
I love novelWriter, and I am really gratefully for what the software has given me as a long user. :)
All reactions