Gtk Port: New editor implementation

[TheJackiMonster]

I think the new implementation of the editor (which is probably the most important GUI element in Manuskript) has a lot of potential to improve things for a lot of users. As far as I'm aware there have been requests in the past to allow more features in terms of formatting text, making it WYSIWYG and improving performance for spell or grammar checking.

However this makes it kind of complicated to implement. For one reason because GTK only provides a rather basic GtkTextView which utilizes a GtkTextBuffer that allows certain formatting options but not everything you could think of. So I think before doing anything abnormal in terms of features, we should focus on getting existing features done and having better performance as well as integration for plugins like spellcheckers.

Using a GtkTextBuffer provides two major options doing formatting like for example underlining spelling mistakes:

• You can create a table of GtkTextTag objects defining specific formatting attributes that apply to a selected portion of the text. You can think of it as abstracting a markup language. However this comes with disadvantages of loosing fine control and mixing attributes gets more difficult.
• You can insert markup directly into the buffer which allows full usage of Pango markup. It is still limited in terms of features but you have very much control and it allows mixing attributes however you like (as long as you implement some own function to apply it properly). The disadvantage is that you have only the insert_markup() method to do everything.

I personally think for the best experience we pretty much have to use the Pango markup option. Because even though I tried using the tag system first, I ran into issues very fast that could not really be solved without extremely ugly workarounds.

Still I think this will cost quite some time and effort to get everything right. Mostly because of the ability to edit text in the editor. Not just that but also editing text that belongs to different scenes or chapters while being displaying as a complete document. Spellcheckers need to process singular words which implies we have to split the whole text into pieces, process each piece, stitch them together with proper markup added between and compile it into one complete string to insert.

In current state I've started by appending scenes one for one into the buffer and I think this approach should even go further. Ideally we want a system that can stream text from the scene files into the spellchecker into some sort of queue that puts everything together into the buffer. So the user would be able to edit the text while the format corrects itself according to changes and while we are potentially loading additional text pieces into the buffer at another position. We would also need to convert each change in the buffer again removing formatting from spellcheckers into local changes of scene text to then store it at the right place when a user saves the project.

In other words the representation of text in the buffer needs to be treated as completely different instance than the actual data model with some sort of translation unit between that while update local changes in either of them into local changes of the other. It's gonna be a beast to implement but probably worth it. Because if we do that right, we can probably deal with a lot of feature requests way easier down the line.

[peter88213]

Congratulations on the progress made with the GTK port!

I just took a look at Pango, and it looks really good, especially since you can probably use standard Markdown routines for data transfer due to the HTML-like syntax.

Over the past weeks, as part of my own project, I have been working intensively with the tkinter text widget, which basically contains quite a powerful tag system. However, since my data model is XML-based (similar to XHTML), I have to convert in both directions, which leads to unexpected error-proneness and high testing effort. It's amazing what users can enter into the editor, which then has to be assembled into well-formed XML. Something like Pango could be a help here and might reduce the risk of errors and the effort involved, but I have now settled on tkinter. Incidentally, tkinter has proven to be very stable and fit for everyday use, so no complaints.

Incidentally, the reference to Pango markup led me to wonder whether it would be advantageous for Manuskript to switch from Markdown to an XML-based file format (for instance, the ability to mark text passages as foreign language for the spell checker is a decisive argument for me). If this is ever considered, I could provide an exemplary XML data format that may cover a portion of Manuskript's functionality. I have a DTD, a CSS stylesheet, and experimental XSLT stylesheets, as well as Python routines for reading and writing that are based entirely on the standard library.

[TheJackiMonster]

@peter88213 Theoretically Manuskript does not enforce Markdown from my understanding even though in current state I assume only Markdown is supported as format for all outline files. But the files are generally a custom MultiMarkdown format which even mentions a type attribute, generally specifiying "md" for Markdown.

So in theory a user could also write XML-based files or even HTML or LaTeX files and add them to their project. Because in the final export process all files could be converted using Pandoc or other tools before stitching them together. At least that is the idea.

For the editor we probably go with Pango markup. But that will mostly be for adding visual hints to certain elements (for example in case of using the spellcheckers). So in case we want to support more formats, there will be converters needed and conversion tools to the markup for the editor. The new approach is to make everything modular and allow new features like this via plugins. So maybe we could build something to support data formats like the thing, you suggest.

I wouldn't say that's impossible. For now I'm still planning on the whole editor architecture which is why I opened the issue for my current ideas to implement it.

[peter88213]

@TheJackiMonster Yes, perhaps I didn't express myself clearly enough. If the existing native file format is suitable for displaying all the features of the new text editor, there may be no reason to switch to a different format. As far as I understand, the old Qt-based text editor is essentially a plain text editor, whereas the new GTK-based editor is supposed to be a more powerful rich text editor.

By the way, an excellent FOSS implementation of a GTK-based word processor, which I have been using myself for a long time, is Zim Desktop Wiki. There are plugins and spell check, and overall it works smoothly for me.

[TheJackiMonster]

@peter88213 The Qt version is mostly a text editor but at the same time, not really. That's the main issue. You can select a whole chapter or the main book outline folder and the editor can actually display a concatenated version of the whole document with all scenes inside the selected element, adding formatted titles and such. Also you get colored and underlined hints for spellchecking and word selection on right-click with a custom context menu depending on interaction options.

It's essentially using markdown elements but in very few cases that it feels like a text editor. But it's never displaying the original file contents. So that is why we need some element in between displaying text and making changes to the actual files.

[peter88213]

Virtually all open source writing software uses the editor control of a GUI framework. I wonder whether these widgets can be powerful enough to handle really extensive novels.

In 2018, I evaluated Manuskript from the source code on Windows 10. It was able to load my 250-page work in progress, and the export function also worked satisfactorily. However, it was not possible to work smoothly; the program repeatedly slowed down to the point of freezing and also crashed on several occasions. Although the concept behind the software is compelling, I couldn't bring myself to work with it seriously.

When I look at other writing programs with a similar concept, such as yWriter and novelWriter, I notice that their editors work scene by scene. Even the commercial Scrivener couldn't do anything different on Windows at first; the implementation of its composite view, called "scrivenings" or so, seemed to have delayed the release of the current Windows version by years. The most successful writers' program in the German-speaking world, Papyrus Author, appears to be based on a word processor developed in-house.
I actually wonder whether the GTK editor control ultimately has similar limitations to that of Qt, and whether it is even worthwhile investing so much effort in customizing it.

For my own writing, I still rely on OpenOffice, which has always proven itself with very large text documents, and, for my own use, I have built a novel management program around it using Python. LibreOffice is even faster with extremely large documents in its 64-bit version, but unfortunately doesn't work with the Duden spellchecker extension.

But I too was captivated by the idea of tinkering with a rich text word processor, which is why, as mentioned above, I created a “distraction-free writing mode” based on the tkinterText widget. Since the Python bindings communicate with the tk substructure via control strings, I kept my expectations low from the outset and only load the text in sections. But that's not a problem, because I can scroll through the sections very quickly using Ctrl-PgUp and Ctrl-PgDn. Overall, it runs very smoothly and, compared to OpenOffice, is truly distraction-free and really swift. A spell check and a global search/replace function would be feasible, but in order to avoid distractions while writing, I have deliberately decided not to implement them. When it comes to editing and formatting, I have the full power of Office word processing at my disposal.

So much for the exchange of ideas. I am really curious to see which direction the Manuskript software will take. In my opinion, however, its main appeal lies in the organization, i.e., in conveying the "big picture" of a novel.