Current Project Roadmap

[arch1t3cht]

To avoid repeating myself in other issue threads, I'll describe my current large-scale plans for the project here.
This is in no way set in stone and these plans can change in the future; I am mainly writing this to set the correct expectations for contributors and issue authors.

See also: arch1t3cht#177
For more specific short-term plans, see #475 .

The Problems

I have taken over upstream Aegisub development after it went 10 years without any official release.
But this does not mean that no Aegisub development has happened in these 10 past years: There have been multiple forks (some overlapping, some diverging) of the program, various changes and big internal refactors, and some third-party tooling (DependencyControl) has been made semi-official.

In particular, before the release of Aegisub 3.4.0 I maintained my own personal fork and added multiple big features there.

While Aegisub is in a much, much better state now than it was 7 years ago, all of this history has left the ecosystem somewhat messy and fractured in places:

• Some of the changes and refactors since 3.2.2 have caused regressions. Many of these have been fixed, but some still aren't. (e.g. Inconsistency When Snapping Markers to Cursor Time #421, Aegisub 3.4.2 Audio Spectrum mode scale clipping? #455)

• The vast majority of advanced users still use arch1t3cht/Aegisub due to the added features (perspective tool, line folding, new video providers) that have not yet been merged upstream. This fractures the ecosystem and means that fewer people are testing nightly upstream releases.

• DependencyControl is indispensable for advanced users, but currently has many problems. At the moment it's available unofficially in Windows builds (i.e. disabled by default in the installer) due to the security risks. The problems include:

  • Feeds are discovered recursively, so only one feed in the entire tree needs to be compromised in order for the entire tree to be compromised. At the moment we have a policy of not allowing transitive knownFeeds and forcing all new script repositories to be moved to the TypesettingTools organization, but this is purely a convention and several old feeds point to external repositories.
  • DependencyControl is unmaintained and its latest master is broken for reasons nobody has looked into yet. (Instead, the latest release tag is shipped to users.) There is an unfinished 0.7 branch switching to an SQL configuration whose state is also unknown.
  • DependencyControl drastically increases the loading time of scripts using it for reasons that are not yet fully known. (According to folklore it's related to BadMutex, but when I investigated this it seemed like this may not be the whole story.) This in turn increases the startup time of Aegisub itself.
  • DependencyControl is currently only shipped with Aegisub on Windows. It is not yet clear how it should interact with system package managers on Linux (Self-updating on systems with a package manager (Linux) DependencyControl#12).
  • DependencyControl uses several C libraries loaded via luajit's FFI for actions that aren't otherwise possible (e.g. downloading scripts from the internet). This can cause some issues in and of itself (DependencyControl fails to load on Windows if Aegisub's automation path contains special characters #458), but also makes it harder to install DependencyControl on some platforms: In particular, there is no prebuilt release for Apple Silicon, so users need to compile it themselves there (and then manually install it, which is also quite tedious).

• There are also other popular (indispensable to typesetters) automation scripts and libraries (Yutils and ILL) that use C functions and libraries via FFI, in particular to obtain text metrics and shape data. Again, this creates headaches on some unsupported platforms like Apple Silicon.

• Many (many) third-party automation scripts have been written to help with advanced use cases like typesetting. Many of these scripts need to parse ASS override tags (and drawings) to some extent, which they do in many (many) different ways. Many scripts use their own ad-hoc partial parsing (often with Lua patterns), others use some third-party parsing library like ASSFoundation or ILL.

  None of these parsing solutions fully match the actual ASS format¹ and all of them have parsing bugs. Some are less severe than others, but either way the consequence is that advanced users need to not only make sure that their syntax parses correctly on libass and VSFilter, but also that it conforms to the inaccurate mental models of the format of all the authors of the scripts they may want to use.

  Aegisub's itself is not exempt from this either. Its dialogue parser has many issues itself (e.g. Setting color within a transform tag will place it outside of the transform #12) and, even worse, its file parser will reject "valid"² (though maybe unconventional) files.

  There is a discussion to be had here about how Postel's law was a mistake and implementations should reject nonconforming files, but in this situation all files are in some sense conforming, and it's not Aegisub's place (and much less the place of third-party scripts) to dictate some more narrow format. Aegisub could definitely use some form of "compliance checker" one day, but refusing to open "valid" files is not acceptable.

The Goals

My current highest priority for Aegisub development is addressing these problems (roughly in the above order of priority).
More specifically, this means

• Fixing any regressions from 3.2.2
• Migrating over the remaining features from arch1t3cht/Aegisub.
  The reason why this is taking "so long" is that the remaining big features (in particular line folding and the perspective tool) entail storing additional information in the subtitle file. In order to avoid breaking backwards compatibility in the future, this needs to be done correctly once and for all. There are certain problems with the current implementations that I'd like to address when migrating the features, but this needs to be done carefully. See Standardized format for visible third-party data in lines libass/libass#906 for example.
• Somehow improving the DependencyControl situation. There is no clear path for this yet, but it'd probably entail
  • Figuring out and fixing the current state of the repository
  • Adding first-party APIs in Aegisub for the features that DependencyControl currently uses FFI for
  • Offloading some not yet decided amount of DependencyControl's logic to Aegisub
  • Improving DependencyControl's feed system somehow (?)
• Similarly, adding first-party APIs in Aegisub to obtain accurate³ text metrics and shape data (see [WIP/POC] Add API to obtain metrics and shape data libass/libass#856)
• Similarly, implementing a correct subtitle and dialogue parser (possibly just partially copy-pasting libass code?) and exposing it to Lua.

The later points here are more long-term goals, but the first two have some of the highest priority.

What this Means for Contributors

Once again, this roadmap is not set in stone and is certainly not the only thing I am working on. I will still work on bugfixes and simple features and review small pull requests, just like I've done so far. (This has been the plan for a long time, I'm just typing it out now.)

The main thing I'd like to communicate here is that as long as I'm still working on the above points (at the very least the first two), I'm unlikely to work on any big feature requests or pull requests. I know that this will delay "progress" in some sense, but I would like to first get the current situation in order (or something resembling it) before starting further big projects.

For issue authors, this means that bigger feature requests will probably not be addressed for quite some time. Of course, you're still welcome to open them, I just want to set the correct expectations. And, as said above, I will still work on bugfixes. This is just about big feature requests.

Similarly, for contributors, this means that I will probably not merge any pull requests adding big new features before resolving this situation. This is the main reason why I'm writing out this post; I'd like avoid a situation where someone works on a big new feature and then gets frustrated when their PR is ignored for a long time. Once again, you're still very welcome to open any and all PRs - I won't close them outright just because they're not on the current roadmap - I just want to set the correct expectations. And, just like above, bugfixes are still very welcome. If you're unsure whether your PR fits the roadmap, the best thing to do is open an issue about it before spending time to work on the code. (This is true for almost all open source projects! Communication is crucial to prevent frustration on both the contributor and maintainer sides, which is exactly why I'm writing this issue.)

Footnotes

1. Experienced users will know that there is no such thing as an ASS specification. Still, the format is "specified" in the sense that it's defined by a reference implementation. A more accurate way to word the problem would be that none of these scripts match VSFilter's or libass's parsing. ↩

2. Seeing as modern VSFilter should never outright crash when loading a file, virtually every file should be considered a "valid" ASS file. At most, the file might not contain any valid events (or styles/headers/attachments) and be effectively empty. ↩

3. aegisub.text_extents exists but is not accurate on non-Windows ↩