Consider alternatives to Flask

[Daniel-Mietchen]

I don't have concrete plans here at the moment, just bringing this up to have it on the radar.

There are various compilations that compare Flask to other Python-based web frameworks, e.g. here with Django and here with FastAPI.

[WolfgangFahl]

An upgrade to https://github.com/WolfgangFahl/pyFlaskBootstrap4 which is based on https://github.com/greyli/bootstrap-flask which is an improvement of the current https://pypi.org/project/Flask-Bootstrap4/ would IMHO be a good first step. Especially the use of Widgets and Server Side Event Handling might be an improvement for the UI.

[dpriskorn]

I don't have concrete plans here at the moment, just bringing this up to have it on the radar.

There are various compilations that compare Flask to other Python-based web frameworks, e.g. here with Django and here with FastAPI.

Speed is nice, but isn't WDQS more of a bottleneck?

Blazegraph takes seconds to return some queries while QLever finish in milliseconds.

[WolfgangFahl]

The example query: scholarly articles publish via proceedings (author part not even activated - takes 11.9 secs on wikidata query services when tested 2022-01-17
try it!

# Scholarly articles with full text
SELECT ?paper ?paperLabel ?publishedIn ?publishedInLabel ?event ?eventLabel ?fullText 
WHERE 
{
  ?paper wdt:P31 wd:Q13442814.
  ?paper rdfs:label ?paperLabel filter(lang(?paperLabel) = 'en').
  #OPTIONAL
  #{    
  #   SELECT (GROUP_CONCAT(?authorLabel) as ?authors) WHERE {
  #     ?paper wdt:P50 ?author.
  #     ?author rdfs:label ?authorLabel filter(lang(?authorLabel) = 'en').
  #   } GROUP BY ?paper
  #}
  ?paper wdt:P953 ?fullText filter (strends(str(?fullText), ".pdf" )). 
  ?paper wdt:P1433 ?publishedIn.
  ?publishedIn rdfs:label ?publishedInLabel  filter(lang(?publishedInLabel) = 'en' 
                                                    # && strstarts(?publishedInLabel,"Proceedings")
  ).
  #OPTIONAL {
    ?publishedIn wdt:P4745 ?event. # proceedings from
    ?event rdfs:label ?eventLabel filter(lang(?eventLabel) = 'en').                         
  #}
}

[WolfgangFahl]

https://qlever.cs.uni-freiburg.de/wikidata/ayZR1r doesn't like the query so i couldn't test it.

[fnielsen]

Bootstrap is more or less just visual changes. And I suppose that PyFlaskBootstrap4 is also "just" visual.

I do not see any reason to change to Django. We are not using ORM at the moment.

Most of our dynamic data comes from Javascript. The async of FastAPI could be usefull but I do not see how it can help currently.

[Daniel-Mietchen]

I don't have concrete plans here at the moment, just bringing this up to have it on the radar.
There are various compilations that compare Flask to other Python-based web frameworks, e.g. here with Django and here with FastAPI.

Speed is nice, but isn't WDQS more of a bottleneck?

Yes, WDQS seems to be our major bottleneck — most of our tickets under the performance label are essentially about that, and

• Assess implications of Blazegraph failure playbook for Scholia #1721

specifically is about the expected Blazegraph failure within the next 5 years or so. Comments about how Scholia might make use of potential Blazegraph alternatives (e.g. working with dumps, or using other query engines like Virtuoso or QLever) would be welcome via those tickets. This current ticket is about the web framework that Scholia uses, for which there is certainly room for improvement too.

Having said that, I did look at the QLever query discussed above and noticed the following:

• the immediate reason why the query at https://qlever.cs.uni-freiburg.de/wikidata/ayZR1r fails is because the statements before the filter calls do not end with a period (WDQS/ Blazegraph are fine with that, and I think so is SPARQL 1.1)
• some SPARQL commands like string functions or lcase do not seem to be fully supported by QLever yet (at least inside filter clauses)
• replacing the string commands with a regex results in a query that actually gives useful results: https://qlever.cs.uni-freiburg.de/wikidata/J8PSek .

[hannahbast]

https://qlever.cs.uni-freiburg.de/wikidata/ayZR1r doesn't like the query so i couldn't test it.

This equivalent query works on QLever and takes less than a second: https://qlever.cs.uni-freiburg.de/wikidata/6fEwJ4

The reasons the original query did not work were minor (and will all be fixed soon):

1. QLever currently wants a dot before the FILTER
2. QLever currently wants quotes instead of apostrophes for strings
3. String functions in filters are not yet implemented, but a FILTER REGEX works equally well

PS: Just saw that @Daniel-Mietchen basically wrote the same thing 17 hours ago, never mind :-)

[prototyperspective]

Short question which I'll remove if the issue is about something else:
does it have to be a Python framework? I would recommend a Javascript one.

As fnielsen said "Most of our dynamic data comes from Javascript" and even though I really like Python, Javascript is far more advanced modern well-looking and easy to learn and develop for the Web and apps these days. It was decided that Wikipedia will use Vue.Js and I'd suggest something similar. This would also enable simply generating mobile apps (iOS and Android) and a desktop app from the Website code without having to write any additional code.

[dpriskorn]

Short question which I'll remove if the issue is about something else: does it have to be a Python framework? I would recommend a Javascript one.

As fnielsen said "Most of our dynamic data comes from Javascript" and even though I really like Python, Javascript is far more advanced modern well-looking and easy to learn and develop for the Web and apps these days. It was decided that Wikipedia will use Vue.Js and I'd suggest something similar. This would also enable simply generating mobile apps (iOS and Android) and a desktop app from the Website code without having to write any additional code.

I tend to agree here. I started learning React recently. It seems very flexible and well engineered.

[prototyperspective]

I tend to agree here. I started learning React recently. It seems very flexible and well engineered.
React is great technology too. However, that's not what I'd propose here.

There are good reasons for why Vue.Js is the framework with the most stars from devs on Github and for why it was decided to be used by Wikipedia (issue/discussion about this can be found here).

If you like React, you'd probably enjoy Vue.Js even more: it's very straightforward to use and quick to learn (especially if you already know Js and even React; I suggest starting with a simple project and looking at these docs whenever something is unclear). While React is developed/owned by Meta/Facebook, Vue.Js is developed by an open-source oriented community, just like Wikipedia/MediaWiki. There are components ready to be used for almost every use-case. It has tons of other advantages, including being pretty future-proof and more attractive to developers who could help develop Scholia (including people who only recently became devs due to both the rising prevalence of Js & this framework and the ease of use).

There are multiple Vue.Js frameworks and frameworks that can easily be used in combination with it (such as Node.Js for the backend). The VueJs framework I'd like to propose is the Quasar Framework. Maybe all of this should be a separate issue. But I'll just shortly outline why I think it would be the best choice.

Most reasons are listed on the "Why Quasar?" page at the link or are the same as the ones for VueJs. It's very easy to use this framework and it has most things and components standardized. You can create Android, iOS, desktop and other apps from the same source code simply by running some command. It looks very well and really is a state-of-the-art flexible UI (the current site looks a bit outdated). It could even be used to create a browser extension (in this case from almost the same source, probably not exactly the same) whereby you can get some parts of the Scholia displayed on pages of studies or authors or topics etc directly, which could make Scholia drastically more useful (especially once it has more data; creating a browser extension would be a separate issue). Basically it would cut the time and effort of frontend-related development to maybe a fifth while upgrading the UI, improving the UX while also enabling apps. If you have questions or need some help with it, I could you help out and we could chat on Matrix/Element (or IRC).