How to manage contents with a CMS and get translations done

[alexgarel]

We want contents on Open Food Facts to be updated more quickly and to be more attractive.
For that we need a tool that enables content creators to do it easily. We still need contents to be massively translated.

@4nt0ineB has done a work to have contents for open food facts in wordpress.
The plan was to use WPML with Crowdin to translate the contents.
But as we tested it, it does not fit our needs:

• it creates completely new file on each article modification
• it does not handle already translated items, and partial translations…

We want to explore further more what to do.

Possible ideas:

• don't use crowdin, and use wordpress for translations, managing users and so on (wikimedia does this)
• use a static site generator, and handle translations with github / crowdin
• dump html contents from wordpress and handle translations with github / crowdin

[alexgarel]

One way would be:

• use CMS (wordpress) to generate content in english
• push to static html in a github repo (using a small script with wordpress API)
• send to crowdin for translations
• serve it as we do today for openfoodfacts-web

One problem that remains is that the way we handle crowding PRs currently is messy, because we got too much translations at once, and it's not easy to validate.

With a quick read at github action for crowdin it may be possible to have a better strategy:

• do one PR per language (easy)
• do one PR per file (and per language ?) (a bit harder because we have to add a github action for each file, using matrix strategy + output of file list)
• only download fully translated files (make sense for article translations)

I must experiment with this, but I think it could be a huge improvement also for other projects.

@teolemon would you agree that for contents the best would be:

• one PR per file (article) and per language
• but only download complete files (we can eventually add "only approved translations" if we have a manager per language)

[CharlesNepote]

Some path to investigate?

Crowdin CMS connectors managing minor content updates

Some Crowdin CMS connectors seem to manage minor content updates, suggesting you don't have to reload the whole page when there is a small fix.

At least I have found the following connectors/CMS do this:

• https://store.crowdin.com/strapi (free) : "Quick sync and translation of even minor content updates, like a new sentence or a changed word."
• https://store.crowdin.com/directus (free) : idem
• https://store.crowdin.com/buttercms (trial) : idem
• etc., see https://www.google.com/search?q=even+minor+content+updates+site:crowdin.com

It seems that all these CMSs are headless.

What kind of contents Crowdin is able to translate with ease?

How Crowdin is able to translate contents in a fine grain approach (per paragraphs, blocs, etc.)?
Are there Crowdin connectors already fine for some kind of format? JSON? Markdown? Is possible to build a simple connector based on specific format?

Then find the collection of tools allowing to produce this kind of format, and generate web contents based on it.

The web content management approach

Some web tools allow to build rich web layout based on flat files. It might be possible to export these contents to feed Crowdin.

Eg. to investigate:

• Automad, a flat file CMS that can be headless; Rich https://automad.org/user-guide/using-blocks
• Typemill, a flat file CMS that handles blocks: https://typemill.net/

What are the main needs for editors?

The main needs should be listed to select/reject some tools:

• rich editing
• image management: upload, resize, crop...
• block editing (like in Wordpress, Odoo, etc.): allowing image integration, multi-colomns eventually
• easy revision process
• easy publication process
• ...

[CharlesNepote]

Also look at: "Manage taxonomies properties in an external CMS ", #8536

[alexgarel]

I looked at Hugo, and saw it has a columns widget.

It also supports multilingual content.

I see some point in using it:

1. it is very popular
2. it's fast
3. if you write your markdown logically, crowdin will cut at line endings, thus making life easier for translators (translate logical blocks)
4. there would be very few html in pages, and we can validate every translation is valid markdown (and translations PRs can validate website is still building
5. it lives in a git repository

One big cons is that there is no real wysiwyg editor. But hugo shortcodes helps making beautiful content. (I see there are a lot of additional shortcodes around, and easy to install and customize)

Frontmatter might be a thing to explore for contributors, it brings abotu live visualization of rendered content.

[VaiTon]

If content editors can live with Markdown, then there are a lot of tools for the job, translation and everything. If we can't, then HTML becomes the only solution and translating it directly via Crowdin is possible.

[rpowis]

@alexgarel @CharlesNepote I'd love to help research CMS options. I can also set up demo instances of potential choices so developers and editors can try them out and see what works best.

Aside from the mention of a rich text editor and Crowdin for translations, is there a list of requirements documented somewhere?

As @CharlesNepote said, having that will make it easier to rule things out—there are a lot of CMS options. It doesn’t need to be super detailed at this point, but even a rough list of things like editor/tech features, hosting requirements, FOSS vs. SaaS, budget, necessary integrations, etc. would be really useful.

[alexgarel]

@rpowis after some thinking and looking at things, it seems to me that we could really give a try to https://decapcms.org/ + hugo

[alexgarel]

@rpowis dropping some points quickly:

1. budget: low ;-) Normally we would prefer not to have to pay for it (more-over many options are user based and we have a lot of potential contributors)
2. translations needs to be able to be done in a lot of languages, with an easy way to review / validate. Content might change during translation process (which may span a long time, because done on a volunteer basis).
3. If possible we want to leverage our existing crowdin community. But they are not contractors so it's important that the work can be done when people have time, etc.
4. we want editors to be able to easily redact "fancy content" (with some freedom in page layout, the level of a vanilla wordpress is a good comparison point)
5. learning curve must not be too steep for editors (who are not tech). Wysiwyg editing would really be better (or at least live preview).

[alexgarel]

No opinion on the format of the content, but I forgot this requirement:

• it must in someway integrate in the website (it's even better if it's serving pages directly but just mimic the website header / footer)

[CharlesNepote]

About point 3: this one seem very important. The synchronisation between the CMS and Crowdin should be done per paragraph, or even at a finer grain. Translators shouldn't have to redo whole page translation because a word has changed.

About point 4: I think we can manage a compromise between Wordpress and Markdown. Markdown might be too poor for our needs, but on the other hand Wordpress blocks are also sometimes overcomplicated. To my experience, good image and column management are important. I have investigated multi-columns with Markdown, but the proposed implementations are too geeky to be used in Markdown sources IMHO:

• https://github.com/ckRobinson/multi-column-markdown
• https://github.com/sander76/markdown-columns
• https://mattbierner.github.io/markdown-two-column-documentation-example/
• https://bookdown.org/yihui/rmarkdown-cookbook/multi-column.html
• https://marketplace.visualstudio.com/items?itemName=jebbs.markdown-extended
• etc.

Very few CMS allows block editing, and we need to know how these blocks will be exported to Crowdin.

We could also build our own simple system/syntax to allow columns in Markdown.

[alexgarel]

@CharlesNepote I agree with you. For column management, Hugo as a plugin (as noted above)

[rpowis]

Thanks all, very helpful! Another thought I had was allowing anyone on the internet to suggest/make an edit (for example, if they spot a typo or could propose re-wording) Is that a wanted feature?

[alexgarel]

Is that a wanted feature?
Why not but it's clearly not a priority.

[rpowis]

I also just stumbled across https://github.com/openfoodfacts/openfoodfacts-explorer. I couldn't find anything written about why it exists anywhere 🤔

Are there any goals for that project or is it just an experiment? Is the plan to eventually have it replace the current website? Or is the current website not going anywhere and it will run along side it? Does it also need to be considered in the context of a CMS?

[alexgarel]

@rpowis yes it can be considered. We have no plan to replace current website with off-explorer in the near future, but we are happy to see it grow and eventually replace current website with it, or part of current website ! It's a project started by a long standing contributor.

What is sure is that we are trying to have a separation of concern. So IMO the web pages are a separate matter that could integrate to both.

[alexgarel]

https://content.nuxt.com/blog/studio-module-alpha is a promising solution that would fit our needs I think (just gave a quick look though)

[CharlesNepote]

Interesting. We should investigate. It's difficult to understand how to integrate a translation process with Crowdin.

[CharlesNepote]

Nuxt Studio has been released and opensourced: https://content.nuxt.com/blog/studio-oss

I'm a bit afraid of the technical stack, though: it relies on Nuxt, itself based on Vue and many third-party libraries. It's definitely not a low-tech solution. Complexity is costly.

We could have a try, though.

[CharlesNepote]

Could Quarto be another solution?

Pros:

• Markdown code...
• ... and a visual editor in Rstudio (but it is not absolutely linked to it, I use Quarto in VSCode)
• there is a specific syntax to use CSS class to build complex layout (eg ::: {.column width="50%"})
• Quarto allows to compute things in Python/R/Bash/others when rendering; very interesting if we want to update key numbers: "Discover more than X millions of products" (I have used Quarto to generate this .md file)

Cons:

• the visual editor does not display columns as columns
• Quarto might be a bit too geeky for editors?

Quarto visual editor

Columns in the visual editor

Rendered document

[alexgarel]

Another solution that I have seen: using https://hedgedoc.org/ but add some markdown plugins (it's not officially supported but might be easy to do). It should support any markdown-it npm library. Using markdown-it-container + a custom CSS could do the thing.
The advantage would be real time collaboration + immediate preview.
The drawback is that we need to customize hedgedoc to include this library + specific CSS (I don't know if it's hard).

[teolemon]

at this point, we could ask for a LLM to experiment with the DB dump from zammad with all of the proposed solutions and publish it on github with github actions

[CharlesNepote]

Hedgedoc is very interesting, but I found it better suited for real time pads. I'm not sure Hedgedoc has features to manage a website hierarchy, to name files, to create links between the documents, etc.

Do we really need real time collaboration for this need?

If real time preview is important, many tools are doing which are more aligned with our practices (git, CI...): VSCode, github.dev, Quarto, https://stackedit.io, https://dillinger.io, etc. But I'm not sure they can manage rich layout preview.

At least https://stackedit.io is worth a try, as it can be synchronized with a Github repository. I have opened a test repository to do so: https://github.com/openfoodfacts/cms-test