Fix Mastodon URL previews not showing anything useful (#19231)

Fixes #18444. Inside of UrlPreviewer, we need to combine two dicts (one
from oEmbed, and one from OpenGraph metadata in the HTML) and in Mastodon's case they were very
different.

Single Page Applications (SPAs) seem to sometimes provide better information in the OpenGraph tags
than the oEmbed stubs, because the oEmbed stubs are filled in with JavaScript that Synapse does
not execute.

This change improves previews on Mastodon and YouTube (for the same reason).

Tested to not regress previews of Twitter or GitHub.

Author: redstrate

changelog.d/19231.bugfix

@@ -0,0 +1 @@
+Fix a bug where Mastodon posts (and possibly other embeds) have the wrong description for URL previews.

synapse/media/url_previewer.py

@@ -331,10 +331,16 @@ async def _do_preview(self, url: str, user: UserID, ts: int) -> bytes:
                 # response failed or is incomplete.
                 og_from_html = parse_html_to_open_graph(tree)
 
-                # Compile the Open Graph response by using the scraped
-                # information from the HTML and overlaying any information
-                # from the oEmbed response.
-                og = {**og_from_html, **og_from_oembed}
+                # Compile an Open Graph response by combining the oEmbed response
+                # and the information from the HTML, with information in the HTML
+                # preferred.
+                #
+                # The ordering here is intentional: certain websites (especially
+                # SPA JavaScript-based ones) including Mastodon and YouTube provide
+                # almost complete OpenGraph descriptions but only stubs for oEmbed,
+                # with further oEmbed information being populated with JavaScript,
+                # that Synapse won't execute.
+                og = og_from_oembed | og_from_html
 
                 await self._precache_image_url(user, media_info, og)
             else: