Python script to generate a .json.gz file per each locale by abaevbog · Pull Request #5 · zotero/zotero-schema

abaevbog · 2023-05-02T04:40:30Z

No description provided.

dstillman

Let's also have this output the necessary lines to copy into .htaccess to map locale parameter values to the files between written. We mostly won't need to update them, but good to keep them coupled to the available locales, and the logic might need tweaking.

And for that logic, we basically want to implement this code:

https://github.com/zotero/zotero/blob/3e12f3f20b097d56006c040ea466fb422ba78308/chrome/content/zotero/xpcom/utilities_internal.js#L1581-L1620

…but entirely with rewrite rules. (I haven't totally though through how possible this is, but I think it's mostly doable by writing out a lot of rules in a given order. Let's see what we can do.)

en-US → en-US (exact match)
ar → ar (exact match)
de-DE → de (matching language part)
ca → ca-AD (matching language part)
en-NZ → en-US (prefer en-US over other available en locales for inexact match)
pt → pt-PT (prefer country code matching language code if unspecified — this one's debatable, and 217 million Brazilians would presumably disagree, but it's what we do elsewhere)
zh → zh-CN (this we just get from sorting the country codes, but it makes sense for our userbase)
zz → full file (ignore locale parameter if language code is unknown)

dstillman · 2023-05-02T09:40:33Z

+    schema_text = f.read()
+    schema = json.loads(schema_text)
+
+if not os.path.exists('../locales'):


Script should be runnable from any folder. You're already getting the parent folder above, so you should just use that for other paths.

You should also wipe locales to remove any existing files. (Locales will pretty much never be removed, but just to fix possible bugs, etc.)

Understood, this has been done.

dstillman · 2023-05-02T09:41:58Z

+            for creator_type in item_type['creatorTypes']:
+                if creator_type['creatorType'] in current_locale["itemTypes"]:
+                    creator_type['creatorType'] = current_locale["itemTypes"][creator_type['creatorType']]
+        del schema_with_one_locale['locales']


Definitely don't need all of this — sorry for not specifying. Can just keep the one locale keyed properly in locales, as below.

Got rid of it!

abaevbog · 2023-05-02T15:29:55Z

Added the generation of the .htaccess file. The example output is as below:

RewriteRule ^schema/(af-ZA|ar|bg-BG|br|ca-AD|cs-CZ|da-DK|de|el-GR|en-GB|en-US|es-ES|et-EE|eu-ES|fa|fi-FI|fr-FR|gl-ES|he-IL|hr-HR|hu-HU|id-ID|is-IS|it-IT|ja-JP|km|ko-KR|lt-LT|mn-MN|nb-NO|nl-NL|nn-NO|pl-PL|pt-BR|pt-PT|ro-RO|ru-RU|sk-SK|sl-SI|sr-RS|sv-SE|th-TH|tr-TR|uk-UA|vi-VN|zh-CN|zh-TW)$ /zotero-schema/locales/$1.gz [L]
...
RewriteRule ^schema/he(-.*)?$ /zotero-schema/locales/he-IL.gz [L]
RewriteRule ^schema/hr(-.*)?$ /zotero-schema/locales/hr-HR.gz [L]
RewriteRule ^schema/hu(-.*)?$ /zotero-schema/locales/hu-HU.gz [L]
RewriteRule ^schema/id(-.*)?$ /zotero-schema/locales/id-ID.gz [L]
...
RewriteRule ^schema/* /zotero-schema/schema.json.gz [L]

Each of these rules matches /schema/{country_code}-... or /schema/{country_code} but not /schema/{country_code}something_else

dstillman · 2023-05-03T09:59:50Z

+os.mkdir(locales_folder)
+
+# String that accumulates the rules to paste into htaccess
+htaccess_rules = f"RewriteRule ^schema/({'|'.join(schema['locales'].keys())})$ /zotero-schema/locales/$1.gz [L]"


It should be a locale query string parameter, not a path component.

dstillman · 2023-05-03T10:00:04Z

+# Catch all for default schema with all locales
+htaccess_rules += f'\nRewriteRule ^schema/* /zotero-schema/schema.json.gz [L]' 
+
+print("--- .htacess rules --- \n" + htaccess_rules + "\n---  ---")


Can skip the header and footer

dstillman · 2023-05-04T03:30:42Z

Can replace update-gz with update-gz.py, no extension, mode 755

dstillman · 2023-05-04T03:37:54Z

I think we'll need some additional conditions/rules for the locale gz files, like we already have for schema.json.gz:

https://github.com/zotero/dataserver/blob/5fba6e4c4a9df97a305764a69ebf70a872d38cd8/htdocs/.htaccess#L22-L38

E.g., checking Accept-Encoding

Let's just generate the whole block in this script. And we don't need newlines — better to keep this as a single block.

dstillman · 2023-05-04T03:54:28Z

Not sure it matters, but since we're generating this in a script anyway and know the total count, we can probably add a skip flag (S) to skip all of these rules unless the URL begins with schema. These will be checked for every single dataserver request, so we do want to avoid any slowdown.

(Will this all be meaningfully faster than just having a single schema.php file that generate a locale-specific file and dumped it in memcached? Unclear. But this is certainly the more fun way to do it…)

…rect content type, and filematch

abaevbog · 2023-05-04T17:50:25Z

After testing it out with a version of dataserver I ran locally, I had to add one more condition RewriteCond %{SCRIPT_FILENAME} !/zotero-schema/ at the end for everything to work properly. This is a gist of the .htaccess file I ended up with

https://gist.github.com/abaevbog/23a986e6966000325f609652cd25e6ce

dstillman · 2023-05-05T11:23:19Z

I think we can do a slightly simpler block:

https://gist.githubusercontent.com/dstillman/82db17dca11e2bda0bd87b4d27c02c89/raw/2398339bc434c5dfef7d44e8824b58df09122381/gistfile1.txt

Specifically:

Last for everything
Removed the lines with E=no-gzip:1 and move those things (setting correct content type, preventing double-gzip) into the FilesMatch
No need for zotero-schema in the Skip RewriteCond, since we're setting Last
No need for RewriteCond %{SCRIPT_FILENAME} !/zotero-schema/ at the end, since we're setting Last

abaevbog · 2023-05-05T12:23:44Z

Yes, you are right...
Adding the [L] flag solved whatever issues I had

dstillman · 2023-05-06T08:53:03Z

-htaccess_rules = f'''RewriteCond %{{REQUEST_URI}} !^/(schema|zotero-schema) 
-RewriteRule ".?" "-" [S=LINES_TO_SKIP]
+htaccess_rules = f'''RewriteCond %{{REQUEST_URI}} !^/schema 
+RewriteRule ".?" "-" [S=LINES_TO_SKIP,L]


L isn't right here, though — that breaks the dataserver completely by skipping the main redirect at the bottom of the file.

I see what you mean - I removed it.

For some reason when I was testing it on my local dataserver setup, all non /schema requests reached index.php as they were supposed to even with that L flag. Probably can blame it on my local dataserver or apache setup

dstillman · 2023-05-07T09:16:55Z

+RewriteCond %{{HTTP:Accept-Encoding}} !gzip
+RewriteRule ^schema(/.*)?$ /zotero-schema/schema.json [QSD,L]
+RewriteCond %{{QUERY_STRING}} (?:^|&)locale=({'|'.join(schema['locales'].keys())})(?:&|$)
+RewriteRule ^schema(/.*)?$ /zotero-schema/locales/%1.json.gz [QSD]


Missing L here, so exact matches (e.g., locale=en-US) don't work

dstillman · 2023-05-07T09:17:22Z

+# For every country code, sort locale candidates and add rule to htacecss
+for country_code in htaccess_mapings.keys():
+    htaccess_mapings[country_code].sort(key=locale_sort_key)
+    # Each rule is only applid is gzip encoding is accepted


applid is → applied if

Bogdan Abaev added 2 commits May 2, 2023 00:38

python script to generate a .json.gz file per each locale

c7e4f2d

typo fix

def1906

dstillman requested changes May 2, 2023

View reviewed changes

generating .htaccess file + cleanup

d15f816

abaevbog requested a review from dstillman May 2, 2023 19:58

slightly refined htaccess rules

962f3c3

dstillman requested changes May 3, 2023

View reviewed changes

htaccess rules with locale as query parameter instead of path

31e232e

abaevbog requested a review from dstillman May 3, 2023 14:59

defaulting to schema.json if gzip is not accepted, rules to serve cor…

b90db18

…rect content type, and filematch

skipping rules if url does not start with /schema

498b190

abaevbog force-pushed the master branch from 76e6c3f to 498b190 Compare May 4, 2023 18:31

updated filter to not skip rules on zotero-schema

4ccd056

simplifications for the rules

e3e1ed5

dstillman reviewed May 6, 2023

View reviewed changes

No L in skip rule

7aef704

dstillman requested changes May 8, 2023

View reviewed changes

typo fix and last L

393ffe5

abaevbog requested a review from dstillman May 8, 2023 12:44

Conversation

abaevbog commented May 2, 2023

Uh oh!

dstillman left a comment

Choose a reason for hiding this comment

Uh oh!

dstillman May 2, 2023

Choose a reason for hiding this comment

Uh oh!

abaevbog May 2, 2023

Choose a reason for hiding this comment

Uh oh!

dstillman May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abaevbog May 2, 2023

Choose a reason for hiding this comment

Uh oh!

abaevbog commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstillman May 3, 2023

Choose a reason for hiding this comment

Uh oh!

dstillman May 3, 2023

Choose a reason for hiding this comment

Uh oh!

dstillman commented May 4, 2023

Uh oh!

dstillman commented May 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstillman commented May 4, 2023

Uh oh!

abaevbog commented May 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstillman commented May 5, 2023

Uh oh!

abaevbog commented May 5, 2023

Uh oh!

dstillman May 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abaevbog May 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dstillman May 7, 2023

Choose a reason for hiding this comment

Uh oh!

dstillman May 7, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

dstillman May 2, 2023 •

edited

Loading

abaevbog commented May 2, 2023 •

edited

Loading

dstillman commented May 4, 2023 •

edited

Loading

abaevbog commented May 4, 2023 •

edited

Loading

dstillman May 6, 2023 •

edited

Loading

abaevbog May 6, 2023 •

edited

Loading