Skip to content

Python script to generate a .json.gz file per each locale #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
abaevbog wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Python script to generate a .json.gz file per each locale #5

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
c7e4f2d
python script to generate a .json.gz file per each locale
May 2, 2023
def1906
typo fix
May 2, 2023
d15f816
generating .htaccess file + cleanup
May 2, 2023
962f3c3
slightly refined htaccess rules
May 2, 2023
31e232e
htaccess rules with locale as query parameter instead of path
May 3, 2023
b90db18
defaulting to schema.json if gzip is not accepted, rules to serve cor…
abaevbog May 4, 2023
498b190
skipping rules if url does not start with /schema
abaevbog May 4, 2023
4ccd056
updated filter to not skip rules on zotero-schema
abaevbog May 4, 2023
e3e1ed5
simplifications for the rules
abaevbog May 5, 2023
7aef704
No L in skip rule
abaevbog May 6, 2023
393ffe5
typo fix and last L
abaevbog May 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions scripts/update-gz
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ if os.path.exists(locales_folder):
os.mkdir(locales_folder)

# Begin by checking if gzip is supported. If not - return schema.json
htaccess_rules = f'''RewriteCond %{{REQUEST_URI}} !^/(schema|zotero-schema)
RewriteRule ".?" "-" [S=LINES_TO_SKIP]
htaccess_rules = f'''RewriteCond %{{REQUEST_URI}} !^/schema
RewriteRule ".?" "-" [S=LINES_TO_SKIP,L]
Copy link
Copy Markdown
Member

@dstillman dstillman May 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L isn't right here, though — that breaks the dataserver completely by skipping the main redirect at the bottom of the file.

Copy link
Copy Markdown
Author

@abaevbog abaevbog May 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean - I removed it.

For some reason when I was testing it on my local dataserver setup, all non /schema requests reached index.php as they were supposed to even with that L flag. Probably can blame it on my local dataserver or apache setup

RewriteCond %{{HTTP:Accept-Encoding}} !gzip
RewriteRule ^schema(/.*)?$ /zotero-schema/schema.json [QSD,L]
RewriteCond %{{QUERY_STRING}} (?:^|&)locale=({'|'.join(schema['locales'].keys())})(?:&|$)
Expand Down Expand Up @@ -65,21 +65,21 @@ for country_code in htaccess_mapings.keys():
htaccess_mapings[country_code].sort(key=locale_sort_key)
# Each rule is only applid is gzip encoding is accepted
Copy link
Copy Markdown
Member

@dstillman dstillman May 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applid is → applied if

htaccess_rules += f'''RewriteCond %{{QUERY_STRING}} (?:^|&)locale=({country_code}(-.*)?)(?:&|$)
RewriteRule ^schema(/.*)?$ /zotero-schema/locales/{htaccess_mapings[country_code][0]}.json.gz [QSD]
RewriteRule ^schema(/.*)?$ /zotero-schema/locales/{htaccess_mapings[country_code][0]}.json.gz [QSD,L]
'''

# Catch all for default schema with all locales
htaccess_rules += '''RewriteRule ^schema$ /zotero-schema/schema.json.gz [QSD]'''
htaccess_rules += '''RewriteRule ^schema$ /zotero-schema/schema.json.gz [QSD,L]'''

#Rules to serve correct content type and prevent double gzip matching all locales
htaccess_rules += '''
RewriteRule ^zotero-schema/schema\.json.gz$ - [T=application/json,E=no-gzip:1]
RewriteRule ^zotero-schema/locales/.*\.json.gz$ - [T=application/json,E=no-gzip:1]'''

htaccess_rules += f'''
<FilesMatch \"(schema|{'|'.join(schema['locales'].keys())})\.json\.gz$\">
# Serve correct encoding type
Header append Content-Encoding gzip
# Set correct content type
Header set Content-Type application/json
# Prevent double-gzip
SetEnv no-gzip 1
# Force proxies to cache gzipped & non-gzipped schema file separately
Header append Vary Accept-Encoding
# CORS
Expand Down