Add bcp47lang field to wiki_mapping, and corvertor to commonswiki description. #1369
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a new bcp47Lang field to the wiki_mapping database table to store BCP 47-compliant language codes alongside the original Wikipedia language codes. It also refactors the WikiLangConverter class by moving it from the net.osmand.wiki.commonswiki package to net.osmand.wiki for broader reusability, and applies language normalization to commonswiki descriptions.
Key changes:
- Added
bcp47Langcolumn towiki_mappingtable with corresponding indexes - Moved
WikiLangConverterfromcommonswikitowikipackage for better organization - Applied BCP 47 conversion to commonswiki descriptions using
WikiLangConverter.normalizeLang()
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| WikiLangConverter.java | Moved from net.osmand.wiki.commonswiki to net.osmand.wiki package |
| WikiLangConverterTest.java | Updated import statements to reflect package change |
| WikiDataHandler.java | Added bcp47Lang field to schema, prepares statement, and index; converts wiki language codes to BCP47 during data insertion |
| CommonsWikimediaPreparation.java | Applied WikiLangConverter.normalizeLang() to description metadata to normalize language codes in JSON |
| WikiDatabasePreparation.java | Updated schema and queries to include bcp47Lang field, added index for bcp47Lang lookups |
| IndexCreatorSettings.java | Updated comment to document new bcp47Lang field in schema |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| mappingPrep.setString(2, siteLink.lang()); | ||
| mappingPrep.setString(3, siteLink.title()); | ||
| mappingPrep.setString(2, siteLang); | ||
| mappingPrep.setString(3, WikiLangConverter.toBcp47FromWiki(siteLang)); |
There was a problem hiding this comment.
WikiLangConverter.toBcp47FromWiki(siteLang) can return null when the language code cannot be recognized (see method documentation). This null value should be handled explicitly before inserting into the database, either by providing a default value or by skipping the insertion. Consider using a fallback such as the original siteLang or an empty string when the conversion returns null.
https://github.com/osmandapp/OsmAnd-Issues/issues/3122