Merge pull request #33 from bxparks/develop

merge 1.1.1 into master

Author: bxparks

.github/workflows/ccpp.yml renamed to .github/workflows/aunit_tests.yml

@@ -1,4 +1,4 @@
-name: C/C++ CI
+name: AUnit Tests
 
 on: [push]
 
@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-18.04
 
     steps:
-    - uses: actions/checkout@v1
+    - uses: actions/checkout@v2
 
     - name: Setup
       run: |
@@ -17,9 +17,12 @@ jobs:
         git clone https://github.com/bxparks/AceButton
         git clone https://github.com/bxparks/AceRoutine
         git clone https://github.com/bxparks/AUnit
+        git clone https://github.com/bxparks/AceUtils
+
     - name: Verify examples
       run: |
         make -C examples
+
     - name: Verify tests
       run: |
         make -C tests

.github/workflows/pythonpackage.yml renamed to .github/workflows/python_tools.yml

@@ -1,7 +1,7 @@
 # This workflow will install Python dependencies, run tests and lint with a variety of Python versions
 # For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
 
-name: Python package
+name: Python Tools
 
 on:
   push:
@@ -12,30 +12,34 @@ on:
 jobs:
   build:
 
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-18.04
     strategy:
       matrix:
         python-version: [3.7, 3.8]
 
     steps:
     - uses: actions/checkout@v2
+
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v1
       with:
         python-version: ${{ matrix.python-version }}
+
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        # pip install -r requirements.txt
-        pip install flake8 mypy pytz python-dateutil
+        pip install -r tools/requirements.txt
+
     - name: Lint with flake8
       run: |
         cd tools
         make flake8
+
     - name: Type checking with MyPy
       run: |
         cd tools
         make mypy
+
     - name: Test with unittest
       run: |
         cd tools

.github/workflows/validation.yml

@@ -0,0 +1,70 @@
+# See the following for information on how to setup the Python workflow for a
+# single version:
+# https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: Validation Tests
+
+on: [push]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-18.04
+
+    # Tried to use
+    # https://github.com/actions/checkout#Checkout-multiple-repos-side-by-side,
+    # but it looks like GITHUB_WORKSPACE is set to
+    # /home/runner/work/AceTime/AceTime, so if path is set to 'main' as
+    # suggested in the article, then the repos is set to
+    # /home/runner/work/Acetime/AceTime/main/AceTime, which is really
+    # confusing. Instead, use 'cd ..' to go up a level and call 'git clone'
+    # manually.
+    steps:
+    - name: Checkout AceTime
+      uses: actions/checkout@v2
+
+    - name: Checkout 3rd Party Repos
+      run: |
+        cd ..
+        git clone https://github.com/bxparks/UnixHostDuino
+        git clone https://github.com/bxparks/AUnit
+        git clone https://github.com/eggert/tz
+        git clone https://github.com/HowardHinnant/date
+
+    - name: Install libcurl4-openssl-dev
+      run: |
+        sudo apt update
+        sudo apt install -y libcurl4-openssl-dev
+
+    - name: Set up Python 3.7
+      uses: actions/setup-python@v1
+      with:
+        python-version: '3.7'
+
+    - name: Display Python version
+      run: python -c "import sys; print(sys.version)"
+
+    - name: Install Python Dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r tools/requirements.txt
+
+    - name: Build compare_cpp
+      run: |
+        cd tools/compare_cpp
+        make
+
+    # Run just BasicHinnantDateTest and ExtendedHinnantDateTest. When a new
+    # TZDB version comes out, the Python pytz, dateutil, and Java validation
+    # tests will always fail because they depend on the obsolete TZ database on
+    # the hosted Operating System.
+
+    - name: BasicHinnantDateTest
+      run: |
+        make -C tests/validation/BasicHinnantDateTest
+        make -C tests/validation/BasicHinnantDateTest runtests
+
+    - name: ExtendedHinnantDateTest
+      run: |
+        make -C tests/validation/ExtendedHinnantDateTest
+        make -C tests/validation/ExtendedHinnantDateTest runtests

CHANGELOG.md

@@ -1,6 +1,29 @@
 # Changelog
 
 * Unreleased
+* 1.1.1 (2020-10-18, TZ DB version 2020c)
+    * Add documentation for using the `PrintStr<N>` class from the AceUtils
+      library (https://github.com:bxparks/AceUtils) on the various `printTo()`
+      methods. The string content can be retrieved using the
+      `PrintStr::getCstr()` method which returns a normal `const char*`
+      C-string. The `PrintStr<N>` class replaces an earlier, unreleased version
+      that was called `CstrPrint<N>`.
+    * Add 'Validation Tests' GitHub workflow for running
+      `tests/validation/*Test`.
+    * Create `blacklist.json` file for each `compare_xxx` tools, to disable
+      validation checks for DST or abbreviations due to bugs in the 3rd party
+      libraries.
+    * Add `ValidationScope` enum to provide better control over whether the
+      DST or abbrev fields are validated.
+    * Print better diagnostic messages when `tests/validation` fails in
+      `BasicTransitionTest` and `ExtendedTransitionTest` classes.
+    * Upgrade target version numbers of 3rd party libraries used for
+      tests/validation: pytz from 2019.3 to 2020.1, JDK 11.0.6 to 11.0.8.
+    * Upgrade to TZDB 2020c.
+    * Restrict GitHub Actions workflow to run just BasicHinnantDateTest and
+      ExtendedHinnantDateTest, because the other Python and Java tests break
+      every time a new TZDB version comes out.
+    * Add `DEVELOPER.md` file containing notes mostly for myself.
 * 1.1 (2020-04-25, TZ DB version 2020a)
     * Fix broken links in `README.md`.
     * Fix typos in `USER_GUIDE.md` and update its version number to 1.0 as it

DEVELOPER.md

@@ -0,0 +1,182 @@
+# Developer Notes
+
+Information which are useful for developers and maintainers of the AceTime
+library.
+
+## Zone DB Files
+
+As explained in the README.md, the AceTime library comes with 2 versions of the
+[TZ Data](https://www.iana.org/time-zones):
+
+* `src/ace_time/zonedb/*`: files used by the `BasicZoneProcessor` class
+* `src/ace_time/zonedbx/*`: files used by the `ExtendedZoneProcessor` class
+
+There are 2 main files in these directories:
+
+* `zone_infos.h', `zone_infos.cpp`
+* `zone_policies.h`, `zone_policies.cpp`
+
+The format and meaning of these files are probably best explained in the header
+files:
+
+* `src/ace_time/internal/ZoneInfo.h`
+* `src/ace_time/internal/ZonePolicy.h`
+
+Some of the tricky encoding schemes were created to preserve resolution, while
+keeping the binary size of the data structures as small as possible. For
+example, these files are able to support DST transitions with a 1-minute
+resolution, but the DST offsets have only a 15-minute resolution. Fortunately,
+these limitations are sufficient to represent all time zones since about 1972.
+
+Other bits of information are added by the code generator as comments in the
+generated files and some of these are explained below.
+
+### `zone_infos.h`
+
+This file contains one entry of the `basic::ZoneInfo` or
+`extended::ZoneInfo` data structure for each supported time zone.
+
+The zone identifier is named `kZone{region}_{city}`, where the
+`{region}_{city}` comes directly from the TZ Database. Some minor character
+transformations are applied to create an valid C++ identifier. For example, all
+dashes `-` are converted to underscore `_`.
+
+The `kTzDatabaseVersion` string constant identifies the version of the TZ
+Database that was used to generate these files, e.g. "2019a".
+
+The `kZoneContext` variable points to an instance of `common::ZoneContext`
+which identifies the `startYear` and `endYear` of the current database. These
+will normally be 2000 and 2050 respectively, but a custom version of the
+zonedb files could be generated whose startYear and endYear could be smaller (to
+produce smaller ZoneDB files).
+
+Near end of the `zone_info.h` file, we list the zones which were
+deliberately excluded by the tool.
+
+Also at the end of the `zone_info.h` file, there may be warnings about
+known inaccuracies for a particular zone.
+
+### `zone_infos.cpp`
+
+Each zone entry in the `zone_info.cpp` contains a comment section that
+describes some metadata about the given entry. For example, the entry for
+`zonedb::kZoneEraAmerica_Los_Angeles` contains the following, showing how
+much memory it will consume on the 8-bit Arduino controllers and the 32-bit
+Arduino controllers:
+
+* `// Zone name: America/Los_Angeles`
+* `// Era count: 1`
+* `// Strings: 24`
+* `// Memory (8-bit): 43`
+* `// Memory (32-bit): 53`
+
+### `zone_policies.h`
+
+An entry in `zone_info.cpp` may refer to a zone policy defined in
+`zone_policies.h`. For example, the `kZoneAmerica_Los_Angeles` has a pointer
+to a `kPolicyUS` data structure which is defined in `zone_policies.h`.
+
+Each policy entry starts with a comment secion that contains some metadata
+about the policy. For example:
+```
+// Policy name: US
+// Rule count: 5
+// Memory (8-bit): 51
+// Memory (32-bit): 55
+```
+Just like `zone_infos.cpp`, the Memory section describes the amount of static
+RAM consumed by the particular `ZonePolicy` data structure (and associated
+`ZoneRule`.
+
+## Upgrading ZoneInfo Files to a New TZDB Version
+
+About 2-4 times a year, a new TZDB version is released. Here are some notes
+(mostly for myself) on how to create a new release after a new TZDB version is
+available.
+
+* Update the TZDB repo (https://github.com/eggert/tz). This should be a
+  sibling to the `AceTime` repo (since it will cause the least problems
+  for various internal scripts):
+    * `$ cd ../tz`
+    * `$ git pull`
+    * Check that the correct tag is pulled (e.g. `2020c` tag if that's the
+      version that we want to upgrade to).
+* Update the Hinnant date repo (https://github.com:HowardHinnant/date). This
+  should be a sibling to the `AceTime` repo:
+    * `$ cd ../date`
+    * `$ git pull`
+* Verify that `AceTime` and Hinnant `date` agree with each other using the
+  same TZDB version.
+    * BasicHinnantDateTest
+        * `$ cd tests/validation/BasicHinnantDateTest`
+        * Update the `TZ_VERSION` variable in the `Makefile` with the commit tag
+          of the new TZDB version. (e.g. `TZ_VERSION = 2020c`).
+        * `$ make clean`
+        * `$ make`
+        * `$ ./BasicHinnantDateTest.out | grep failed`
+        * There should be no failures: `TestRunner summary: 268 passed, 0
+          failed, 0 skipped, 0 timed out, out of 268 test(s).`
+    * ExtendedHinnantDateTest
+        * `$ cd tests/validation/ExtendedHinnantDateTest`
+        * Update the `TZ_VERSION` variable in the `Makefile` with the commit tag
+          of the new TZDB version. (e.g. `TZ_VERSION = 2020c`).
+        * `$ make clean`
+        * `$ make`
+        * `$ ./ExtendedHinnantDateTest.out | grep failed`
+        * There should be no failures: `TestRunner summary: 387 passed, 0
+          failed, 0 skipped, 0 timed out, out of 387 test(s).`
+* Update the `src/ace_time/zonedb` and `zonedbx` zone info files:
+    * zonedb
+        * `$ cd src/ace_time/zonedb`
+        * Update the `TZ_VERSION` in the `Makefile`.
+        * `$ make clean`
+        * `$ make`
+    * zonedbx
+        * `$ cd src/ace_time/zonedbx`
+        * Update the `TZ_VERSION` in the `Makefile`.
+        * `$ make clean`
+        * `$ make`
+* Update the CHANGELOG.md.
+* Commit the changes to git
+    * `$ git add ...`
+    * `$ git commit -m "..."`
+
+There are 6 other validation tests in `tests/validation` that compare AceTime
+with other 3rd party librarties (Python pytz, Python dateutil, and Java date).
+Unfortunately, they all seem to use the underlying TZDB version provided by the
+Operating System, and I have not been able to figure out how to manually update
+this dependency manually. When a new TZDB is released, all of these other tests
+will fail until the underying timezone database of the OS is updated.
+
+## Release Process
+
+* Update and commit the version numbers in various files:
+    * `src/AceTime.h`
+    * `README.md`
+    * `USER_GUIDE.md`
+    * `docs/doxygen.cfg`
+    * `library.properties`
+    * `CHANGELOG.md`
+    * `$ git commit -m "..."`
+* Update and commit the Doxygen docs. This is done as a separate git commit
+  because the Doxygen changes are often so large that they obscure all other
+  important changes to the code base:
+    * `$ cd docs`
+    * `$ make clean`
+    * `$ make`
+    * `$ git add .`
+    * `$ git commit -m "..."`
+* Merge the `develop` branch into `master`.
+    * Normally I do this with a PR on GitHub to keep an audit trail.
+    * Go to https://github.com/bxparks/AceTime.
+    * Create a Pull Request from `develop` to `master`
+    * Approve and merge the PR.
+* Create a new Release.
+    * Go to https://github.com/bxparks/AceTime
+    * Click on "Releases"
+    * Click on "Draft a new release"
+    * Enter a tag version (e.g. 'v1.2`), targeting the `master` branch.
+    * Enter the release title.
+    * Enter the release notes. I normally just copy and paste the latest changes
+      from `CHANGELOG.md`.
+    * Click Publish release.

README.md

@@ -211,7 +211,9 @@ Conversion from an epochSeconds to date-time components including timezone
 * 2.8 microseconds on an ESP32,
 * 6 microseconds on a Teensy 3.2.
 
-**Version**: 1.1 (2020-04-25, TZ DB version 2020a)
+**Version**: 1.1.1 (2020-10-18, TZ DB version 2020c)
+
+**Changelog**: [CHANGELOG.md](CHANGELOG.md)
 
 ## Examples
 
@@ -508,10 +510,6 @@ I will occasionally test on the following hardware as a sanity check:
 * Teensy 3.2 (72 MHz ARM Cortex-M4)
 * Mini Mega 2560 (Arduino Mega 2560 compatible, 16 MHz ATmega2560)
 
-## Changelog
-
-See [CHANGELOG.md](CHANGELOG.md).
-
 ## License
 
 [MIT License](https://opensource.org/licenses/MIT)