ITM Inconsistent Path Loss with Minor Terrain Differences #933
Description
In the past in a small number of cases, I have noticed discrepancies in ITM path loss between Windows and Linux that can result in differences greater than 1 dB (which can cause PAT subtests to fail). After further digging, this appears to be due to small differences in terrain profile. The same test harness Python code (in this repo) and terrain data was used to create the terrain profiles on Windows and Linux, but different OS results in small differences in terrain profile. The extremely small differences (max difference approx. 1E-9) can cause > 1 dB difference in pathloss in some cases due to numeric error in the LinearLeastSquaresFit function. See NTIA/itm#21.
In the version of ITM used in this repository, I think the LinearLeastSquaresFit function is z1sq1. For the ITS ITM version, I created a pull request that adds some rounding to fix this issue.