1.0.0 draft before major version upgrade

Author: mcaceresb

README.md

@@ -403,18 +403,15 @@ TODO
 ----
 
 `gtools` is feature-frozen. Other than bug fixes, no new features will
-be added until version 1.0. All that remains is improving the coverage
-of the debug checks before submitting to SSC.
+be added until version 1.0.
 
-Roadmap to 1.0
-
-- [ ] Improve coverage of debug checks.
+- [X] Improve coverage of debug checks.
     - [X] Test `nunique` for gegen and gcollapse (vs `gunique`)
-    - [ ] Have corner cases for ALL commands
-    - [ ] Test all the options in every command
-    - [ ] Test errors (i.e. make sure commands fail as expected).
-    - [ ] Test weights in `gtop`.
-    - [ ] Test `gen` and `nolocal` in `glevelsof`.
+    - [X] Have corner cases for ALL commands
+    - [X] Test all the options in every command
+    - [X] Test errors (i.e. make sure commands fail as expected).
+    - [X] Test weights in `gtop`.
+    - [X] Test `gen` and `nolocal` in `glevelsof`.
 - [X] Weights in `gquantiles`.
 - [X] Add (partial) support for `strL` variables.
 - [X] Add support for `by` in `gunique`
@@ -436,7 +433,7 @@ Roadmap to 1.0
 - [X] Add weights in `gtop`.
 
 These are options/features I would like to support, but I don't have an
-ETA for them (and they almost surely won't make it to the 1.0 release).
+ETA for them (and they won't make it to the 1.0 release).
 
 - [ ] Add support for binary `strL` variables.
 - [ ] Minimize memory use.

build/_gtools_internal.ado

@@ -2057,6 +2057,9 @@ program _gtools_internal, rclass
                     J(1, max(`=scalar(__gtools_xtile_nq2)', `nquantiles' - 1), 0)
                 local __gtools_xtile_nq_extra `__gtools_xtile_nq_extra' quantiles
             }
+            else if ( (`=scalar(__gtools_xtile_nq2)' > 0) & ("`_pctile'" != "") ) {
+                * matsize for nq2 was already checked
+            }
             else if ( "`_pctile'" != "" ) {
                 disp as txt "(option _pctile ignored)"
             }

build/changelog.md

@@ -1,6 +1,13 @@
 Change Log
 ==========
 
+## gtools-1.0.0 (2018-07-21)
+
+First official release! This will go on SSC once all the tests have
+passed.
+
+------------------------------------------------------------------------
+
 ## gtools-0.14.2 (2018-07-21)
 
 Feature freeze!
@@ -17,6 +24,10 @@ Feature freeze!
 * Added `gen()` to the quick `glevelsof' showcase
 * Added `gen()` to the `glevelsof' usage page and examples.
 
+### Bug Fixes
+
+* `gegen()` with `replace` sets missing values outside `if in` range.
+
 ## gtools-0.14.1 (2018-07-19)
 
 ### Features
@@ -49,6 +60,8 @@ Feature freeze!
   existing variables gets a warning.
 * Fixes https://github.com/mcaceresb/stata-gtools/issues/42 `gunique` typo.
 
+------------------------------------------------------------------------
+
 ## gtools-0.13.3 (2018-05-06)
 
 ### Features
@@ -105,6 +118,8 @@ right-exclusive, would give the wrong quantile.
 - `skew`, `kurt` give the correct answer when the group is a singleton
    or when all observations are the same.
 
+------------------------------------------------------------------------
+
 ## gtools-0.12.8 (2018-04-23)
 
 ### Features
@@ -192,6 +207,8 @@ requests them with weights, the weighted version will still be called
   (OSX pending). gegen does not support weights with multiple
   sources (e.g. `sum(x y z)`).
 
+------------------------------------------------------------------------
+
 ## gtools-0.11.5 (2018-01-16)
 
 ### Features
@@ -279,6 +296,8 @@ requests them with weights, the weighted version will still be called
   more characters than contained in the numfmt local macro. This
   could cause problems on some systems.
 
+------------------------------------------------------------------------
+
 ## gtools-0.10.3 (2017-11-12)
 
 ### Bug fixes
@@ -328,6 +347,8 @@ requests them with weights, the weighted version will still be called
 - Fixed numerical precision issue with quantiles.
   Now ((ST_double) N / 100) is computed first.
 
+------------------------------------------------------------------------
+
 ## gtools-0.9.4 (2017-11-03)
 
 ### Bug fixes
@@ -397,6 +418,8 @@ requests them with weights, the weighted version will still be called
 - Added quotes to `cd cwd` in `gtools`; fixes https://github.com/mcaceresb/stata-gtools/issues/22
 - `gcontract` available; fixes https://github.com/mcaceresb/stata-gtools/issues/23 
 
+------------------------------------------------------------------------
+
 ## gtools-0.8.5 (2017-10-30)
 
 ### Features
@@ -476,6 +499,8 @@ requests them with weights, the weighted version will still be called
 * Fixes https://github.com/mcaceresb/stata-gtools/issues/18
 * Fixes https://github.com/mcaceresb/stata-gtools/issues/11
 
+------------------------------------------------------------------------
+
 ## gtools-0.7.5 (2017-10-08)
 
 ### Features
@@ -545,6 +570,8 @@ requests them with weights, the weighted version will still be called
 
 * `gcollapse, unsorted` no longer supported (due to internal sorting)
 
+------------------------------------------------------------------------
+
 ## gtools-0.6.17 (2017-09-17); fixes #15
 
 ### Bug fixes
@@ -761,7 +788,7 @@ requests them with weights, the weighted version will still be called
   the implementation this may be slower because adding variables takes
   longer with more variables in memory.)
 
----
+------------------------------------------------------------------------
 
 ## gtools-0.5.2 (2017-06-15)
 
@@ -870,7 +897,7 @@ requests them with weights, the weighted version will still be called
 * If you sort the data in C, then assert the sort is unique and
   print "(hashed correctly grouped observations: resulting sort is unique)"
 
----
+------------------------------------------------------------------------
 
 ## gtools-0.4.1 (2017-05-29)
 
@@ -929,7 +956,8 @@ requests them with weights, the weighted version will still be called
   print "(hashed correctly grouped observations: resulting sort is unique)"
 * Be smart about memory management when J is small relative to N.
 
----
+------------------------------------------------------------------------
+
 
 ## gtools-0.3.3 (2017-05-21)
 

build/fasterxtile.ado

@@ -24,15 +24,16 @@ program define fasterxtile
                                     ///
         method(passthru)            /// Quantile method: (1) qsort, (2) qselect
         strict                      /// Exit if nquantiles > # non-missing obs
+                                    ///
+        compress                    /// Try to compress strL variables
+        forcestrl                   /// Force reading strL variables (stata 14 and above only)
         Verbose                     /// Print info during function execution
-        BENCHmark                   /// Benchmark function
-        BENCHmarklevel(passthru)    /// Benchmark various steps of the plugin
-        HASHmethod(passthru)        /// Hashing method: 1 (biject), 2 (spooky)
+        BENCHmark                   /// print function benchmark info
+        BENCHmarklevel(passthru)    /// print plugin benchmark info
+        HASHmethod(passthru)        /// Hashing method: 0 (default), 1 (biject), 2 (spooky)
         hashlib(passthru)           /// (Windows only) Custom path to spookyhash.dll
         oncollision(passthru)       /// error|fallback: On collision, use native command or throw error
                                     ///
-        forcestrl                   ///
-        compress                    ///
         debug(passthru)             ///
         GROUPid(passthru)           ///
         tag(passthru)               ///

build/gcollapse.ado

@@ -21,8 +21,6 @@ program gcollapse, rclass
                                      ///
         merge                        /// Merge statistics back to original data, replacing if applicable
         replace                      /// Allow replacing existing variables with output with merge
-        compress                     /// Try to compress strL variables
-        forcestrl                    /// Force reading strL variables (stata 14 and above only)
         freq(passthru)               /// Include frequency count with observations per group
                                      ///
         LABELFormat(passthru)        /// Custom label engine: (#stat#) #sourcelabel# is the default
@@ -40,16 +38,18 @@ program gcollapse, rclass
         forcemem                     /// Use memory for writing/reading collapsed data
         double                       /// Generate all targets as doubles
                                      ///
+        compress                     /// Try to compress strL variables
+        forcestrl                    /// Force reading strL variables (stata 14 and above only)
         Verbose                      /// Print info during function execution
         BENCHmark                    /// print function benchmark info
         BENCHmarklevel(int 0)        /// print plugin benchmark info
-                                     ///
         HASHmethod(passthru)         /// Hashing method: 0 (default), 1 (biject), 2 (spooky)
+                                     ///
         hashlib(passthru)            /// (Windows only) Custom path to spookyhash.dll
         oncollision(passthru)        /// error|fallback: On collision, use native command or throw error
                                      ///
-        debug                        /// (internal) Allow replacing by variables with output
-        DEBUG_level(int 0)           /// (internal) Allow replacing by variables with output
+        debug                        /// (internal) Debug
+        DEBUG_level(int 0)           /// (internal) Debug (passed to internals)
         debug_replaceby              /// (internal) Allow replacing by variables with output
         debug_io_read(int 1)         /// (internal) Read IO data using mata or C
         debug_io_check(real 1e6)     /// (internal) Threshold to check for I/O speed gains
@@ -395,7 +395,8 @@ program gcollapse, rclass
 
         qui mata: st_addvar(__gtools_gc_recasttypes, __gtools_gc_recastvars, 1)
         if ( `=_N > 0' ) {
-            cap noi _gtools_internal, recast(targets(`gtools_recastvars') sources(`gtools_recastsrc'))
+            cap noi _gtools_internal, ///
+                recast(targets(`gtools_recastvars') sources(`gtools_recastsrc'))
             if ( _rc ) {
                 local rc = _rc
                 CleanExit
@@ -481,13 +482,16 @@ program gcollapse, rclass
     local sources  sources(`__gtools_gc_vars')
     local stats    stats(`__gtools_gc_stats')
     local targets  targets(`__gtools_gc_targets')
-    local opts     missing replace `keepmissing' `compress' `forcestrl' debug(`debug_level')
-    local opts     `opts' `verbose' `benchmark' `benchmarklevel' `hashlib' `oncollision' `hashmethod'
+    local opts     missing replace `keepmissing' `compress' `forcestrl'
+    local opts     `opts' `verbose' `benchmark' `benchmarklevel' `hashmethod'
+    local opts     `hashlib' `oncollision' debug(`debug_level')
     local opts     `opts' `anymissing' `allmissing' `rawstat'
     local action   `sources' `targets' `stats'
 
     local switch = (`=scalar(__gtools_gc_k_extra)' > 3) & (`debug_io_check' < `=_N')
-    local mem    = ("`forcemem'" != "") | ("`merge'" != "") | (`=scalar(__gtools_gc_k_extra)' == 0)
+    local mem    = ("`forcemem'" != "") ///
+                 | ("`merge'"    != "") ///
+                 | (`=scalar(__gtools_gc_k_extra)' == 0)
     local io     = ("`forceio'"  != "") & (`=scalar(__gtools_gc_k_extra)' > 0)
 
     if ( `io' ) {
@@ -518,7 +522,10 @@ program gcollapse, rclass
         * Benchmark adding 2 variables to gauge how long it might take to
         * add __gtools_gc_k_extra variables.
         tempvar __gtools_gc_index __gtools_gc_ix __gtools_gc_info
-        cap noi benchmark_memvars, index(`__gtools_gc_index') ix(`__gtools_gc_ix') info(`__gtools_gc_info')
+        cap noi benchmark_memvars,     ///
+            index(`__gtools_gc_index') ///
+            ix(`__gtools_gc_ix')       ///
+            info(`__gtools_gc_info')
         if ( _rc ) {
             local rc = _rc
             CleanExit
@@ -561,10 +568,14 @@ program gcollapse, rclass
         gtools_timer info 97 `"`msg'"', prints(`bench')
 
         if ( ("`forceio'" == "forceio") & (`=scalar(__gtools_gc_k_extra)' == 0) ) {
-            if ( `verb' ) di as text "(ignored -forceio- because sources are being used as targets)"
+            if ( `verb' ) {
+                di as text "(ignored -forceio- because sources are being used as targets)"
+            }
         }
 
-        if ( "`added'" != "" ) qui mata: st_addvar(__gtools_gc_addtypes, __gtools_gc_addvars, 1)
+        if ( "`added'" != "" ) {
+            qui mata: st_addvar(__gtools_gc_addtypes, __gtools_gc_addvars, 1)
+        }
         local msg "Generated additional targets"
         gtools_timer info 97 `"`msg'"', prints(`bench')
 
@@ -672,7 +683,10 @@ program gcollapse, rclass
         * If we collapsed to disk, read back the data
         * -------------------------------------------
 
-        if ( (`=_N > 0') & (`=scalar(__gtools_gc_k_extra)' > 0) & ( `used_io' | ("`forceio'" == "forceio") ) ) {
+        local ifcond (`=_N > 0')                          ///
+                   & (`=scalar(__gtools_gc_k_extra)' > 0) ///
+                   & ( `used_io' | ("`forceio'" == "forceio") ) 
+        if ( `ifcond' ) {
             gtools_timer on 97
 
             qui mata: st_addvar(__gtools_gc_addtypes, __gtools_gc_addvars, 1)
@@ -805,7 +819,10 @@ end
 
 cap mata: mata drop gtools_get_collapsed()
 mata
-real matrix function gtools_get_collapsed(string scalar fname, real scalar nrow, real scalar ncol)
+real matrix function gtools_get_collapsed(
+    string scalar fname,
+    real scalar nrow,
+    real scalar ncol)
 {
     real scalar fh
     real matrix X
@@ -1212,7 +1229,11 @@ program parse_keep_drop, rclass
         else {
             * We only recast integers. Floats and doubles are preserved unless
             * requested or the target is a sum.
-            parse_ok_astarget, sourcevar(`var') targetvar(`var') stat(`collstat') `double' `weights'
+            parse_ok_astarget,   ///
+                sourcevar(`var') ///
+                targetvar(`var') ///
+                stat(`collstat') ///
+                `double' `weights'
             local recast = !(`r(ok_astarget)')
 
             if ( `recast' ) {