Can azmul(my_constant, my_dynamic) simplify if the constant is 1 or 0?

[perrydv]

Hi @bradbell, we have had CppAD successfully deployed with nimble for a while now. Thanks for making this possible.

In azmul.hpp, lines 191-199 (the case where neither argument is variable and at least one is dynamic), would it work to catch cases when x or y is constant and exactly 0 or 1 and then avoid putting a zmul_dyn operator in the vector of dynamic parameter operators used by new_dynamic? In the preceding cases where x or y is variable, there is careful catching of cases with the other argument a constant 0 or 1 to avoid recording unnecessary operations. However, am I following correctly that this doesn't happen for the case where x and y are one dynamic and one constant?

The same question applies to add.hpp (and possibly others?).

I very well may not be following the internal details sufficiently.

Here is a summary of the use case that led me to this. (I can provide code but will try to explain.) Consider taping calculation of a quadratic form y = x^T A x where x is an n-by-1 matrix and A is an n-by-n matrix. y is a scalar. Everything is taped with your scalar multiplication and addition operators. Then consider obtaining the Hessian of this with respect to all elements of x (i.e., the correct answer is 2A) by triple taping: (i) tape y = x^T A x; (ii) tape calculation of the Jacobian of (i) by playing (i) forward 0 and reverse 1; and (iii) tape calculation of the Hessian of (i) by playing (ii) forward 0 (once) and then reverse 1, n times, once each with a single element of w being 1 instead of 0. Finally, optimize() the third tape.

In this case, if elements of A are all constant, the result is ideal: The optimize()d third tape has successfully obtained the result as simply twice the elements of A, already calculated as constants, and virtually no computations are needed. (Before optimize(), the third tape still includes the forward 0 calculations, but optimization successfully detects that none of that is needed for the Hessian elements, which all result from multiplication and addition of only constant arguments.)

However, if elements of A are all dynamic (for all three levels of taping), then the new_dynamic operations in the third tape get very large at a fast rate in relation to the size of A. And they are all (or almost all) multiplications by 1 or additions of 0. With n=500, creating the third tape (before optimizing it) looks like it takes about 10Gb. Playing that tape takes at least an additional 10Gb. (And that is before optimizing it, which seems to shorten the new_dynamic operations somewhat but does not eliminate identity multiplications or additions).

What I mean by using dynamic elements for all three levels of taping is that each tape include a dynamic vector that is used as the elements of A. When recording one tape by playing another tape, the playing is preceded by call to new_dynamic so that the dynamic vector for the new tape is used as the updated dynamic values of the tape that will be played.

I was surprised that constant vs dynamic would give such a huge difference in memory performance. Is there a way to make recording of dynamic operations catch cases where efficiency can be gained? Above, I tried to see where that might be done but might be off track.

I can definitely work around this but wonder if it is feasible and/or sensible to improve.

Thank you.
Perry

[bradbell]

Looking at the code in
https://github.com/coin-or/CppAD/blob/master/include/cppad/core/azmul.hpp

It appears that the 0, 1 case optimization for varialbes is not also done for dynamic parameters.

Do you think you could modify that azmul.hpp and do the zero / one optimization for dynamic paraemters (and make a corresponding pull request) ?

If so, please first make sure that you can run the check_all.sh script on your stystem; see
https://github.com/coin-or/CppAD/blob/master/bin/check_all.sh

This may require some setup like installing xrst
https://xrst.readthedocs.io/latest/

[perrydv]

Thanks. Sounds like this makes sense to you. I think I see what is needed. I'll give it a try.

[bradbell]

Make sure you use the master branch and can run bin/check_all.sh before making any changes.

[perrydv]

I made PR #232..

Note I describe that it is lacking tests and so likely isn't ready to merge, that pow looks like it might have a bug because it doesn't check that x or y are Constant when it checks for IdenticalZero (but I didn't test, and I didn't modify), and that I could not get bin/check_all.sh running despite spending a fair bit of time mucking around. I won't go through a lot of details. I got as far as:

sphinx-build -b linkcheck -j 1 build/rst build/html

pkgconfig/CMakeLists.txt:71: [broken] https://www.gnu.org/prep/standards/html_node/Directory-Variables.html: HTTPSConnectionPool(host='www.gnu.org', port=443): Max retries exceeded with url: /prep/standards/html_node/Directory-Variables.html (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7afe8e2e8aa0>, 'Connection to www.gnu.org timed out. (connect timeout=None)'))
Error: see messages above.

It does appear to me that the actual code tests ran. I am not sure. I also ran make check.