chkpoint_two & atomic_three: constructor cannot be called in parallel mode.

[pnik073]

Why we cannot or how can we use checkpoints in parallel mode?

I have fully working parallel execution following the basics described in https://cppad.readthedocs.io/latest/sthread_get_started.cpp.html .

However, when I try to introduce checkpoints I get the error atomic_three: constructor cannot be called in parallel mode when I use more than one thread. With 1 thread the same code with checkpointing works as expected. For the checkpoints I am following https://cppad.readthedocs.io/stable-2025/chkpoint_two_get_started.cpp.html#

My use case assumes AD vectors ax (the independent), some ay_common common intermediate output and multiple ay_specific specific outputs per thread. The pseudo-code looks like this:

...
CppAD::ADFun<double> f_1(ax, ay_common); // Here I tried also with the .Dependent as in the last ADFun call
CppAD::chkpoint_two<double> ay_common_checkpoint( f_1, "check_1", false, false, false, true);

...
CppAD::ADFun<double> f_2(ay_common, ay_specific); // Here I tried also with the .Dependent as in the last ADFun call
CppAD::chkpoint_two<double> ay_specific_checkpoint( f_2, "check_2", false, false, false, true);

...

// Record ay operations from tape ax
CppAD::Independent(ax);

// Define checkpoint sequence
ay_common_checkpoint(ax, ay_common); // I want to re-use this checkpoint
ay_specific_checkpoint(ay_common, ay_specific); // I want to produce more ay_specific inside the thread

CppAD::ADFun<double> f;
f.Dependent(ax, ay_specific); //ADFun with thread-local ax
f.capacity_order(0);
f.function_name_set("Thread_ADFun_"+std::to_string(iThread));

[bradbell]

Is the call

CppAD::chkpoint_two<double> ay_common_checkpoint( f_1, "check_1", false, false, false, true);

during sequential mode; i.e., when there is only one thread ?

[pnik073]

No, the above snippet is inside multiple concurrent threads. The goal is to reuse the ‘ay_common_checkpoint’ within each of these threads.

[bradbell]

Try moving chekpoint constructor

CppAD::chkpoint_two<double> ay_common_checkpoint( f_1, "check_1", false, false, false, true);

to the sequential section of the program. You should then be able to use that checkpoint object in parallel; i.,e., the calls of the form

ay_common_checkpoint(ax, ay_common)

can be made by many threads in parallel. In other words one checkpoint object can be used by many threads at the same time.

[bradbell]

@pnik073 Did the suggestion above work for you ?

[pnik073]

I am going to test now.

I am also looking into this to get some more ideas: https://cppad.readthedocs.io/stable-2025/chkpoint_two_compare.cpp.html

See full definition in the above link:

// Record a version of z = f[g(x)] without checkpointing
Independent(ax);
g_algo(ax, ay); 
f_algo(ay, az);
CppAD::ADFun<double> check_not(ax, az);

// Record a version of z = f[g(x)] with checkpointing
Independent(ax);
g_chk(ax, ay);
f_chk(ay, az);
CppAD::ADFun<double> check_yes(ax, az);

In my case, it might make sense (if it works) to try combining g_chk with f_algo inside each thread:

// Record a version of z = f[g(x)] with checkpointing mix inside the threads
Independent(ax);
g_chk(ax, ay);    // where g_chk was instantiated once in master thread
f_algo(ay, az);
CppAD::ADFun<double> check_mix(ax, az);

Additional background

The calculation case I am facing is a Monte-Carlo simulation where without AAD I would:

1. Evaluate g_algo 1000 times (in multi-threading mode), to produce Monte-Carlo time-paths
2. Evaluate f_algo 1000 times * 10000 "specific test" cases, by reusing the results of g_algo (also in multi-threading mode)

Also:

• The g_algo are stochastic differential equations (no conditionality in them)
• The f_algo are conditional calculation cases, given the state of the g_algo time-paths

[pnik073]

The checkpoint mix inside the threads indeed worked!

// Record a version of z = f[g(x)] with checkpointing mix inside the threads
Independent(ax);
g_chk(ax, ay);    // where g_chk was instantiated once in master thread
f_algo(ay, az);
CppAD::ADFun<double> check_mix(ax, az);

Now the challenge is if I can avoid re-evaluating g_chk 1000*10000 times ( as explained above ), while in no AAD mode I only need to evaluate 1000 times and re-use the results in 10000 "test cases". Is there a way to do it more efficiently?

The only way I can think of is to pre-calculate the derivatives I need (maybe sparse) for the g_algo/g_chk and apply the chain rule together with the derivatives of f_algo within the threads. Though this would require to pre-store an very big amount of derivatives (if not the derivatives are not sparse enough) before I enter in multi-threading mode.

[bradbell]

Glad to head that moving the checkpoint function constructor into the sequential code worked for you.

When you mention using AD are you differentiating f[g(x)] ?
Are you using forward or reverse mode ?
What is the dimension of the domain and range space of f ang g ?

[pnik073]

yes, I need the Jacobian of $f(g(x))$ with respect to x, $\mathbb{J_f(x)}$.
For dimensions assume something like,

$g: \mathbb{R}^{1000} \rightarrow \mathbb{R}^{1000\times200}$ and $f: \mathbb{R}^{1000\times200} \rightarrow \mathbb{R}^{10}$ per each Monte-Carlo scenario,

while $g$ is re-useable from multiple $f$ operations.

With respect to the AD mode,

• if I would evaluate from x to f at once, I would do reverse
• if I would split it in 2 steps from x to g ( I would do forward mode) and from g to f ( I would do reverse )

Makes sense?

[bradbell]

The simplest thing to do is to evaluate the Jacobian of f(g(x)) using reverse mode, once for each scenario.
Any other choice should be tested to make sure that it is faster than this choice.

When tryiing a method for splitting the calculation into g'(x) and f'( g(x) ), try some of the different sparse derivative choices for g'(x) if it is sparse.

If g'(x) is dense, perhaps it would be faster to make multiplies copies of its ADFun function (one for each thread) and do its Jacobian calculation in parallel,

[pnik073]

Indeed, I need to compare all possible combinations of implementations. The sparcity and ADFun copies are also candidates. I hope I can share a positive update about this. Thanks for the suggestions.

[bradbell]

Are you recording a different f for each sample ? If so, perhaps you should use dynamic parameters for the random samples ?
https://cppad.readthedocs.io/latest/Independent.html#dynamic

[pnik073]

Yes, each Monte-Carlo path will use its own ADFun f. If understand correctly, by building up on the above f(g(x)) example with the mixed check point, it could look like this then:

// In MASTER thread
std::vector<AD<double>> adynamic_randoms{1.,1.};

CppAD::Independent(ax, adynamic_randoms);

auto ay = someAdFunctionWithRandoms(ax,adynamic_randoms);

CppAD::ADFun<double> f_master(ax, ay);

// create checkpoint versions of f and g
bool internal_bool = false;
bool use_hes_sparsity = false;
bool use_base2ad = false;
bool use_in_parallel = true;

CppAD::chkpoint_two<double> f_chk( f_master, "check_mc", internal_bool, use_hes_sparsity, use_base2ad, use_in_parallel);

// -----------------
// In WORKER threads (with concurrent execution)
CppAD::Independent(ax_thread);

std::vector<double> adynamic_randoms_in_thread{10.,10.};
f_chk.new_dynamic(adynamic_randoms_in_thread); // Apply Random numbers on tape

f_chk(ax_thread,ay_g_thread);
ay_f_thread[0]  = ay_g_thread[0]*ay_g_thread[0]; // Dummy function

CppAD::ADFun<double> f;
f.Dependent(ax_thread, ay_f_thread); // ADFun with thread-local ax_thread
f.capacity_order(0);
f.function_name_set("sample_"+std::to_string(iPath));

auto jac_final = f.Jacobian(x_derivative);

The above pseudo-code works. Is that what you meant?

[bradbell]

If I understand the description above correctly, you can define
h(x,p) = f( g(x), p)
where x are the variables and p, the independent dynamic parameters, is one random sample.

The for each sample of the random parameters, you would use new_dynamic
https://cppad.readthedocs.io/latest/new_dynamic.html
to set the samples in h and them reverse mode to compute the Jacobian of h w.r.t. x for that p.