Love The Tool

[tclayton0594]

Hi Again,

I had posted a quick question that you guys were quick to help me debug (very much appreciated) and I was able to use exact-star to get some very promising results. We had been trying a few other reachable set solvers and have found this one to be one of the best. I was wondering if you guys had any sort of timeline/were still working on the activation functions that your solver allows. I know on your home page you mention using tanh/sigmoid as well. It would be really helpful to my team (Next Gen Controls at GM) if it did have this flexibility, especially with tanh as it is MATLAB's default and tends to give best performance on our nets.

Thanks,
Taylor

[trhoangdung]

Hi Taylor,

Thank you very much for your compliment. It encourages us to do a better job for CPS + ML verification. Our tool currently supports tanh and sigmoid. If you want to know how to use it, please send us a toy example and we will write an example code for you.

Best,

Tran

[ttj]

Thanks for the positive feedback, we appreciate it and are glad to hear you've found it useful!

We'll also work on a simple overall illustrative example with tanh/sigmoid in a network, but for now, you can see some usage in the existing tests for these, albeit it's not in a full network:

https://github.com/verivital/nnv/blob/master/nnv/tests/nn/fnn/logsig/test_LogSig_reach_star_approx.m

https://github.com/verivital/nnv/blob/master/nnv/tests/nn/fnn/tansig/test_TanSig_reach_star_approx.m

Sorry, the above links are now broken due to a recent repository restructuring, these should work:

https://github.com/verivital/nnv/blob/master/code/nnv/tests/nn/fnn/tansig/test_TanSig_reach_star_approx.m

https://github.com/verivital/nnv/blob/master/code/nnv/tests/nn/fnn/logsig/test_LogSig_reach_star_approx.m

[tclayton0594]

Awesome! I have time now to explore so will see if I can figure it out or I will be reaching out for additional support.

[tclayton0594]

How could I send you toy data that you would be able to write example code for me in case I can't get it working?

[ttj]

A network in a reasonable format (ONNX, Keras, matlab, etc.) would be easiest. If that's not possible, training data also could be fine as long as the overall training process is clear.

If it's fine to publicly share, you can add as an attachment to the issue, make a pull request, etc. If it's not, you can email us directly, our contact info should be available from the readme.

[tclayton0594]

Ok I am going to make a basic example in Matlab on one of their sample datasets and then post publicly then. I will include the trained model as a .mat file and in there I will also include the input and output sets as well.

[tclayton0594]

I attached a zip file with a .mat file in it. The ANN has 2 hidden layers and 10 neurons each. In the mat file, I also saved the inputs lower and upper bounds. I did not apply normalization pre-training but MATLAB does use mapminmax by default to map inputs between -1 and 1. Please let me know if you need anything else.
Toy_Tansig.zip

[trhoangdung]

Hi Taylor, I will test it and let you know soon. Tran

…

On Fri, Sep 13, 2019 at 10:22 AM tclayton0594 ***@***.***> wrote: I attached a zip file with a .mat file in it. The ANN has 2 hidden layers and 10 neurons each. In the mat file, I also saved the inputs lower and upper bounds. I did not apply normalization pre-training but MATLAB does use mapminmax by default to map inputs between -1 and 1. Please let me know if you need anything else. Toy_Tansig.zip <https://github.com/verivital/nnv/files/3610643/Toy_Tansig.zip> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18?email_source=notifications&email_token=ACSH2CHBNPSE2IQ7WNJCA7LQJOV45A5CNFSM4IUBAJP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6VLKGQ#issuecomment-531281178>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACSH2CCB4P36DQKMCTFOBQDQJOV45ANCNFSM4IUBAJPQ> .

[trhoangdung]

Hi Taylor,

Please find the analysis code for your example here: https://github.com/verivital/nnv/tree/master/code/nnv/examples/NN/Toy_TanSig

Please clone the newest version of nnv to run the code and let me know if you have any question.

load Engine_Toy_Tansig_net.mat;

nnvNet = FFNNS.parse(net); % parse the network trained from matlab

lb = [-1; -1];
ub = [1; 1];

I = Star(lb, ub);

fprintf('\nReachable set of the toy network:\n');
reachSet = nnvNet.reach(I, 'approx-star') % perform reachability analyis using approx-star

fprintf('\nRange of the output of the toy network:\n');
B = reachSet.getBox % get the output range

Best,

Tran

[tclayton0594]

Awesome, thanks a bunch Tran. I will let you know if I have any issues.