@@ -466,3 +466,176 @@ sent). Catching this is simpler than it might be because we assume that the
466466client only sends one ping at a time, and waits for the previous ping to be
467467acknowledged before sending a different ping. What should be the client's
468468behavior when it receives a duplicate (late) message?
469+
470+
471+ ### More on Search Tests
472+
473+ Now we will explore how the search tests work in more detail. This will help you
474+ to understand errors that you see so that you can modify your implementation to
475+ work best with the model checker. To do this, we'll see how the search test
476+ checks the correctness of our ping-pong implementation. Before continuing, you
477+ should understand the ping-pong protocol. You should also review the "Search
478+ Tests" section of the [ top-level README] ( ../../README.md ) .
479+
480+ At this point, we've told you that search tests check the correctness of the
481+ implementation by exploring the state graph (a directed graph where the vertices
482+ are states of the system and there is an edge from state ` u ` to ` v ` if there is
483+ a message or timer which can be delivered in state ` u ` to reach state ` v ` ). The
484+ state of the system consists of three parts: (1) the states of the nodes in the
485+ system, (2) the queue of timers pending for each node, and (3) the state of the
486+ network. Let's describe each of these in more detail.
487+
488+ First, the state of the nodes is relatively self-explanatory: it consists of all
489+ ` Node ` objects involved in the system. Next, the timers pending for each node
490+ can be seen as a queue where the queue ordering respects durations (as described
491+ in the top-level README). What's left is the state of the network, and here
492+ things differ from the view presented in the visual debugger. Specifically, in
493+ the visual debugger, when a message is duplicated, a new copy of the message
494+ appears. As messages can be duplicated any number of times, if the network state
495+ included the number of duplicates present in the system, then the state graph
496+ would be infinite. To avoid this, in search tests, the state of the network is
497+ represented by a set of all messages which have been sent. Messages are not
498+ removed from this set when they are delivered. This incorporates duplications,
499+ delays, and drops: once a message is sent, it can be delivered 0 times (drops),
500+ 1 time, or many times (duplicates), and it can be delivered after other messages
501+ and timers have been delivered (delays).
502+
503+ Now we can proceed to the example. The search test for lab 0 has a single server
504+ and a single client which sends 10 different ` Ping ` s. The test first attempts to
505+ find a sequence of events that leads to the client receiving ` Pong ` s for all of
506+ its ` Ping ` s. It then explores the entire state graph and checks that the ` Pong ` s
507+ received by the client match the ` Ping ` s sent. To simplify the example, we'll
508+ consider a client which only sends two ` Ping ` s with values "ping-1" and
509+ "ping-2". This is the search that would occur if you replace the first line of
510+ the search test in [ the test file] ( ./tst/dslabs/pingpong/PingTest.java ) with
511+
512+ ``` java
513+ initSearchState. addClientWorker(client(1 ), repeatedPings(2 ));
514+ ```
515+
516+ Let's explore what happens when we run the first BFS which looks for a state
517+ where the client has received results for all of its sent commands. To begin, we
518+ show the states and edges of this state graph, and then we explain the graph in
519+ more detail.
520+
521+ ![ State Graph] ( ./img/state-graph.png )
522+
523+ The vertex labels in the state graph describe the state of the nodes, timer
524+ queues, and network.
525+ * Recall that the state of a node (server/client) is simply the fields of the
526+ server/client object. The ` PingServer ` has an ` app ` field, while the
527+ ` PingClient ` has a ` serverAddress ` , ` ping ` , and ` pong ` field. The first two
528+ lines describe the state of the two nodes in the system (we omit the fields in
529+ the ` Node ` superclass as those details are not important to us).
530+ * The next two lines describe the client and server timer queues. These are
531+ given as a list where the list ordering must respect durations.
532+ * The last line(s) store the state of the network, all messages ever sent. These
533+ describe both the message contents and the sender/receiver of the message.
534+
535+ The very first state in the graph is the system's initial state -- that is, the
536+ state after all nodes have been initialized and the client has been told to send
537+ the first command (via ` sendCommand(Ping("ping-1")) ` ). At this point, the client
538+ has sent a ` PingRequest ` to the server and has set a ` PingTimer ` , but it has not
539+ yet received any results. After construction and initialization, the server's
540+ state is rather boring: it has a ` PingApplication ` with no fields, and it
541+ doesn't set any timers in ` init() ` .
542+
543+ From this starting state there are two possible events: we can fire the timer,
544+ or deliver the message.
545+ * If we fire the ` PingTimer ` , then the client will see that the timer matches
546+ the last sent ` Ping ` and there is no ` Pong ` received for the ` Ping ` , so it
547+ will resend the ` PingRequest ` to the server and reset the timer. Thus, the
548+ client's timer queue again has ` [PingTimer(Ping("ping-1"))] ` . Moreover, the
549+ network set already has this ` PingRequest ` from client to server, so the
550+ network set doesn't change either. Therefore, we return to the initial state.
551+ * If we deliver the ` PingRequest ` , the server executes the request and replies
552+ with a ` PongReply ` . The server's state and timer queue remain the same, but
553+ now the network has the ` PongReply ` that was sent from server to client. The
554+ original ` PingRequest ` remains in the network and can be delivered again.
555+
556+ This explains all of the edges from the starting state, and it explains how we
557+ reach the second state in the graph. The client is still waiting for results, so
558+ the BFS continues.
559+
560+ From this state, we can fire the timer or deliver either of the messages.
561+ * ** Test your understanding:** you should now be able to explain why firing the
562+ ` PingTimer ` or delivering the ` PingRequest ` returns us to the same state.
563+ * When delivering the ` PongReply ` , many things happen.
564+ * The client determines that the ` PongReply ` is a reply for the current
565+ command, so it updates its ` pong ` field to match.
566+ * The framework determines that the client received a result (via
567+ ` hasResult() ` ) and saves the result (via ` getResult() ` ) in the ` results `
568+ list.
569+ * Since the client received a result, the framework tells the client to send
570+ the next command (` sendCommand(Ping("ping-2")) ` ).
571+ * The client updates its ` ping ` field to this new command, and sets its
572+ ` pong ` field to null.
573+ * The client sends a ` PingRequest ` for this command to the server.
574+ * The client sets a ` PingTimer ` for this command. Since the timer ordering
575+ must respect durations, ` PingTimer(Ping("ping-1")) ` must fire before
576+ ` PingTimer(Ping("ping-2")) ` .
577+
578+ So, we've explained the edges leaving the second state, and we've explained why
579+ the third state is different from the second. In the third state, the client is
580+ still waiting for a result, and all received results match the expected results.
581+ Therefore, the BFS continues.
582+
583+ We can fire the first timer in queue or deliver any of the messages.
584+ * If we fire the first ` PingTimer ` in queue, then the client will recognize that
585+ the timer is out-of-date and therefore drop it. No other actions will be
586+ taken. So, we get a new state which is almost the same as the original, but
587+ the new client queue has only the second timer.
588+ * If we deliver ` ping-1 ` 's ` PingRequest ` or ` PongReply ` , then we return to the
589+ same state. Once again, you should be able to explain why this is the case.
590+ * If we deliver the ` PingRequest ` for ` ping-2 ` , then the server executes the
591+ ` Ping ` and replies. Thus, node states and timer queues remain the same, but
592+ the network now has a ` PongReply ` for ` ping-2 ` .
593+
594+ The BFS continues on in this fashion until it either exhausts all edges, finds
595+ an invariant violation (a state where the ` Pong ` returned doesn't match the
596+ ` Ping ` sent), or finds a state matching our goal.
597+
598+ In this case, the BFS finishes when it finds the state in the bottom-right,
599+ where the client is done and has received ` Pong ` s for all ` Ping ` s in its
600+ workload. All received results match the expected results. So, the BFS finishes
601+ and we have found a state matching the goal. At this point, we know that our
602+ system can reach a result. We also are more confident in the safety of our
603+ system, since in the states we examined, we checked that the received results
604+ matched the expected results.
605+
606+ Here are some questions you can use to test your understanding of search tests:
607+ * Consider the modification outlined earlier in this section: in
608+ ` handlePongReply ` , the client doesn't check that the pong value matches the
609+ ping value. Carry out the BFS by hand as we did above, and give the state at
610+ which an invariant violation occurs.
611+ * From the correct solution (the initially provided one), comment out the line
612+ ` send(new PongReply(p), sender) ` in the server's ` handlePingRequest ` . Now, it
613+ should be impossible for the client to receive results.
614+ * First, the theory: write out the state graph for this modified solution. How
615+ can the search test tell that the client's workload cannot be finished?
616+ * Now, the practice: run the test and see the error that the search test gives
617+ you.
618+ * Optionally, open the visual debugger and try to deliver messages to get a
619+ result to the client. For this simple liveness issue, it may seem
620+ unnecessary to use the visual debugger, but for the more complicated
621+ protocols you will work with, the debugger can help you to see where your
622+ implementation differs from your protocol design (or a situation you
623+ missed when designing the protocol).
624+ * The test in lab 0 has another search:
625+ ``` java
626+ searchSettings. clearGoals(). addPrune(CLIENTS_DONE )
627+ bfs(initSearchState);
628+ assertSpaceExhausted();
629+ ```
630+
631+ * This will run the BFS as described earlier, checking that the results match
632+ the expected results, but this test does not have a goal. When the search
633+ reaches a state where the client is done, the test will not end; instead, it
634+ will continue examining any remaining states where the client is not done.
635+ That is to say, the test "prunes" any parts of the search space which follow a
636+ state where the clients are finished. The test asserts that it can exhaust the
637+ search space; in other words, that in the time allotted, it can examine the
638+ entire graph (except the states which were pruned).
639+
640+ Carry out this pruned BFS by hand (still assuming that the client sends only 2
641+ commands rather than 10), and confirm that you can exhaust the search space.
0 commit comments