prep 0.8.0 release

Author: frankmcsherry

CHANGELOG.md

@@ -1,19 +1,36 @@
 # Changelog
+
 All notable changes to this project will be documented in this file.
 
 ## Unreleased
-- Many logging events have been rationalized. Operators and Channels should all have a worker-unique identifier that can be used to connect their metadata with events involving them. Previously this was a bit of a shambles.
 
-- The `Scope` method `scoped` now allows supports new scopes with non-`Product` timestamps. Instead, the new timestamp must implement `Refines<_>` of the parent timestamp. This is the case for `Product` timestamps, but each timestamp also refines itself (allowing logical regions w/o changing the timestamp), and other timestamp combinators (e.g. Lexicographic) can be used.
+## 0.8.0
 
-- Timestamps no longer need to be `Product<RootTimestamp,_>`. Instead, the `_` can be used as the timestamp. The exception here is that the creation of loop variables requires a `Product<_,_>` structured timestamp, and so as much as you might like to have a scope with a `usize` timestamp and a loop, and you *should* be able to, you cannot yet without making the type `Product<(), usize>`.
+This release made several breaking modifications to the types associated with scopes, and in particular the generic parameters for the `Child<'a, G: ScopeParent, T: Timestamp>` type. Where previously the `T` parameter would be the *new coordinate* to add to `G`'s timestamp, it is now the *new timestamp* including `G`'s timestamp as well. This was done to support a broader class of timestamps to be used, beyond always requiring product combinations with new timestamps.
 
-- The `RootTimestamp` and `RootSummary` types have been excised. Where you previously used `Product<RootTimestamp,T>` you can now use `Product<(),T>`, or even better just `T`. The requirement of a worker's `dataflow()` method is that the timestamp type implement `Refines<()>`, which .. ideally would be true for all timestamps but we can't have a blanket implementation until specialization lands (I believe).
+Beneficial fallouts include our ability to remove `RootTimestamp`, as dataflows can now be timestamped by `usize` or other primitive timestamps. Yay!
+
+### Added
 
 - The communication crate now has a `bincode` feature flag which should swing serialization over to use serde's `Serialize` trait. While it seems to work the ergonomics are likely in flux, as the choice is crate-wide and doesn't allow you to pick and choose a la carte.
 
-- The `loop_variable` operator now takes a timestamp summary for the timestamp of its scope, not just the timestamp extending its parent scope. The old behavior can be recovered with `Product::new(Default::default(), summary)`, but the change allows cycles in more general scopes and seemed worth it. The operator also no longer takes a `limit`, and if you need to impose a limit other than the summary returning `None` you should use the `branch_when()` operator.
+- Timestamps may now implement a new `Refines` trait which allows one to describe one timestamp as a refinement of another. This is mainly used to describe which timestamps may be used for subscopes of an outer scope. The trait describes how to move between the timestamps (informally: "adding a zero" and "removing the inner coordinate") and how to summarize path summaries for the refining timestamp as those of the refined timestamp.
+
+### Changed
+
+- Many logging events have been rationalized. Operators and Channels should all have a worker-unique identifier that can be used to connect their metadata with events involving them. Previously this was a bit of a shambles.
+
+- The `Scope` method `scoped` now allows new scopes with non-`Product` timestamps. Instead, the new timestamp must implement `Refines<_>` of the parent timestamp. This is the case for `Product` timestamps, but each timestamp also refines itself (allowing logical regions w/o changing the timestamp), and other timestamp combinators (e.g. Lexicographic) can be used.
+
+- Root dataflow timestamps no longer need to be `Product<RootTimestamp,_>`. Instead, the `_` can be used as the timestamp.
+
+- The `loop_variable` operator now takes a timestamp summary for the timestamp of its scope, not just the timestamp extending its parent scope. The old behavior can be recovered with `Product::new(Default::default(), summary)`, but the change allows cycles in more general scopes and seemed worth it. The operator also no longer takes a `limit`, and if you need to impose a limit other than the summary returning `None` you should use the `branch_when` operator.
+
+### Removed
+
+- The `RootTimestamp` and `RootSummary` types have been excised. Where you previously used `Product<RootTimestamp,T>` you can now use `Product<(),T>`, or even better just `T`. The requirement of a worker's `dataflow()` method is that the timestamp type implement `Refines<()>`, which .. ideally would be true for all timestamps but we can't have a blanket implementation until specialization lands (I believe).
 
+- Several race conditions were "removed" from the communication library. These mostly involved rapid construction of dataflows (data received before a channel was constructed would be dropped) and clean shutdown (a timely computation could drop and fail to ack clean shutdown messages).
 
 ## 0.7.0
 

Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 
 name = "timely"
-version = "0.7.0"
+version = "0.8.0"
 authors = ["Frank McSherry <[email protected]>"]
 readme = "README.md"
 
@@ -26,7 +26,7 @@ abomonation = "0.7"
 abomonation_derive = "0.3"
 timely_bytes = { path = "./bytes", version = "0.7" }
 timely_logging = { path = "./logging", version = "0.7" }
-timely_communication = { path = "./communication", version = "0.7" }
+timely_communication = { path = "./communication", version = "0.8" }
 
 [dev-dependencies]
 timely_sort="0.1.6"

examples/realtime.rs renamed to examples/openloop.rs

@@ -15,6 +15,9 @@ fn main() {
         let index = worker.index();
         let peers = worker.peers();
 
+        // re-synchronize all workers (account for start-up).
+        timely::synchronization::Barrier::new(worker).wait();
+
         let timer = std::time::Instant::now();
 
         let mut input = InputHandle::new();
@@ -66,7 +69,7 @@ fn main() {
             //
             //   1. Wait until previous batch acknowledged.
             //   2. Tick at most once every millisecond-ish.
-            //   3. Geometrically increasing outstanding batches.
+            //   3. Geometrically increase outstanding batches.
 
             // Technique 1:
             // let target_ns = if acknowledged_ns >= inserted_ns { elapsed_ns } else { inserted_ns };
@@ -78,6 +81,7 @@ fn main() {
             let scale = (inserted_ns - acknowledged_ns).next_power_of_two();
             let target_ns = elapsed_ns & !(scale - 1);
 
+            // Common for each technique.
             if inserted_ns < target_ns {
 
                 while ((insert_counter * ns_per_request) as u64) < target_ns {

src/progress/nested/reachability_neu.rs

@@ -72,7 +72,7 @@
 //! assert_eq!(results[2], ((Location::new_target(2, 0), 17), -1));
 //! ```
 
-use std::collections::{BinaryHeap, HashMap, HashSet, VecDeque};
+use std::collections::{BinaryHeap, HashMap, VecDeque};
 use std::cmp::Reverse;
 
 use progress::Timestamp;
@@ -249,8 +249,6 @@ pub struct Tracker<T:Timestamp> {
     /// Compiled summaries from each internal location (not scope inputs) to each scope output.
     // output_summaries: HashMap<Location, Vec<Antichain<T::Summary>>>,
     output_changes: Vec<ChangeBatch<T>>,
-
-    global_frontier: HashSet<(Location, T)>,
 }
 
 struct PerOperator<T: Timestamp> {
@@ -378,7 +376,6 @@ impl<T:Timestamp> Tracker<T> {
             worklist: BinaryHeap::new(),
             pushed_changes: ChangeBatch::new(),
             output_changes,
-            global_frontier: HashSet::new(),
         }
     }
 
@@ -486,37 +483,6 @@ impl<T:Timestamp> Tracker<T> {
         &mut self.pushed_changes
     }
 
-    /// A reference to the minimal pointstamps in the scope.
-    pub fn global(&self) -> &HashSet<(Location, T)> {
-
-        // A pointstamp (location, timestamp) is in the global frontier exactly when:
-        //
-        //  1. `self.pointstamps[location]` has count[timestamp] > 0.
-        //  2. `self.implications[location]` has count[timestamp] == 1.
-        //
-        // Such a pointstamp would, if removed, cause a change to `self.implications`,
-        // which is what we track for per operator input frontiers. If the above do not
-        // hold, then its removal either 1. shouldn't be possible, or 2. will not affect
-        // the output of `self.implications`.
-        //
-        // As we grind through changes to `self.implications` we *should* be able to
-        // notice changes to the above properties. At least, we can notice as the counts
-        // for `self.implications` changes to and from 1.
-
-        // There are some monotonicity properties we could perhaps exploit. A pointstamp
-        // may have its `self.poinstamps` incremented to non-zero only when it is in
-        // advance of its `self.implications`, it may then enter the global frontier
-        // when this count goes to one, and it may then depart when its count goes to zero.
-        // A pointstamp cannot return to the global frontier once departed.
-
-        // Alternately, perhaps, we could just have a slighly more complicated test for
-        // "is in global frontier" where we just test these properties for stashed updates.
-        // We might need to re-check for many (all?) stashed updates, unless we can track
-        // which have potentially changed (which we can do, I think).
-
-        unimplemented!()
-    }
-
     /// Indicates if pointstamp is in the scope-wide frontier.
     ///
     /// A pointstamp (location, timestamp) is in the global frontier exactly when: