Prepare rel/stable 2.0.4 release (#783)

* Prepare beta release build 9 (#673)

* add lease to asset cmds (#575)

* fix Disassemble when multiple bnz have the same target label (#612)

add test

* Add PeerConnections to network telemetry (#607)

* Add PeerConnections to network telemetry.

* omit Endpoint for incoming connections.

* fix asset unit name display in goal account list (#633)

* Add --no-sig flag to goal clerk multisig sign (#647)

* add --no-sig flag to goal clerk multisig sign

* update err message

* change preimage -> template

* change template -> information

* Scan for ledger wallets more often (#638)

* add more robust ledger scanning, fix infinite recursion bug

* fix comment

* undo scan change

* still delete wallets we fail to close

* Improve missing msig preimage error message (#648)

* improve missing msig preimage error message

* improve err msg

* Add support for https for telemetry servers (#649)

* Add support for https for telemetry servers.

* typo : udo -> udp

* Fixed few typos.

* goal listpartkeys display error (#641)

* Add logging for the telemetry server connections (#661)

* Add logging for the telemetry server connections.

* Revert unintended change.

* Improve error message.

* Avoid upgrading boost on travis Mac builds (#669)

* specify a boost version for the mac build.

* try to prevent boost update on travis mac builds.

* Build 9

* Abort algod startup if logging.config file has bad permissions (#662)

* This should prevent telemetry event loses on systems with
  invalid permissions on ~/.algorand/logging.config file
* Another possible workaround is to relax default config path mask in
  **cmd/goal/commands.go:ensureCacheDir** from 700 to 744.
  This is not implemented because of possible security risk.

Co-authored-by: Max Justicz <[email protected]>
Co-authored-by: algobolson <[email protected]>
Co-authored-by: Evan Richard <[email protected]>
Co-authored-by: pzbitskiy <[email protected]>

* Fix - Indexer now shows received transactions (#684) (#688)

-- Adding receiver function to transaction that returns the receiver of a transaction
-- Fix indexer to show received transactions

Co-authored-by: Rotem Hemo <[email protected]>

* Release Beta 2.0.10 (#743)

* Bump mainnet pregen to 1.0. (#569)

* add lease to asset cmds (#575)

* fix Disassemble when multiple bnz have the same target label (#612)

add test

* Replacing apt by apt-get (#610)

* Add PeerConnections to network telemetry (#607)

* Add PeerConnections to network telemetry.

* omit Endpoint for incoming connections.

* Fix license errors, enable check_license in travis.

* Remove trailing whitespace.

* add ?raw=1 to local block api to return msgpack bytes with full data (#621)

* Let dsign sign arbitrary bytes, not just txids (#577)

* Add markdown docs for `limit-order-a`, Fix `hltc` -> `htlc` (#619)

* Created `test_release.sh` to test centos|fedora|ubuntu images (#613)

* Created `test_release.sh` to test centos|fedora|ubuntu images

* Incorporate some review suggestions (more to come):

- change `apt` to `apt-get`
- remove command to start the node
- add `ENTRYPOINT` command to build image and test in one command
- streamline command that downloads release and cleanup
- moved script to `./test/packages/'
- make `apt-get update` with the env var a one-liner

* Add ability to pass bucket, channel and aws creds

* Ensure aws creds are in env before starting

* Make colorized text more readable

* Break script into `build` and `run` operations

* Run `update.sh` at RUN time

This is another intermediate step.  The installer is now being run at
runtime, but it's not allowing for testing any binaries, such as
`algod`.

At this point, there are a couple different options to proceed, and I
think it's best if Will, Tsachi and I talk more about the options.

* We're not writing the Dockerfile to disk before running it.

See my explanatory comment in the script.

* Added new `post_deploy` stage and our script

* Adding new `scripts/travis/test_release.sh` script

This simply calls `./test/packages/test_release.sh`.
Also, added name to `allow_failures`.

* Add filtering for new `post_deploy` stage

* Simplified the release scripts that build images to push to docker hub (#623)

* Simplified the release scripts that build images to push to docker hub

In pushing the updated images to docker hub, I noticed that the
Dockerfiles and the shell scripts were only differentiated by the
network name (stable|testnet).

The only file in the dir is now `build_stable.sh`.  It accepts a sole
argument, `-n` or `--name`.  It will default to "stable", so the for
that image it's only necessary to run `./build_stable.sh` with no args.

For "testnet", simply call the script like this: `build_stable.sh -n
testnet`.

The Dockerfile will be automatically created and passed to the `docker
build` command via `stdin`.

* Removed the case block for cli arguments

Now, testing for either "mainnet" or "testnet" and returning early if
neither value is present (defaults to "mainnet").

Also, changed the name to `build_releases.sh` since "stable" is no
longer applicable.

* Add `export SHELLOPTS` to teal tests. (#627)

* Add `goal ledger block` (#622)

* add goal ledger rawblock cmd

* Bring `shellcheck` into the build process (#626)

* Bring `shellcheck` into the build process

Let's use bitwise operations to determine package presence

* Added `check_shell` target to Makefile

* Move install of shellcheck into `scripts/configure_dev.sh`

Also, add shellcheck dependency to other dockerfiles.

* Use `find` command in make target instead of recursive globbing

What's up with the `exec +` syntax?

From the man page:
```
-exec command {} +
      This variant of the -exec action runs the specified
command on the selected files,  but  the  command  line  is  built  by
      appending  each  selected  file name at the end; the total
number of invocations of the command will be much less than the
      number of matched files.  The command line is built in
much the same way that xargs builds its command  lines.   Only  one
      instance  of  `{}'  is  allowed within the command, and
(when find is being invoked from a shell) it should be quoted (for
      example, '{}') to protect it from interpretation by
shells.  The command is executed in the starting  directory.   If  any
      invocation  returns  a  non-zero  value  as  exit status,
then find returns a non-zero exit status.  If find encounters an
      error, this can sometimes cause an immediate exit, so some
pending commands may not be run at all.  This variant of  -exec
      always returns true.
```

* Only check for missing dependencies

List any that are missing and the echo the script to run to install.

* Fix issue on macOS to make script portable (#632)

* Remove "Created new rootkey/partkey" spam message. (#629)

* fix asset unit name display in goal account list (#633)

* Ensure that the proper channel is passed to `test_release.sh` (#634)

* Minor improvements to `test_release.sh` script (#636)

- Removed a redundant `exit` statement.
- Added script name to error statement.

* Cleanup evalAux (#628)

* remove evalAux which hasn't been used since before 1.0
* comment removal of auxdata column

* Add --no-sig flag to goal clerk multisig sign (#647)

* add --no-sig flag to goal clerk multisig sign

* update err message

* change preimage -> template

* change template -> information

* Scan for ledger wallets more often (#638)

* add more robust ledger scanning, fix infinite recursion bug

* fix comment

* undo scan change

* still delete wallets we fail to close

* Exit early if `test_release.sh` script fails (#643)

* Improve missing msig preimage error message (#648)

* improve missing msig preimage error message

* improve err msg

* Add support for https for telemetry servers (#649)

* Add support for https for telemetry servers.

* typo : udo -> udp

* Fixed few typos.

* goal listpartkeys display error (#641)

* Fixing arm64 environment issues (#653)

1) python3-venv libffi-dev libssl-dev
libffi-dev (and libssl-dev) are needed by the cryptography package builder for python in e2e_basic_start_stop.

2) exporting GOPATHBIN needed to run algotmpl in template e2e tests.

* Test pre-packaged executable on variety of linux platforms (#651)

* Add platform testing using docker for generated binaries.

* Fix path.

* Apply reviewer's requested changes.

* Reduce e2e_go_tests execution time twice (#645)

There are seven major contributors to integration tests running time
TestOnlineOfflineRewards (1248.64s)
TestAssetConfig (364.71s)
TestRewardRateRecalculation (226.78s)
TestStartAndEndAuctionTenUsersOneBidEach (196.34s)
TestNoDepositAssociatedWithBid (189.74s)
TestDeadbeatBid (188.70s)
TestStartAndCancelAuctionNoBids (183.35s)

This commit considers only first three.

1. Fixing rewards interval in config for TestRewardRateRecalculation
from 25 to 10 reduces time twice:
TestRewardRateRecalculation (119.34s)

2. Fixing initialRound in TestOnlineOfflineRewards test
from 301 to 11 reduces time 15 times:
TestOnlineOfflineRewards (73.80s)

3. TestAssetConfig looks long by design - commits and waits max allowed assets

4. Address TODO in run_integration_tests.sh.
Now e2e_client_runner calls 'goal network delete' to reflect this removal

Refers #508

* Promote test_release.sh so that it won't conflict with release testing. (#655)

* Fix concurrent access to wallet handles cache in goal (#654)

* Fix concurrent access to wallet handles cache in goal

* In rare cases (i.e. e2e tests run in parallel on the same network)
a race cond happens when accessing goal.cache/walletHandles.json file
* Introduce advisory locking on the mentioned file
* Implementation is extendable by implementing *locker* interface
  for specific platform and providing a new *newLockedFile* constructor.

* Address PR review notes

* Do no truncate before obtaining the lock
* Increase waiting interval to 10 ms
* Simplify newLockedFile constructor

* Allow upgrades to specify the delay before their execution. (#650)

This replaces UpgradeWaitRounds with MinUpgradeWaitRounds and
MaxUpgradeWaitRounds.  Proposers specify an upgrade's delay given
their own ApprovedUpgrades, encoding the proposed delay in
the UpgradeVote.  Verifiers check that the delay sits between
MinUpgradeWaitRounds and MaxUpgradeWaitRounds (inclusive).

This commit adds this functionality but does not change current behavior.

* Set explicit 30 sec timeout for AlgorandGoal::RawSend in expect test (#658)

* Should help with sporadic failures when we send and TEAL in groups

* Support variable-delay protocol upgrades in ConsensusFuture. (#659)

Also add some unit tests for variable-delay protocol upgrades.

* Shant/catchup stop on unapproved (#660)

* A fix for arm64 failures

One observation from the failures is that the test timeouts could be the cause of the failure.
Expect scripts when called from go test using CombinedOutput is behaving strange (slow).
Replacing CombinedOutput with Run.

* DRAFT: this PR is a draft to experiment with test failures on ARM system.

Disabling tests, that failes sporadically on mac, on ARM as well.
Adding a utility to controll test skips.

adding missing file and change.

* DRAFT: this PR is a draft to experiment with test failures on ARM system.

Disabling tests, that failes sporadically on mac, on ARM as well.
Adding a utility to controll test skips.

adding missing file and change.
Fixing errors and adding comments.

* Fixing merge and comment.

* added comment

* Stop catchup on unapproved protocol round

Catchup to stop before fetching the next round if the round protocol is not approved by the node

* Some fixex. Review comments from Tsachi.

* File accidentally added here. removing.

* Reverting changes mistakenly added to this branch.

* Adding comment changes.

* Partially working test

* Adding test to catchup stop on unsupported block
Using s.cancel we are droppng the last block.

* More tests and development to the catchup service

* Stop the catchup before fetching the round with un-approved protocol.

The catchup service will save the round when an an-approved protocol
update will take place. Then, before creating a task to fetch a round,
will check if the next round is when an an-approved protocol round
begins, and stops the catchup service.

The ledger should have the round with NextProtocolSwitchOn to stop
the un-approved round from getting fetched.

The added test covers the edge cases which may or may not happen
when the service runs.

* Stop the catchup before fetching the round with un-approved protocol.

The catchup service will save the round when an an-approved protocol
update will take place. Then, before creating a task to fetch a round,
will check if the next round is when an an-approved protocol round
begins, and stops the catchup service.

The ledger should have the round with NextProtocolSwitchOn to stop
the un-approved round from getting fetched.

The added test covers the edge cases which may or may not happen
when the service runs.

Addressing Tsachi's review comments.

* Combine condition blocks

* Fixing an error in the log info statement.

* Compile linux/amd64 binaries with static linking (#625)

* Test static compilation.

* remove -fPIC

* Try with ubuntu 18.04, since it has newer GCC.

* exclude buildmode from test builds.

* Fixed missed buildmode.

* Refactor.

* Add logging for the telemetry server connections (#661)

* Add logging for the telemetry server connections.

* Revert unintended change.

* Improve error message.

* add bool support to algocfg (#667)

e.g.
`algocfg set -p EnableProcessBlockStats -v true`

* Reduce execution time of expect tests (#665)

* CombinedOutput blocks on copying empty stderr stream from expect that
  causes at least 60 sec timeout for most of the tests
* This implementation uses a temp time for stderr accumulation.
  In this case exec.Cmd does not run goroutines for reading child's actual stderr.
* 655 sec (before) vs 205 sec (after)

* Avoid upgrading boost on travis Mac builds (#669)

* specify a boost version for the mac build.

* try to prevent boost update on travis mac builds.

* Abort algod startup if logging.config file has bad permissions (#662)

* This should prevent telemetry event loses on systems with
  invalid permissions on ~/.algorand/logging.config file
* Another possible workaround is to relax default config path mask in
  **cmd/goal/commands.go:ensureCacheDir** from 700 to 744.
  This is not implemented because of possible security risk.

* Add error logging for getting a cached wallet handle (#663)

Needed to debug
'Couldn't read password: inappropriate ioctl for device' error message in tests

* Update license date 2019 -> 2020 (#674)

* Change 2019 -> 2020

* Update readme.

* Update copyright to use date range. (#676)

* Tee existing tests so we can review output before piping it forward. (#677)

* Make gracefull exit of a node that is waiting for WaitForBlock call (#679)

* Make gracefull exit of a node that is waiting for WaitForBlock call.

* Add comment.

* Remove tput where not supported by terminal (#682)

* Remove tput where not supported by terminal.

* send tput errors to dev/null

* Fix bad constants.

* Avoid waiting for block that won't be reached due to unsupported protocol upgrade. (#681)

* Fix - Indexer now shows received transactions (#684)

-- Adding receiver function to transaction that returns the receiver of a transaction
-- Fix indexer to show received transactions

* Undo teeing to dev/tty as it doesn't work well in terminal free environments. (#689)

* Improve lockFile error handling (#687)

* Better lockFile error handling.

* Make blocking locker.

* Fix F_OFD_GETLK constant.

* bugfix.

* Try platform specific code.

* use unix package to include F_OFD_SETLKW

* remove unused imports.

* Rename files.

* Catchup service stop on unsupported and e2e test  (#685)

* A fix for arm64 failures

One observation from the failures is that the test timeouts could be the cause of the failure.
Expect scripts when called from go test using CombinedOutput is behaving strange (slow).
Replacing CombinedOutput with Run.

* DRAFT: this PR is a draft to experiment with test failures on ARM system.

Disabling tests, that failes sporadically on mac, on ARM as well.
Adding a utility to controll test skips.

adding missing file and change.

* DRAFT: this PR is a draft to experiment with test failures on ARM system.

Disabling tests, that failes sporadically on mac, on ARM as well.
Adding a utility to controll test skips.

adding missing file and change.
Fixing errors and adding comments.

* Fixing merge and comment.

* added comment

* Stop catchup on unapproved protocol round

Catchup to stop before fetching the next round if the round protocol is not approved by the node

* Some fixex. Review comments from Tsachi.

* File accidentally added here. removing.

* Reverting changes mistakenly added to this branch.

* Adding comment changes.

* Partially working test

* Adding test to catchup stop on unsupported block
Using s.cancel we are droppng the last block.

* More tests and development to the catchup service

* Stop the catchup before fetching the round with un-approved protocol.

The catchup service will save the round when an an-approved protocol
update will take place. Then, before creating a task to fetch a round,
will check if the next round is when an an-approved protocol round
begins, and stops the catchup service.

The ledger should have the round with NextProtocolSwitchOn to stop
the un-approved round from getting fetched.

The added test covers the edge cases which may or may not happen
when the service runs.

* Stop the catchup before fetching the round with un-approved protocol.

The catchup service will save the round when an an-approved protocol
update will take place. Then, before creating a task to fetch a round,
will check if the next round is when an an-approved protocol round
begins, and stops the catchup service.

The ledger should have the round with NextProtocolSwitchOn to stop
the un-approved round from getting fetched.

The added test covers the edge cases which may or may not happen
when the service runs.

Addressing Tsachi's review comments.

* Combine condition blocks

* Fixing an error in the log info statement.

* Draft:

Test for upgrading a node while keeping another node not upgradable
goal node status field for informing if the node is upgradable

* Catchup service stop on unsupported, ode status message, and e2e test

In this change:
Updated catchup service to stop on unsupported and not unupgradable.
Updated goal node status to inform when the catchup service is stopped.
Updated goal node status by removing last synced information.
Added e2e test for stopped catchup service on unsupported protocol.

* Separating goal changes from this PR.

Separating goal changes from this PR.
goal changes are in PR: #686

* review comment: use NotEqual instead of True

* Make ARM64 build mandatory. (#694)

* Updates to the goal node status (#686)

* Updates to the goal node status

This change is splitting the goal section from PR: #685
Updated goal node status to inform when the catchup service is stopped.
Updated goal node status by removing "Synced Since Startup" field.

* Adding parameter StoppedAtUnsupportedRound to v1.NodeStatus and node.StatusReport

* Adding check to libgoal Client StoppedAtUnsupportedRound in v1.NodeStatus true and false values.

* Review comments from Tsachi: using the timeout in select

* Updating the test to reflect the removal of: has synced since startup.

* telemetry recorded locally as info log (#666)

config.json: {"TelemetryToLog":true}
logging.config: {"Enable":false,"SendToLog":true}

* Relax StartNetwork regex (#696)

* relax StartNetwork regex.

* Another attempt.

*  Two fixes to basicCatchup_test: cloned node not stopped and env var conflict (#697)

* Updates to the goal node status

This change is splitting the goal section from PR: #685
Updated goal node status to inform when the catchup service is stopped.
Updated goal node status by removing "Synced Since Startup" field.

* Adding parameter StoppedAtUnsupportedRound to v1.NodeStatus and node.StatusReport

* Adding check to libgoal Client StoppedAtUnsupportedRound in v1.NodeStatus true and false values.

* Review comments from Tsachi: using the timeout in select

* Two fixes to basicCatchup_test: cloned node not terminated and env var collision

1) TestBasicCatchup and newly added TestStoppedCatchupOnUnsupported
create a new node by cloning one of the network nodes. When
fixture.Shutdown() stops the original network nodes, leaves the cloned
node running. This change adds function shutDownClonedNode to stop the
cloned nodes.

2) In TestStoppedCatchupOnUnsupported, an env variable is used to
delete ConsensusCurrentVersion, so that the cloned node behaves as if
its binary does not support the consensus version. However, when the
TestBasicCatchup runs in parallel, it also picks up the env variable,
and consequently deletes ConsensusCurrentVersion from the Consensus
map. When this happens, TestBasicCatchup sporadically fails.

In this change, instead of having ConsensusTestUnupgradedProtocol
upgrade to ConsensusCurrentVersion, or deleting
ConsensusCurrentVersion so it cannot be upgraded, it sets up
ConsensusTestUnupgradedProtocol to upgrade to
ConsensusTestUnupgradedToProtocol. Hence, the env variable is used to
delete ConsensusTestUnupgradedToProtocol. This way the conflict with
other tests is eliminated.

* Fixing golint by addint comment.

* Tsachi's review comment: unsetting the env var.

* Make scripts executable. (#702)

* More reliable fetcher unit tests. (#708)

* Avoid starting the Telemetry service when logging is disabled (#703)

if remote telemetry is not enabled, do not start uri update service
add a nil check

* Shutdown kmd when test fixture is going down. (#709)

* Fix unit test. (#711)

* Execute e2e tests one at a time on arm64 (#701)

* Test changes.

* Better error reporting on goalFixture

* Add version query for kmd startup.

* Few more test cases to cover.

* try to wait.

* changes

* Update.

* Move KMD shutdown to network.

* Add some debug messages to figure out what's going on.

* Fix script bug.

* Fix proper KMD shutdown via the KMDFixture

* Run the tests one at a time only on arm64

* Updating according to review.

* Disable pprof endpoints by default (#693)

* enable go profiler for netdeploy

* add EnableProfiler to ConfigJSONOverride

* Update the makefile to skip the static linking when compiling on centos. (#713)

* Fail e2e-go tests when node panics (#699)

* Fail test on panic

* few more touchups.

* sync

* bugfix.

* Update few more usecases.

* Refactoring

* Simplify.

* undo network referencing.

* undo few func-ptr.

* undo some more stuff.

* Update method names

* Few more touchups.

* Build release job (#698)

* Initial commit

* Added Jenkinsfile

* Updated Jenkinsfile

* Works until GPG IPC

* Move build files into new release/ dir

Also, renamed files {build_,}release.sh and {build_,}setup.sh

* Path issues

* Use t2.xlarge instance type (4 vCPUs, 16GB ram)

* Restructuring

* shellchecked

* fix bug

* Added new `socket.sh` file

* Trying to build rpm

* Bump up disk size of ec2 instance

* more attempts to make rpm

* more fixes

* move /stuff -> /root/stuff

* wip

* moved to correct paths

* Have `release` have its own start and kill ec2 instance scripts

* use buildhost scripts after all

* Make sure the gpg key name matches!!!!!

-%_gpg_name Algorand RPM <[email protected]>
+%_gpg_name rpm algorand <[email protected]>

* fixes

* Add upload stage to pipeline

* Add tag stage to pipeline

* more fixes

* Move start/stop ec2 instance scripts back into release/

* Add ability to dynamically set branch

* Added controller/ subdir

* Some cleanup

* Adding tag support

Moved `Jenkinsfile` into controller/ subdir.

* Move build_env build.sh -> setup.sh

Moved socket.sh -> controller/socket.sh

* Revert buildhost changes

* some cleanup

* fix build

* test packages locally

* upload packages to s3 test bucket

* restructure

* misc

* fix build

* Add Jenkins parameters

* fix build

* Move commands into Jenkinsfile into stages/

* fix build

* Make test stage more explicit

* fix build

* Implementing reviewer suggestions

* Added debug info

* fix build

* Merge into master

* implement reviewer suggestions

* turn off test stage

* fix build

* fix build

* fix build

* Update readme

* removed unneeded archive/ dir

* Use service-wide logger instead of logging.Base() in agreement (#714)

* Switch from default logger to pre-configured logger
  in some components of agreement service

* Mark some of the slow e2e tests as such (#719)

* Mark some of the slow e2e tests as such.

* Move shorttest flag to be set at top level.

* Wait test less restrictive. (#718)

* Move slow test to get executed on nightly builds (#721)

* Move some more test to be "slow tests", and modify short test condition so that we will run
the long tests on nightly builds only.

* Fix elif -> else

* Faster upgrade tests. (#722)

* Disable failing test. (#724)

* Generate docs for algokey.

* s/goal/algokey

* Improve algons error logging (#733)

* Write body when erroring on SRV/DNS records update.

* Few more error messages.

* ledger/eval refactor (#700)

refactor ledger/eval block validation
don't do crypto+lsig validation in eval
fix sync in backlog executer queue
clean up lots of logging to make tests quieter

* Fix a bug in Credential.lowestOutput caused by improper domain separation (#716)

* Fix a bug in Credential.lowestOutput caused by improper domain separation

The bug causes larger accounts to be block proposers more often than should happen based on their fraction of online stake. This patch will cause nodes to vote for a protocol upgrade that fixes the buggy behavior. After the protocol upgrade goes through, all the upgrade-related code in this commit should be removed, as it's not necessary to retain the old buggy behavior for catchup. (For convenience code to be removed is marked with a "TODO(upgrade)" comment.)

* Typofix; fix merge issue

* Fix test

* Add a comment to make the linter happy

* Typo fixes

* Goal docs tweaks  (#731)

* test all `goal ... -h` (#730)

* test all `goal ... -h`
ensures no conflicting subcommand options
adds less than 2 seconds to test time

* review tweak, rearrange to sub test script

* actually pass args

* grr, arg

* Move EnsureDigest logic into the catchup service (#726)

* Move EnsureDigest logic into the catchup service.

* update unit tests.

* Add unit testing for new catchup feature.

* updating per review.

* Add handing for concurrently updated round.

* Add comment.

* typo

* Correct the quit semantics.

* Faster stringer implementation for Address (#736)

* Faster stringer implementation.

* Optimize UnmarshalChecksumAddress as well.

* Add comment.

* Interconnect relays on a locally deployed network (#742)

* Revert "ledger/eval refactor (#700)"

This reverts commit c78ada0.

* Revert "Cleanup evalAux (#628)"

This reverts commit 06a488d.

* Build Number 10

* Undo upgrade to ConsensusV21

* undo warning message.

* Change travis builder to bionic.

* set ConsensusCurrentVersion = ConsensusV20

* Revert "set ConsensusCurrentVersion = ConsensusV20"

This reverts commit 506c049.

* Revert "Undo upgrade to ConsensusV21"

This reverts commit 2dc4932.

* Revert "Fix a bug in Credential.lowestOutput caused by improper domain separation (#716)"

This reverts commit ec4d9b5.

* Bugfix: Fix last relevant proposal period in agreement protocol. (#746)

When retrieving the last relevant period corresponding to a proposal-value, the
proposal store inside the agreement protocol does not properly check that the
particular period returned actually matches the passed-in proposal-value.
Instead, the proposal store returns the last period seen for *any*
proposal-value.

When the agreement state machine receives a proposal payload, the proposal store
checks whether this payload matches any proposal-value known to be relevant in
the current round.  If it does, the state machine tells the crypto verifier to
verify the new payload.

As an optimization, the proposal store in the state machine also tags the
payload with the last period in which it is relevant (and whether the matching
proposal-value is pinned).  The crypto verifier halts concurrent verification of
any payload from that period.  Separately, the proposal store does not attempt
to verify payloads more than once, caching past payloads it has pipelined.

For this optimization to be correct, the last relevant period must be correct;
otherwise, the network will permanently stall if the following occurs:

- In period p, the network observes a best proposal value of v, but it sees
  neither the payload B corresponding to v nor a threshold of soft-votes for
  B (seeing such a threshold pins B, preventing the crypto verifier from
  cancelling).

- An attacker is able to see B.

- In period p+1, the network attempts to agree on a new proposal value v'
  corresponding to the payload B'.

- After half of the network has received B' but has _not_ finished verifying it,
  the attacker sends this half the payload B.  This half will cancel
  verification of B' (since it erroneously associates B with period p+1) and
  will permanently ignore any future broadcasts of B' (which was cached in the
  proposal store).

- If the other half has already staged B', the network will stall permanently,
  since it will be unable to commit B'.

Fixes #710. Thanks to @xixisese for reporting this bug.

Co-authored-by: Derek Leung <[email protected]>
Co-authored-by: Max Justicz <[email protected]>
Co-authored-by: algobolson <[email protected]>
Co-authored-by: Will Winder <[email protected]>
Co-authored-by: btoll <[email protected]>
Co-authored-by: Evan Richard <[email protected]>
Co-authored-by: algonautshant <[email protected]>
Co-authored-by: pzbitskiy <[email protected]>
Co-authored-by: Rotem Hemo <[email protected]>
Co-authored-by: algoradam <[email protected]>

Co-authored-by: Max Justicz <[email protected]>
Co-authored-by: algobolson <[email protected]>
Co-authored-by: Evan Richard <[email protected]>
Co-authored-by: pzbitskiy <[email protected]>
Co-authored-by: Rotem Hemo <[email protected]>
Co-authored-by: Derek Leung <[email protected]>
Co-authored-by: Will Winder <[email protected]>
Co-authored-by: btoll <[email protected]>
Co-authored-by: algonautshant <[email protected]>
Co-authored-by: algoradam <[email protected]>

Author: 11 people

.gitignore

@@ -51,3 +51,6 @@ crypto/libsodium-fork/build-aux/
 
 # doc intermediates
 data/transactions/logic/*.md
+
+*.pem
+

.travis.yml

@@ -1,4 +1,4 @@
-dist: xenial
+dist: bionic
 go:
   - "1.12"
 go_import_path: github.com/algorand/go-algorand
@@ -17,14 +17,16 @@ stages:
     if: branch =~ /^rel\// AND type != pull_request
   - name: deploy
     if: branch =~ /^rel\// AND type != pull_request
+  - name: post_deploy
+    if: branch =~ /^rel\// AND type != pull_request
 
 jobs:
   allow_failures:
-    - name: External ARM64 Build
     - name: External ARM64 Deploy
     - name: External ARM64 Integration Test
     - name: External ARM Build
     - name: External ARM Deploy
+    - name: Test Release Builds
   include:
     - stage: build_commit
       os: linux
@@ -110,7 +112,9 @@ jobs:
     - stage: deploy
       name: Ubuntu Deploy
       os: linux
-      script: scripts/travis/deploy_packages.sh
+      script:
+        - scripts/travis/deploy_packages.sh
+        - scripts/travis/test_release.sh
     - # same stage, parallel job
       name: MacOS Deploy
       os: osx

Makefile

@@ -15,11 +15,21 @@ BUILDCHANNEL     ?= $(shell ./scripts/compute_branch_channel.sh $(BUILDBRANCH))
 DEFAULTNETWORK   ?= $(shell ./scripts/compute_branch_network.sh $(BUILDBRANCH))
 DEFAULT_DEADLOCK ?= $(shell ./scripts/compute_branch_deadlock_default.sh $(BUILDBRANCH))
 
+GOTAGSLIST          := sqlite_unlock_notify sqlite_omit_load_extension
+
 ifeq ($(UNAME), Linux)
 EXTLDFLAGS := -static-libstdc++ -static-libgcc
+ifeq ($(ARCH), amd64)
+# the following predicate is abit misleading; it tests if we're not in centos.
+ifeq (,$(wildcard /etc/centos-release))
+EXTLDFLAGS  += -static
+endif
+GOTAGSLIST  += osusergo netgo static_build
+GOBUILDMODE := -buildmode pie
+endif
 endif
 
-GOTAGS          := --tags "sqlite_unlock_notify sqlite_omit_load_extension"
+GOTAGS      := --tags "$(GOTAGSLIST)"
 GOTRIMPATH	:= $(shell go help build | grep -q .-trimpath && echo -trimpath)
 
 GOLDFLAGS_BASE  := -X github.com/algorand/go-algorand/config.BuildNumber=$(BUILDNUMBER) \
@@ -56,6 +66,9 @@ vet:
 check_license:
 	./scripts/check_license.sh
 
+check_shell:
+	find . -type f -name "*.sh" -exec shellcheck {} +
+
 sanity: vet fix lint fmt check_license
 
 cover:
@@ -127,7 +140,7 @@ $(KMD_API_SWAGGER_INJECT): $(KMD_API_SWAGGER_SPEC) $(KMD_API_SWAGGER_SPEC).valid
 build: buildsrc gen
 
 buildsrc: crypto/lib/libsodium.a node_exporter NONGO_BIN deps $(ALGOD_API_SWAGGER_INJECT) $(KMD_API_SWAGGER_INJECT)
-	go install $(GOTRIMPATH) $(GOTAGS) -ldflags="$(GOLDFLAGS)" ./...
+	go install $(GOTRIMPATH) $(GOTAGS) $(GOBUILDMODE) -ldflags="$(GOLDFLAGS)" ./...
 
 SOURCES_RACE := github.com/algorand/go-algorand/cmd/kmd
 
@@ -243,4 +256,4 @@ dump: $(addprefix gen/,$(addsuffix /genesis.dump, $(NETWORKS)))
 install: build
 	scripts/dev_install.sh -p $(GOPATH1)/bin
 
-.PHONY: default fmt vet lint check_license sanity cover prof deps build test fulltest shorttest clean cleango deploy node_exporter install %gen gen NONGO_BIN
+.PHONY: default fmt vet lint check_license check_shell sanity cover prof deps build test fulltest shorttest clean cleango deploy node_exporter install %gen gen NONGO_BIN

README.md

@@ -189,6 +189,6 @@ A number of packages provide utilities for the various components:
 
 Please see the [COPYING_FAQ](COPYING_FAQ) for details about how to apply our license.
 
-Copyright (C) 2019, Algorand Inc
+Copyright (C) 2019-2020, Algorand Inc.
 
 [developer site url]: https://developer.algorand.org/

THANKS.md

@@ -14,3 +14,4 @@ In no particular order:
 
 ### Bug Reports
 - Nanyan
+- xixisese

agreement/abstractions.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/actions.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify
@@ -20,7 +20,6 @@ import (
 	"context"
 	"fmt"
 
-	"github.com/algorand/go-algorand/logging"
 	"github.com/algorand/go-algorand/logging/logspec"
 	"github.com/algorand/go-algorand/logging/telemetryspec"
 	"github.com/algorand/go-algorand/protocol"
@@ -312,7 +311,7 @@ func (a pseudonodeAction) do(ctx context.Context, s *Service) {
 		case errPseudonodeNoProposals:
 			// no participation keys, do nothing.
 		default:
-			logging.Base().Errorf("pseudonode.MakeProposals call failed %v", err)
+			s.log.Errorf("pseudonode.MakeProposals call failed %v", err)
 		}
 	case repropose:
 		logEvent := logspec.AgreementEvent{
@@ -336,7 +335,7 @@ func (a pseudonodeAction) do(ctx context.Context, s *Service) {
 			// do nothing
 		default:
 			// otherwise,
-			logging.Base().Errorf("pseudonode.MakeVotes call failed for reproposal(%v) %v", a.T, err)
+			s.log.Errorf("pseudonode.MakeVotes call failed for reproposal(%v) %v", a.T, err)
 		}
 	case attest:
 		logEvent := logspec.AgreementEvent{
@@ -360,7 +359,7 @@ func (a pseudonodeAction) do(ctx context.Context, s *Service) {
 			s.demux.prioritize(voteEvents)
 		default:
 			// otherwise,
-			logging.Base().Errorf("pseudonode.MakeVotes call failed(%v) %v", a.T, err)
+			s.log.Errorf("pseudonode.MakeVotes call failed(%v) %v", a.T, err)
 			fallthrough // just so that we would close the channel.
 		case errPseudonodeNoVotes:
 			// do nothing; we're closing the channel just to avoid leaving open channels, but it's not

agreement/actor.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify
@@ -18,8 +18,6 @@ package agreement
 
 import (
 	"fmt"
-
-	"github.com/algorand/go-algorand/logging"
 )
 
 // An actor is a state machine which accepts events and returns sequences of actions.
@@ -89,12 +87,12 @@ func (l checkedActor) handle(r routerHandle, in event) []action {
 
 	for _, pre := range cerrpre {
 		if pre != nil {
-			logging.Base().Warnf("precondition call violation: %v", pre)
+			r.t.log.Warnf("precondition call violation: %v", pre)
 		}
 	}
 	for _, post := range cerrpost {
 		if post != nil {
-			logging.Base().Warnf("postcondition call violation: %v", post)
+			r.t.log.Warnf("postcondition call violation: %v", post)
 		}
 	}
 	//   for _, pre := range terrpre {

agreement/agreeInstall.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/agreementtest/keyManager.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/agreementtest/simulate.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify
@@ -189,7 +189,7 @@ func Simulate(dbname string, n basics.Round, roundDeadline time.Duration, ledger
 		Logger:         log,
 		Accessor:       accessor,
 		Clock:          stopwatch,
-		Network:        gossip.WrapNetwork(new(blackhole)),
+		Network:        gossip.WrapNetwork(new(blackhole), log),
 		Ledger:         ledger,
 		BlockFactory:   proposalFactory,
 		BlockValidator: proposalValidator,

agreement/agreementtest/simulate_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/asyncVoteVerifier.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/autopsy.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/bundle.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/bundle_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/cadaver.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/certificate.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/certificate_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/common_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/coservice.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify
@@ -44,8 +44,8 @@ type coserviceMonitor struct {
 }
 
 type coserviceListener interface {
-	inc(sum uint)
-	dec(sum uint)
+	inc(sum uint, state map[coserviceType]uint)
+	dec(sum uint, state map[coserviceType]uint)
 }
 
 func (m *coserviceMonitor) inc(t coserviceType) {
@@ -62,7 +62,7 @@ func (m *coserviceMonitor) inc(t coserviceType) {
 	m.c[t]++
 
 	if m.coserviceListener != nil {
-		m.coserviceListener.inc(m.sum())
+		m.coserviceListener.inc(m.sum(), m.c)
 	}
 }
 
@@ -83,7 +83,7 @@ func (m *coserviceMonitor) dec(t coserviceType) {
 	m.c[t]--
 
 	if m.coserviceListener != nil {
-		m.coserviceListener.dec(m.sum())
+		m.coserviceListener.dec(m.sum(), m.c)
 	}
 }
 

agreement/cryptoRequestContext.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/cryptoRequestContext_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/cryptoVerifier.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/cryptoVerifier_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/demux.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/demux_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/doc.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/encoding_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/errors.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/events.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/fuzzer/bandwidthFilter_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/fuzzer/catchupFilter_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/fuzzer/clockedFilter_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/fuzzer/dropMessageFilter_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/fuzzer/duplicateMessageFilter_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/fuzzer/filter_test.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify

agreement/fuzzer/fuzzer.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2019 Algorand, Inc.
+// Copyright (C) 2019-2020 Algorand, Inc.
 // This file is part of go-algorand
 //
 // go-algorand is free software: you can redistribute it and/or modify