Network performance is slow/stalls with three holochain conductors

[pythagorean]

This is with Holochain v0.0.12-alpha1 and n3h v0.0.11-alpha.

Local Zome updates are fast now, but when using multiple holochain conductors there is a significant delay when even two are used, with three, some updates never fully propagate at all.

I have three agents defined in separate conductor configs. The second and third conductors are instantiated with the bootstrap node of the first conductor after it loads.

This is being tested with Coolcats2, clone and make startnet to initialize two instances that can be accessed on localhost:8000 and localhost:8001 by default, edit the Makefile first to uncomment the third holochain instance at the bottom of the dna-startnet definition to enable localhost:8002 for performance testing. Run make stopnet and make reset between tests to clean up.

[zippy]

Hey there @pythagorean this should be fixed in recent releases. Can you confirm?

[pythagorean]

At the moment I cannot confirm because the Conductor is not working for me at all.

#1413

[willemolding]

@pythagorean does networking with 3 agents seem to be working now the conductor is working again?

[pythagorean]

Still not reliable with three agents, and the new Diorama test sporadically generates the Hachiko error in my Coolcats2 collision test. This is with Holochain 0.0.22-alpha1.

$ RUNTEST=collisions make dna-test
...
(cd dna-src; rustup run nightly-2019-01-24 hc test -s) | egrep -v '^[[:blank:]]*(info:|$)'
Running tests in test/index.js
> node test/index.js
  collisions
Starting instances...
Starting interfaces...
Starting UI servers
Starting all servers
Initializing new chain...
Initializing new chain...
Initializing new chain...
Bob creates a new handle the first time he uses coolcats
{ value: 'QmQ19PsiG92X1Jc2zjV6CTE68CNY1X1W4WUDGjBnCE5kze' }
    ✔ should be equal
Alice can retrieve a list of all handles
{
  value: [
    {
      address: 'QmQ19PsiG92X1Jc2zjV6CTE68CNY1X1W4WUDGjBnCE5kze',
      handle: 'bob'
    }
  ]
}
    ✔ should be equal
Bob can retrieve a list of all handles
{
  value: [
    {
      address: 'QmQ19PsiG92X1Jc2zjV6CTE68CNY1X1W4WUDGjBnCE5kze',
      handle: 'bob'
    }
  ]
}
    ✔ should be equal
Carol can retrieve a list of all handles
{
  value: [
    {
      address: 'QmQ19PsiG92X1Jc2zjV6CTE68CNY1X1W4WUDGjBnCE5kze',
      handle: 'bob'
    }
  ]
}
    ✔ should be equal
Carol creates a new handle 'Archer' the first time she uses coolcats
{ value: 'QmQz48TQHbpqnF4MEVxwTXmpzQs1kFuFkMDQKc3qMBPTYx' }
    ✔ should be equal
vvvv    hachiko warning    vvvv
a hachiko callback has been waiting for 16 seconds with no change
Still waiting on the following 2 signal(s):
[
  {
    event: '{"Hold":"QmQz48TQHbpqnF4MEVxwTXmpzQs1kFuFkMDQKc3qMBPTYx"}',
    sourceNode: 'carol',
    targetNode: 'alice'
  },
  {
    event: '{"Hold":"QmdLw33g3naBKJ7pRapRvEZ8451zb8jXHzBdv7LPBKM8dJ"}',
    sourceNode: 'carol',
    targetNode: 'alice'
  }
]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vvvv  hachiko timed out!  vvvv
Still waiting on the following 2 signal(s):
[
  {
    event: '{"Hold":"QmQz48TQHbpqnF4MEVxwTXmpzQs1kFuFkMDQKc3qMBPTYx"}',
    sourceNode: 'carol',
    targetNode: 'alice'
  },
  {
    event: '{"Hold":"QmdLw33g3naBKJ7pRapRvEZ8451zb8jXHzBdv7LPBKM8dJ"}',
    sourceNode: 'carol',
    targetNode: 'alice'
  }
]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ✖ (unnamed assert)
    -------------------
      operator: fail
      at: <anonymous> (/Users/mahakal/Development/Apps/coolcats2/dna-src/test/index.js:508:7)
      stack: |-
  Error: (unnamed assert)
  at Test.assert [as _assert] (/Users/mahakal/Development/Apps/coolcats2/dna-src/test/node_modules/tape/lib/test.js:225:54)
  at Test.bound [as _assert] (/Users/mahakal/Development/Apps/coolcats2/dna-src/test/node_modules/tape/lib/test.js:77:32)
  at Test.fail (/Users/mahakal/Development/Apps/coolcats2/dna-src/test/node_modules/tape/lib/test.js:318:10)
  at Test.bound [as fail] (/Users/mahakal/Development/Apps/coolcats2/dna-src/test/node_modules/tape/lib/test.js:77:32)
  at /Users/mahakal/Development/Apps/coolcats2/dna-src/test/index.js:508:7
  at processTicksAndRejections (internal/process/task_queues.js:82:5)
Error: command node test/index.js was not successful