Nimbus Development Update - 2018.12.2

Time for another progress update from the Nimbus team.

TL;DR: We've promised a proof of concept in December, and we're pretty close. With a bit of effort, you'll probably be able to run a Nimbus simulation to get the beacon client to talk its own clones even in this early experimental stage. See below for details and instructions.

State Transition

The most progress in the past month has probably been made in getting our state transition functionality up to par with the very lively specification document.

The nim-beacon-chain repo is a treasure trove of information on this but what it boils down to is that our state transition is now basically ready to go. We're still waiting on fork choice for which we'll pick something naive for the proof of concept - like longest chain.

There's a very descriptive test utility function file in testutil.nim and the state transition code itself is also pretty well documented and further tested (read: experimented on) in test_state_transition.nim.

While we're on tests...

Eth2 tests

There's an ongoing effort to get a new repository of cross-client tests up and running for Ethereum 2.0. All the tests reside in status-im/eth2-testgen, but are still pending review / merge in the official Eth2.0 test repo. One of the problems with tests is that even they need standards, so this is still being discussed.

These tests are not Nimbus specific, and must work across all clients. The Nimbus implementation in nim-beacon-chain still doesn't use them, but will as soon as they're merged and the project itself matures a little past the proof of concept phase. For now, we use a simulation to test things.

Simulation

We have created a simulator script in the nim-beacon-chain repo which will generate a new genesis state and 99 validators and store them in files named validator-xx.json where XX is the number of the validator. The files will contain this type of data:

{
  "privKey": "00000000000000000000000000000000607b42bcae82becbe947fb13e18522b3f9bf50efab3209fa9b7a95b869906872",
  "randao": {
    "seed": "E7C91FC8140FE2B27161AC4DEBAAF7DF2210D4F2AB6F3FC90BFD1960EABC1B73"
  }
}

These will then be used to generate a startup.json file with the registration information of each validator, containing their RANDAO commitment, withdrawal credentials, proof of posession, and everything else the beacon chain needs. From this file a genesis JSON file will be generated in state_snapshot.json, building the validator registry from their previously generated registrations, setting the genesis time at the time the simulation is run, and generating shard committees:

{
    "shard": 40,
    "committee": [28,8],
    "total_validator_count": 99
}

This means validator 28 and validator 8 (out of a total of 99) are in charge of processing shard 40.

Some fake RANDAO shuffles as well as shard crosslinks are also generated but default to zero in Phase 0.

Every set of 9 validators will get attached to their own node, resulting in a set of 10 beacon nodes running on ports from 50000 to 50009. Each node will also have its own enode address generated which you can find in tests/simulation/data-x where x is the number of the node. 9 validators will be left orphaned (all % 10 == 0) just to see what happens when it's their turn to validate and they have nowhere to get the data (no attached beacon node).

To try playing with this yourself, please see the next section.

Running the PoC

To get it to run, just make sure you have a working Nim environment and then:

git clone [email protected]:status-im/nim-beacon-chain
cd nim-beacon-chain
nimble install
nimble test

Alternatively, get our Vagrant box up and running for a test drive - we would love your feedback on how it works for you (if at all). Submit issues in the repo please!

These tests should all pass, they do some nice magic we'll be explaining in separate posts very soon.

Additionally, you can run the aforementioned simulation which generates 100 validators, stores their data into files for testing, and runs a mocked network of Nimbus nodes communicating with each other.

bash tests/simulation/start.sh

This is still in progress but should work relatively smoothly within a few days.

What's next?

• getting the simulation to work reliably
• setting up a common build system that'll work for everyone identically across the board. Especially important for Nimbus since we're targeting an incredibly large footprint of architectures
• adapting nim-beacon-chain to the tests being prepared for eth2.0 and lead the effort in the writing of these tests in close collaboration with other teams (please join the discussions at https://gitter.im/eth2-0-tests/Lobby and https://github.com/ethereum/eth2.0-tests/issues - this affects all of us!)

Eth 1.0

We got our Ethereum 1.0 Nimbus client (status-im/nimbus) to sync up to block 49018 at which point it fails because of invalid block state. This means that our EVM implementation successfully executes the transactions in 49017 blocks which is no small feat. It also means we're constantly being challenged to invent new debugging and state inspection tools in Nim, something that's been quite interesting to play with. You should expect to be able to deploy smart contracts, execute their functions, and issue RPC calls in H1 2019.

We're also working on an alternative build process for Ethereum 1.0 Nimbus, but more on that in another post after we've battle tested it. In the meanwhile, run it as usual. Alternatively, we're experimenting with Vagrant for one-size-fits-all build environments and would love your feedback on how it works for you IRT complexity / resource intensiveness / practicality, etc. Submit issues in the repo please!

---

Please give these tools a spin and let us know which bugs you find. We'd be absolutely thrilled to get fresh eyes on all of this, so with that in mind we'll soon start pushing out code-explaining posts and tutorials that guide you through how Nimbus is doing things specifically, not just conceptually.