Nimbus Status Update - Berlin, Priorities and AMA

We're incredibly grateful to the Ethereum Foundation for allowing us to continue our exploration of Eth2. The continuance of our grant and the added networking grant for further development of the libp2p stack will make sure that Ethereum 2 is as robust as a world computer running programmable money needs to be. Without furthed ado, let's dive into last fortnight's changes!

From August 12th to August 17th, the Nimbus team had their semi-regular dedicated brainstorming retreat to re-prioritize goals, estimate deliverables, discuss problems and cook up solutions.

Back in April when we launched the original testnet, we created our main TODO issue for tasks that still needed doing until Devcon 5. Many things changed since, so we took the opportunity to clean that list up a bit. Here are some of the decisions:

• We'll round up the Thundercloud Genesis launcher in such a way that it'll use Eth2 cryptography to generate keys compatible with Eth1. That way, rather than create Eth2 keys from Eth1 keys like Thundercloud did so far, and risk generating an invalid key for Eth2 (entirely possible), we'll go the other way around making sure both are valid. Admittedly, this means you cannot reuse your current private key to generate an Eth2 key (but to be honest, nor should you!), so no entering the deposit pool with with your exiting MetaMask. Likewise, the mnemonic bootstrap for the generator will have to be changed, given that we're not sure about the HD derivation path in Eth2 just yet. More info on this tool as things develop.
• On that note, we'll iron out nim-web3 to be easily runnable in various example modes. Its intended use right now is to track validator deposits for launching the Eth2 chain but we'll also include straightforward examples for things like auto-tracking certain events for a certain contract, following an address, interacting with a DAO, and more. Keep an eye on Nimbus libs for immediate access to such and similar guides!
• The minimum number of validators we intend to support and optimize for come Devcon is 100,000. On the other end of the performance spectrum, a beacon node + validator combination (1 each) should run on a Raspberry Pi 3+ at devcon (see below).
• We're moving ahead with plans to fix up our Nimbus Vagrant box in a way that'll make it compatible with both Lighthouse and Prysm. The aim is to build a common ground for testing client interoperability. We can already connect to Lighthouse, but we can't do much more than that, so we're debugging the communication right now. A common terrain for both clients to stand on helps there.
• We're working on the wire protocol to make sure we're interop-ready, and to be able to request missing blocks from peers more easily and validate them before propagation.
• Our metrics and JSON-RPC implementations will be done to the point of being able to support output to Etherscan, to a testnet health page, and to other clients for direct RPC communication.
• our build process is changing a little (more info below)

We wrapped the week up with a public video AMA which has since been uploaded to Youtube and can be seen here. Apologies for the potato webcam, next time we'll get a dedicated one! Some of the questions asked and answered:

• Why should someone use Nimbus vs. some other client?
• Why Nim vs. C, Rust, Go?
• Do the Eth2 teams communicate and share knowledge and if so, when and where?
• How far is Nimbus from production?

Build Process

You may know that Nimbus is our Ethereum 1 implementation in the main Nimbus repo, while the beacon chain aspect of the code is in the appropriately named nim-beacon-chain repo. Up until now, the way to build the Eth2 part of Nimbus was to check out the "main" Nimbus repo and then build the beacon nodes inside the vendors subfolder. We are now moving to an architecture where both Nimbus and Nim Beacon Chain will be top level repos, each with their own build system. Here's how you build Nimbus now (instructions also reflected on our website):

git clone https://github.com/status-im/nimbus
cd nimbus
make fetch-dlls # only on windows, on other platforms follow official instructions to install rocksdb and other requirements
make && make test

And here's how you build Nim Beacon Chain.

git clone https://github.com/status-im/nim-beacon-chain
cd nim-beacon-chain
make && make test

Each of these will build its own version of local Nim, which increases build time when looked at together, but makes things more modular when looked at separately. We feel like this is a more streamlined experience for those only interested in either Eth1 or Eth2, not both at the same time.

Libp2p on Windows

We have updated our Libp2p implementation to work on Windows now. You can once again run Nimbus on Windows machines. These changes are not yet in the master branch, so to run them compile the devel branch instead and remember to use testnet1, not testnet0:

git fetch
git checkout devel
git pull origin master
make update deps
cd vendor/nim-beacon-chain
make clean-testnet1 testnet1

But why stop there? Libp2p works just fine standalone, so grab the docs and fire up those examples, then go build your own Libp2p chat app!

Own Testnet demo

Our testnets are running and can be connected to, but we encourage and appreciate experimentation. Thus, we have published a refreshed guide on how to run your own private cross-platform (or cross-VM) testnet.

You can now run a private Ethereum 2 testnet with pre-generated fake validator keys. This will soon be expanded to include Eth1 reading for new deposits, allowing third parties to join your testnet if you open it up.

Low-end devices + WASM

Resource restricted devices have been our mantra since day 0. These days, our state transition simulation even works in the browser:

I know you've all been dying to do this: run the #eth2 beacon chain state transition from the comfort of your browser - well, the wait is over: https://t.co/4DsGBZYxNm pic.twitter.com/1UJJzo9y9B

— Jacek Sieka (@jcksie) August 6, 2019

You can check out the source of the above in this repo and its origin in this PR.

Now, we're doubling down on that resource restriction promise: we intend to get a beacon node with at least one validator running on a last gen Raspberry Pi. We'll be demonstrating the process of building and running this setup at Devcon 5 in a hands on workshop, so if you can, please join us - bring your Pies and go home with a running alpha Nimbus node!

DocGen update

Our Document Generator is a simple script for generating visually pleasing and mobile-friendly documentation for all the various independent and not necessarily blockchain-related libraries coming out of the Nimbus lab. It's built to be language agnostic, though, so you can use it to generate documentation for your own projects, too. Just follow the DocGen's README instructions.

The Nimbus Libs docs generated with this tool now also contain an auto-generated API reference, which lays the foundation for more documented code and more thorough guides - specifically, a separation of tutorials and recipes is coming alongside the current setup of the master overview with an API reference. There are some pending enhancements to the generator that prevent a more widespread deployment right now (most notably, generating docs for modules with multiple main files), so if you're into JavaScript hacking please shoot a PR or two our way once you look at the issues.

The Nimbus Libs docs site is now also being indexed by Algolia and we should have their super-search built in and activated by the time you read this.

Other Updates

• our SSZ and networking is now in sync with spec. These changes were necessary to pave the road to interop.
• due to some issues with YAML test files and Nim, we created a converter of official YAML tests to JSON, for use with tests in nim-beacon-chain.

Last but not least...

We've had two new members join this month - Andri and Dmitriy. Andri has been contributing via bounties for a long time now and we've finally brought him on board officially - he's responsible for almost all of the Eth1 progress! - while Dmitiry, previously at Mustekala and with significant IPFS experience, demonstrated his essence to the project by virtue of capacity for German beer. Welcome guys!