Nimbus Development Update, Feb 2019
What has the Nimbus team done since the last update? What are they doing next?
What's new?
We echo SigmaPrime's Lighthouse in expectations of testnets for March.
TL;DR: we're building a testnet simulation which auto-generates validator entries instead of loading them from a deposit contract, and our aim is to have different Nimbus nodes on the same machine communicate with each other flawlessly. No interoperability is planned with other clients right now because we still have no cross-client test vectors among the implementers so trying to sync up would likely be a time sink at this time.
That said, apart from the updates mentioned in the text below, our team meet in Brussels at FOSDEM resulted in a clear outline for our MVP which you can see here - we'd appreciate help with any of these tasks if you're feeling up to it!
Simulation
Our network simulation has progressed. It was brought in sync with the latest Ethereum 2.0 spec changes - the 0.3 release. It currently fights itself due to some synchronization issues - validators can't keep up because they shuffle too slowly so they start attesting to different states, each to his own. This difference never consolidates and things spiral out of control in fun ways.
To better be able to keep an eye on all this, we've slapped a bit of multitail magic on top to make the following hacker-vision come true:
21:9 screens are pretty nice for running 10 #eth2 beacon nodes concurrently :) pic.twitter.com/NBn63qA0P4
— Jacek Sieka (@jcksie) February 20, 2019
You can now feel like the Matrix' Architect and watch your minions fumble about in the virtual world you've created for them.
To get this view:
git clone https://github.com/status-im/nimbus
cd nimbus
make USE_MULTITAIL="yes" eth2_network_simulation
More about make below.
Unfortunately, we're putting Windows support on the backburner and leaving that OS a little bit behind because of build system complexities. Once we've ironed out the sim on *nix, we're doubling down on Windows. In the meanwhile, there's the handy nim-vagrant box which runs the simulation just fine in a virtual machine on your Windows OS if you'd like to take it for a spin (give it at least 2 CPU cores, preferably 3).
The next steps in terms of the testnet now are getting a client to quickly sync up with the rest if it falls behind, making sure nodes can smoothly restart and reconnect, and adding in some less trivial fork choice - all in progress.
Block debugger
In order to speed up the block syncing efforts of Ethereum 1.0 in Nimbus, we've created an awesome debugger called Premix. Premix reads data from a geth node running in archive mode, and tries to match the state in every geth block with the state in the matching Nimbus block. When it fails, it generates a debug report with exact failure conditions in a nice HTML report. This allows for truly detailed inspections of problematic blocks:
You can find out more about this excellent tool in the documentation, but we need to re-state that you'll need to be running Geth in archive mode, which means 2 TB of sync data and likely a full week of wait time on an SSD.
This progress now lets our Ethereum 1 version of Nimbus reach reach beyond block 500k!
The block debugger is such tremendous help and such excellent work that it made its author the first recipient of the Nimbus Contributor badge.
A new build system experiment
We've moved to makefiles as a way to build our binaries. They seem to work fine on all platforms with minor hiccups on Windows due to compilation steps, but we're getting there. It's an experiment not unlike our nim-vagrant box, so we can see what works and what doesn't for when we eventually prioritize reproducible builds (a high-priority end-game goal of ours).
In order to run the simulation, for example, you now clone nimbus and use make, like so:
git clone https://github.com/status-im/nimbus
cd nimbus
make eth2_network_simulation
To build a Nimbus binary (for Ethereum 1.0), use:
make nimbus
That's it, the build system takes over. Pretty cool, right?
Elsewhere
In some less impactful news...
Nimbus works on NanoPC-T4
... obviously. If you're not familiar with the Nanos, it's related to running Ethereum nodes on ARM devices - micro computers like the NanoPC-T4. The origin story is in this Reddit thread. Nimbus working on this is hardly a surprise since the Nanos are actually quite beefy and we've managed to compile Nimbus before on a Raspberry Pi, a much weaker device, but the fact that Nimbus was able to compile on it is the kicker - you can build Nimbus from source on these little computers and be absolutely certain you're running what you cloned. Pretty neat!
Running smoothly. One Ethereum full node, open RPC, serving light clients. One running the Nimbus client and a national PoA testnet. Ethereum on ARM is 👌
— Bruno Skvorc (@bitfalls) February 23, 2019
260 euro per node total, maintenance: nil, plug and play. pic.twitter.com/YDmugQkfbW
Code janitoring
We've decided to move to a cleaner and leaner repo structure - we'll be moving to a semi-monorepo soon, consolidating a lot of the fragmented libraries we've built on this exciting journey.
This means less overhead on maintenance and licensing, more development, and a renewed effort around documenting the repositories - not only in code, but also on their own dedicated docs pages similar to Nimcrypto's (but better). More on that soon, but expect to be able to understand, pull in, and use any of our Nim libs soon for your own projects 🤘
Stratus
Head honcho Jacek spent one day (?!) in the waning hours of 2018 putting together an alternative Status implementation in Nim using some of the tools we've built for Nimbus. The project is called Stratus and its story is here.
Since then, the team collectively lobbied to turn it into more than a proof of concept, and a separate Stratus repo was born.
Stratus is the latest in a line of Status reimplementation efforts. The ultimate aim is to have an open ecosystem anyone can contribute to and build their own clients for, so projects like Stratus, Status-x, StatusJS API and similar are all a natural evolution of this effort.
What makes Stratus worth mentioning here is that it's using a lot of the same components Nimbus is using under the hood. This not only demonstrates their reusability in the web3 world, but also implies that the dream of replacing the resource-intensive Status-go backend in the Status app with a Nimbus version might not be so far fetched after all.
Note that building Stratus is really hard right now and the project is in no way a priority, so YMMV and contributions welcome!
FOSDEM
The Nimbus team also had representation at FOSDEM where Jacek talked about the process of building an Ethereum client.
That's it for this round - by the time the next update is due, our simulation will be chugging along and maybe, just maybe, talking to some other client's simulation as well.