By Bruno Škvorc in News & Announcements — Apr 28, 2019

Nimbus Development Update - Whisper, 2019 goals, Season of Docs

Nimbus development update: upping our docs game, whisper incoming, byzantium incoming, testnets chugging along!

We started April off with a bang - Status' Chaos Unicorn day was a success, and Nimbus launched a public testnet which is still up and running in its ~fifth iteration now. Here's what else is new!

Networking, sync, and performance

While waiting for Ethereum 2.0 to finalize new spec releases (notably 0.6), we've been optimizing the current version of our beacon nodes. The long-running testnet has allowed us to pinpoint some memory leaks and state transition optimization opportunities, as well as abstract the investigations into new debugging tools.

We now have backward and forward sync in terms of catching up late joiners with the rest of the network. Backward sync will have a client request blocks it knows it's missing, while forward sync will ask a peer to send over a block and all its children. We're keeping both methods in the app for now, we'll see which works better as the chain grows in complexity and as we benchmark some scenarios.

We've also wrapped MiniUPnP which lets our clients find peers without having to manually forward ports or explicitly connect to a list of peers.

Windows Support v1

The downtime between releases also allowed us to revive our Windows support. The technical explanation is that it needed a bigger stack size.

You can now follow the official instructions for joining the testnet, now extended with Windows commands. All it takes is an extra step: make fetch-dlls and it should work, just make sure you run all the commands from a terminal that isn't native to Windows. Use Git Bash rather than CMD or PowerShell. Please note that you'll need MinGW64 installed to go through this, or if you'd rather use make instead of the ming32-make command, make sure you install make via Chocolatey.

Here's the full process:

git clone https://github.com/status-im/nimbus
cd nimbus
mingw32-make update
mingw32-make fetch-dlls # <--- this is new!
cd vendor/nim-beacon-chain
mingw32-make clean-testnet0 testnet0
./build/testnet0_node.exe

Node works

EF State Tests

We're ready to parse all EF state tests, but were blocked by some upstream changes which made hashes incompatible - all tests are signed by 0x0 whereas they should be signed by 0xc0000..., as per spec. Our codebase is unable to deserialize invalid signatures like these without serious hacks, so we've been waiting for fixes in the tests repo. This appears to have been done now and we'll be up and running on tests in no time.

The TL;DR is that with these tests, all the clients can start working towards cross-client testnets, and aim for a launch of Phase 0 in Q4 2019 (see end of this post for our 2019 goals!).

Whisper and RPC basics

Nimbus can now serve as a very basic Whisper client! 😱

This mode doesn't even need ethereum mode activated, so you can use it like you would Status' other Whisper implementations - Status X, Murmur, or Status-JS. Not only that, we're also slowly adding RPC support which will let users pull information from nodes as they're running.

Once you build Nimbus as per instructions at nimbus.status.im/docs, run it in RPC whisper mode:

./build/nimbus --rpc --protocols:shh --log-level:DEBUG

This will run Nimbus Whisper and output its enode address. At this point, we can add another node into the mix:

./build/nimbus --rpc --rpcbind:127.0.0.1.8546 --datadir:testdir --protocols:shh --port:30304 --staticnodes:[email protected]:30303 --log-level:DEBUG

The ports have been added here to make it possible for the two nodes to work on the same machine, otherwise their defaults would clash.

This connects the two nodes, but in order to actually get them to talk, we need a channel through which they can exchange and filter messages. If you have NodeJS installed, you can create the following files:

/* listen.js - to receive messages */
#!/usr/bin/nodejs

const Web3 = require('web3');
const web3 = new Web3(Web3.givenProvider || new Web3.providers.HttpProvider('HTTP://127.0.0.1:8545'), null, {});

web3.shh.getVersion()
.then(function(result) {console.log("Get Whisper version:", result)});

web3.shh.getInfo()
.then(function(result) {console.log("Get Whisper info:", result)});

web3.shh.setMaxMessageSize(1000000)
.then(console.log);

web3.shh.setMinPoW(0.2)
.then(console.log);

// choose a common channel name
let channel = 'nimbus-test-whisper'
// Topic = keccak256 of channel name, first 10 chars
let topic = web3.utils.sha3(channel).substring(0,10);

function readMessages(filterId) {
  web3.shh.getFilterMessages(filterId)
  .then(function(msgs) {
    // console.log(msgs)
    if(msgs.length > 0) {
      console.log(web3.utils.hexToUtf8(msgs[0].payload));
    }
    else {
    // console.log("Nothing arrived")
    }
  });
}

// Subscribe to topic and poll for messages
// SymKey generated from channel name
web3.shh.generateSymKeyFromPassword(channel, function (error, symKey) {
  web3.shh.newMessageFilter({
    symKeyID: symKey,
    topics: [topic]
  }).then(function(filter) {
    web3.shh.getFilterMessages(filter)
    .then(function(msgs) {
      if(msgs.length > 0) {
        console.log(web3.utils.hexToUtf8(msg[0].payload));
      }
      else {
        console.log("Nothing arrived so far.")
      }
    });
    // poll for messages every 2 seconds
    setInterval(readMessages.bind(null,filter), 2000);
  });
});

/* post.js - post a hello message */
#!/usr/bin/nodejs

const Web3 = require('web3');
const web3 = new Web3(Web3.givenProvider || new Web3.providers.HttpProvider('HTTP://127.0.0.1:8546'), null, {});

web3.shh.getVersion()
.then(function(result) {console.log("Get Whisper version:", result)});

web3.shh.getInfo()
.then(function(result) {console.log("Get Whisper info:", result)});

web3.shh.setMaxMessageSize(1000000)
.then(console.log);

web3.shh.setMinPoW(0.2)
.then(console.log);

// choose a common channel name
let channel = 'nimbus-test-whisper'
// Topic = keccak256 of channel name, first 10 chars
let topic = web3.utils.sha3(channel).substring(0,10);

// Message post to topic, with SymKey from channel name
// No signature added
web3.shh.generateSymKeyFromPassword(channel, function (error, symKey) {
  web3.shh.post({
    symKeyID: symKey,
    ttl: 30,
    topic: topic,
    powTarget: 0.2,
    powTime: 1.5,
    payload: web3.utils.asciiToHex("Hello nimbus!")
  });
});

/* selfpost.js - send a message to yourself */
#!/usr/bin/nodejs

const Web3 = require('web3');
const web3 = new Web3(Web3.givenProvider || new Web3.providers.HttpProvider('HTTP://127.0.0.1:8545'), null, {});

web3.shh.getVersion()
.then(function(result) {console.log("Get Whisper version:", result)});

web3.shh.getInfo()
.then(function(result) {console.log("Get Whisper info:", result)});

web3.shh.setMaxMessageSize(1000000)
.then(console.log);

web3.shh.setMinPoW(0.2)
.then(console.log);

let topic = '0x12345678'
// Subscribe and post to topic with symmetric key
web3.shh.addSymKey('0x0000000000000000000000000000000000000000000000000000000000000001', function (error, result) {
  web3.shh.newMessageFilter({
    symKeyID: result,
    topics: [topic]
  }).then(function(filter) {
    web3.shh.post({
      symKeyID: result,
      ttl: 30,
      topic: topic,
      powTarget: 0.2,
      powTime: 1.5,
      payload: web3.utils.asciiToHex("Hello nimbus!")
    }).then(
      web3.shh.getFilterMessages(filter)
      .then(function(msgs) {
        console.log(web3.utils.hexToUtf8(msgs[0].payload));
      })
    )
  });
});

The code above subscribes you to your own messages, but is enough for a proof of concept. Stay tuned as we flesh this out some more - full docs and guides coming to the Nimbus website soon.

Welcome Byzantium 🚀

We are past block 4.370.000 on Ethereum 1, and counting. This means we're in Byzantium!

We have forked to Byzantium 🦄

☁🎉The Nimbus @Ethereum 1.0 sync progress is now at block 4370000 / 7629467, which is well past the half way point of our favorite blockchain's lifetime 🎉☁

████████████░░░░░░░░░ 57.27%
— Nimbus (@ethnimbus) April 24, 2019

The full archive sync state at this block is a whopping 1.2 TB, compared to Geth's 230 GB. First things first, though - getting to 100% sync, and then optimizing for size and performance.

Season of Docs

Nimbus has applied to Season of Docs.

Season of Docs

Season of Docs is a Google effort dedicated to helping aspiring technical writers get paid experience working on challenging and interesting projects alongside the makers themselves. As an organization active in the open source bounty space, SoD is another venue for Status to express their intention to actively participate in the community and share resources and knowledge with whoever is interested.

Because Nimbus is a long-term project dedicated to changing the world and bringing censorship-resistant communication to everyone, everywhere, it is classified as a long-running project and as such will also support applications for long-form collaboration. What this means is that you can tag along and work with us until February 20th, 2020 if you wish. The extended periods are also paid, of course.

Our goals are as follows:

revamp the Nimbus website to be more consistent with the message behind the software: freedom, decentralization, censorship resistance, and all this through resource-restricted devices.
create and populate an in-depth documentation directory covering all out in-house-developed Nim libraries and tools in one comprehensively documented place, along with use cases, guides, audits (where necessary) and an auto-updated API reference.
time permitting, optionally craft several tutorials on multiple in-depth usages of Nimbus-created tools and libraries in external projects, helping disseminate the team's work in other, possibly unrelated projects.

If Nimbus is accepted as a participating organization, we'll be actively mentoring technical writers who join us for the Summer challenge and quite possibly asking some of them to stick around. If you've always wanted to work on cutting edge technology but couldn't get your foot in the door, this is your chance - the door is ajar. Keep an eye out for more announcements, and follow us on Twitter to find out if we've been accepted.

The Next 6 Months

Until DeVcon, we intend to accomplish the following goals:

a Nim version of libp2p will exist, and will be able to communicate with other implementations
given the above, we will have a phase 0 cross-client testnet complete and production-ready
additionally, due to the relative simplicity of Phase 1, we will have a Nimbus testnet of Phase 1
we will have a dedicated and polished implementation of Whisper, allowing for Ethereum-less use of Nimbus for Whisper communication
we will have a proof of concept light client implementation for Ethereum 2.0

We sincerely hope to be able to show you all of this at DeVcon in ~~Tokyo~~ Seoul.