Activating Tenderbake - a story in data

Tezos Blockchain Consensus PBFT Tenderbake
Published on 2022/04/13
Activating Tenderbake - a story in data

This is a joint post with Nomadic Labs.

On April 1st, 2022, the Tezos blockchain successfully executed its most ambitious protocol upgrade to date. At block #2,244,609, the Ithaca2 protocol upgrade was activated on Tezos mainnet, moving the Tezos network away from the Emmy family of consensus algorithms to Tenderbake.

It is not the first time we have made changes to the consensus algorithm via the on-chain amendment process, but this time we have collectively gone where nobody in this space has gone before: hot-swapping to a completely different consensus algorithm on a live network without a hard-fork in just 1815 seconds.

Yes, it took only ~30 minutes for a majority of the network participants to surf the migration waters, get the first block endorsed by 2/3 of the network, and deliver Odysseus home to Ithaca.

We have been monitoring the network closely since then, and we are glad to see that the transition from Emmy* to Tenderbake was generally frictionless, and that the new consensus algorithm is running without interruptions since the first block of the cycle.

In this article, we present some early observations and reflections on the behavior of the network during cycle 468 - that is, the first 8192 blocks of the Ithaca2 protocol, between Friday evening and Monday evening (Paris time).

The data for this article was retrieved from the TzKT API.

Say hello to fast deterministic finality!

The following table discriminates the first 8192 blocks of cycle 468, from the activation of Ithaca2 on block #2,244,609 until block #2,244,609 on CEST Monday, April 4th.

Cycle 468 overall statistics
Table 1: Cycle 468 statistics by block round

Consensus on the head of the chain was both safe and fast: the average time between blocks was 32.058s, close to the theoretical minimum of 30 seconds. This is a direct consequence of having an overwhelming majority of blocks proposed and agreed upon in round 0: 7936 blocks out of 8192 – around 97% of them.

The round number for a block denotes the number of attempts (minus one) that were necessary to reach a consensus on a block at a given level. Lower is better, and a high number of rounds indicates a slow or unreliable network. Under normal network conditions, the chain should reach a consensus about the next block in round 0 – that is, on the first attempt.

Of the remaining 256 blocks, 236 were produced on round 1, 11 on round 2, and 5 blocks in rounds between 3 and 5. The only two outliers were understandably the first two blocks of the protocol migration, which passed respectively on rounds 14 and 12. The latter were, also understandably, the slower blocks of the cycle, with a time between blocks of 1815s **and **1590s.

These facts are better illustrated observing the values for these attributes throughout the whole cycle. Figure 2 plots, for each blockchain level, the payload and block rounds. Further below, Figure 3 plots the time between blocks without outliers above round 3[^nicely].

Otherwise, it does not plot nicely, and we cannot distinguish between blocks on rounds 0, 1, and 2.

Cycle 468 block/payload rounds per level
Figure 2: Block and payload proposal rounds per level in cycle 468

With Tenderbake, we distinguish between the payload producer and the block proposer. The payload is the non-consensus content of a block (transfers, smart contract calls, etc.), and it can be locked even if there was no consensus reached at the round. A baker with rights to propose a block on a higher round must then re-use the same payload – in jargon, it can re-propose the payload. Thus the payload round is the round at which the block’s payload has been first proposed (at the current level), and it is smaller or equal to the block round.

In the majority of cases, the payload and block rounds match, even for blocks proposed at rounds higher than 0. Even when Tenderbake splits baking rewards (awarding 10 ꜩ to the payload producer and a variable bonus to the block producer), in practice, most blocks in the cycle have the whole of the baking rewards allocated to the same baker. This also illustrates that even for higher rounds, the payload was first proposed at the same round as that of the block that was finalized, entailing that the failure to achieve a quorum on the first round was more likely to be caused by absent bakers in the committee, rather than a slow or untimely propagation of preendorsements.

Cycle 468 overall statistics
Figure 3: Time between blocks per level in cycle 468

As we mentioned above, the average time between blocks was 32.058s, pushed downwards by the majority of round 0 blocks. For round 1 blocks, it was 62.521s, or a bit over a minute. In the second graph, we also observe a significant minority of blocks with a 45s time between blocks. This corresponds to round 0 blocks which follow a round 1 block, as the time between blocks measures the difference between two consecutive blocks’ timestamps, and the timestamps are set at the beginning of the round. Thus, the difference in such cases matches the length of the duration of round 1 – exactly 45s. These corner cases also explain why the average values for the time between blocks at a given round presented in the table above, e.g., for round 0, diverge slightly from the theoretical minima. It moreover explains the maximal outliers, e.g., the 90s time between blocks for #2,250,528. The latter is a round 0 block which follows a round 4 predecessor, #2,250,527.

A safe network means higher bonuses

Starting with Tenderbake, a part of the baking rewards of up to 10 ꜩis paid to the baker for having included in a block extra endorsements for its predecessor resulting in more than the minimal 4667 endorsements slots. That is, beyond the 2/3rds of the total validations required to keep the chain live. On average, the blocks in the first cycle had 6795.5 endorsed slots out of 7000 -- or a 97%. This not only resulted in the majority of fast round 0 blocks we described before, but was also reflected in the bonus rewards paid to the bakers: the average bonus included in each block was 9.125202 tez.

Figure 4 plots the distribution of bonus baking rewards for each level of cycle 468: the first few blocks included lower bonuses, as the network was still recovering from the protocol migration and some bakers were catching up. In fact, the bonus rewards averaged 4.227153 tez in the first 100 blocks, but then we see how it rose quickly to stabilize over 8 ꜩ by the 1000th block. Past the quarter-cycle mark, it settled over 9 ꜩ. In fact, the average bonus rewards for the trailing three-quarters of the cycle was 9.379600 ꜩ.

Cycle 468 baker bonus
Figure 4: Bonus rewards awarded per level in cycle 468

Notice that around level #2250609, we can observe a noticeable drop in the bonus reward: this is consistent with a large drop in validation power due to a top public baker with a significant stake being out temporarily. This highlights the fact that given that Tenderbake favors safety over liveness, ensuring that the network is live -- and that everybody reaps the rewards -- relies more than ever on the collective effort.

The sweeping crew is still working

In the days following the activation, a few bakers using Ledger signers have been reporting issues with their baker software: the Ledger seems to either occasionally freeze and disconnect when signing (pre)endorsements, or return parsing errors when signing operations or blocks.

These can cause the baking daemon to freeze for up to ~10 minutes, resulting in endorsement misses and occasionally lost baking slots. This is happening seldomly, and we estimate it will not affect the participation of the affected bakers enough to make them miss the full endorsement rewards for the cycle. Sadly, this is not the case for the occasional blocks lost.

We have been keeping busy with these (and other minor issues), which should be fully addressed by an upcoming release of the Ledger Tezos baking app.

I am hooked on Tenderbake, give me more to read

Along the ride to deliver Tenderbake to Tezos, we have produced and shared reports on the development of Tenderbake and its features. If you want to read more about Tenderbake, we leave you a few good reads:

several blog entries;

a few academic articles;

and, of course, the main documentation entry-point:

Lessons and looking ahead

The activation of the Ithaca2 protocol marks the end of a two-years-plus-long journey undertaken by the research and development teams of Nomadic Labs, Functori, Tweag, and other Tezos core developers and academic partners, like CEA List and Université Paris-Saclay.

The superlative performance of Tenderbake so far is also a consequence of the preparedness of the community. A large majority of bakers were ready for Tenderbake, and had done their homework in time. We are also thankful to the many community members that helped out spreading the word, and were ready to help others in different spaces. After all, this is a decentralized, collective effort.

Tenderbake is alive and Odysseus is finally home on Ithaca, but it does not mean we are done. We have ambitious plans for the future of the Tezos blockchain, and the performance of the early Tenderbake cycles gives us confidence to keep pushing the boundaries.

Sails are going to be raised back up soon.

Fair winds!