Discussions from the Solana Validator Community on operations, upcoming changes, and best practices.

Community Validator Discussions

solana community

The Solana validator community discusses a tutorial from /dev/null | Pumpkin's Pool#8888 on validator identity transitions to a hot backup: https://pumpkins-pool.gitbook.io/pumpkins-pool/.  Next, Solana labs engineers discuss a validator panic on 1.14.x clients that occurred on February 1st.

Chat logs:

00:00:01.217,00:00:04.217
Dmitry Bykov: Hello

00:01:48.149,00:01:51.149
Zantetsu: Put the cat on screen

00:01:51.593,00:01:54.593
Zantetsu: Where is the cat

00:04:52.753,00:04:55.753
Michael Hubbard: yes it is

00:05:14.952,00:05:17.952
Michael Hubbard: there was never any proof that it caused issues other than the correlation of loss of vote credits with the use of this script

00:06:37.239,00:06:40.239
Michael Hubbard: I don't use this method

00:07:49.174,00:07:52.174
Michael Hubbard: mvines has clarified that gossip updates pretty quickly, I always suspected that it was a gossip issue.

GH issue for reference: https://github.com/solana-labs/solana/issues/24527

00:10:19.205,00:10:22.205
Michael Hubbard: the transition itself works great, very smooth, zero downtime, I'm just spooked by my issues in the past, but I hope they don't reproduce, so if enough others use this successfully I might try it again too :)

00:14:11.090,00:14:14.090
Zantetsu: Michael did you notice anything like what Blake mentioned where if the validator is not "healthy" then bad stuff happens when you try to switch identities?

00:14:21.524,00:14:24.524
Zantetsu: Where I suppose "bad stuff" is that the validator ignores the set identity maybe?

00:15:31.529,00:15:34.529
Michael Hubbard: nope

00:15:53.728,00:15:56.728
Michael Hubbard: as long as you somehow kill/stop the primary and run `solana-validator set-identity` on the secondary it has always worked for me

00:17:19.154,00:17:22.154
Blake Bartenbach: Yes, Zan. I think that is exactly the problem I have had. If the validator ignores the set-identity command, I think it needs to be killed.

00:17:49.483,00:17:52.483
Michael Hubbard: if the validator has crashed and is restarting then admin.rpc might not accept the set-identity command, perhaps that is the issue Blake?

00:17:59.181,00:18:02.181
Michael Vines: ya, thanks Blake!

00:18:03.219,00:18:06.219
Sol Blaze: Thanks for creating these docs, Blake!

00:18:38.639,00:18:41.639
Anish Nath: btw my handle is chatGPT in Solana discord and coming from rpcpool :) Brian/Linus Team

00:21:09.495,00:21:12.495
Tim Garcia: https://github.com/solana-labs/solana/issues/30060

00:21:11.589,00:21:14.589
Sol Blaze: Too much bad press surrounding early upgrading to v1.14, at this point we should just call it v1.15 haha

00:21:49.611,00:21:52.611
Tim Garcia: Bug description here: https://github.com/solana-labs/solana/issues/30060

00:22:12.931,00:22:15.931
Sol Blaze: I think it definitely shows that the incremental rollout strategy for Solana updates actually works in stopping these bugs

00:25:30.813,00:25:33.813
Tim Garcia: In discord #releng

00:30:37.453,00:30:40.453
Sol Blaze: There's also some stuff the Solana Staking Alliance is working on in terms of load testing on testnet/mainnet (nothing confirmed yet) so we could repurpose some stuff for testnet for continuous load in order to catch more bugs before new versions start getting incrementally rolled out on mainnet

00:31:16.768,00:31:19.768
Tim Garcia: kirill_l#4694

00:33:08.142,00:33:11.142
Sol Blaze: I am all for larger delegations from Delegation Program to those who do early upgrades. I'm planning to add a sort of "stake boost" on BlazeStake for extra stake to those running newer versions going forward, although that probably won't make as much of an impact, would love if other automated delegation systems joined in as well!

00:33:39.845,00:33:42.845
GNTHR: Will this be put on the youtube channel?

00:33:43.235,00:33:46.235
Sol Blaze: Yes

00:33:54.016,00:33:57.016
Sol Blaze: https://www.youtube.com/playlist?list=PLilwLeBwGuK78yjGBZwYhTf7rao0t13Zw

00:33:57.735,00:34:00.735
GNTHR: Good, thank you

00:38:17.639,00:38:20.639
Zantetsu: Yes thanks Solana devs, much appreciated!

00:38:29.265,00:38:32.265
GNTHR: Thank you

Solana Community Validator Discussion - February 2 2023