EventstoreDB replacing cluster nodes with new ones

KurrentDB
1

Hello,

We have EventstoreDb v24.10 installed in a cluster of three.
We need to replace the old VMs with new VMs into the existing cluster (newer Windows Server OS and VM SKUs) and want to join these new VMs to the cluster one by one, starting with replacing the Followers.

Currently there is

  • A Leader
  • 2 followers
  • 2 other nodes (new VMs that are showing as Initializing in Kurrent Navigator).

My original thinking was that once these new nodes had the data synced over, that if we stopped EventstoreDB via services.msc on one of the old Follower nodes and it displayed as Dead in Kurrent Navigator, that after a certain time (e.g. the DeadMemberRemovalPeriod window, I think is 30 minutes by default?), the old Follower would be removed and one of the new nodes would be promoted to a Follower but this doesn’t seem to be the case.

Our configuration is the following:

  • We currently have the DB, Index and Logs file on the one data disk (let’s say E:) on a virtual machines in Azure.
  • ClusterSize: 3
  • DiscoverViaDns: true

I cannot find the relevant documentation on the steps on how to replace nodes in a running cluster with new ones in a safe manner. If anyone has any documentation or has been through this before, any help is greatly appreciated.

Thank you

2

I’m thinking is it possible that the although I’ve shut down one of the followers and waited over 30 minutes, updated the A record to include the new Virtual machines’ IP, that I need to run ipconfig /flushdns and then restart the EventstoreDb service on the surviving Follower and Leader. I’m thinking this because it’s possible Eventstore still has the old IPs in cache and that’s why it’s not dropping the Dead node.

I also guess that restarting the surviving Follower and Leader one at a time will drop the quorum and will pause writes due to clusterSize being 3 so performing this outside of business hours is the ideal time.

3

I needed to not only stop the EventstoreDb service in Windows but also Shutdown/Stop the VM as well and a Initializing node became a Follower in the cluster. I did this one at a time.