{"id":49243,"date":"2018-07-10T06:00:02","date_gmt":"2018-07-10T04:00:02","guid":{"rendered":"http:\/\/blog.open-e.com\/?p=49243"},"modified":"2025-04-04T06:53:00","modified_gmt":"2025-04-04T06:53:00","slug":"cluster-split-brain-explained-part-2","status":"publish","type":"post","link":"https:\/\/www.open-e.com\/blog\/cluster-split-brain-explained-part-2\/","title":{"rendered":"Cluster Split-Brain explained [part 2]"},"content":{"rendered":"

\t\t\t\tHere it is, the second part of the cluster split-brain article series. In case you have missed the first part, you can read it here.<\/a>
\n<\/span><\/span><\/p>\n

The previous part finished with a Cluster Split-Brain <\/strong>event, so let’s start the second part from the moment when both nodes are issuing the same cluster resources.<\/span><\/p>\n

So, we currently have two servers (nodes) that are parts of a cluster that has been split. The n<\/span>odes do not “communicate” with each other but both nodes have taken over and shared the same cluster resources – ShareX and VIP1. <\/span>Clients have saved various data independently and we cannot connect them.<\/span><\/p>\n

And now!<\/strong> starts the mechanism that protects your whole environment against data loss after the cluster split-brain. <\/span>In case resources have been taken over and communication with a remote node is lost, each node recognizes itself as separated. <\/span>This kind of recognition is used when restoring the communication between nodes. Nevertheless, w<\/span>e will consider what would happen if the mechanism was not included.
\n<\/span><\/p>\n

Imagine the connection path between cluster path \/ mirror path returns. <\/span>Both nodes have Pool0 <\/span>imported,\u00a0<\/span>so both of them start synchronizing <\/span>their data for the other node <\/span>while overwriting the data that is already on it <\/span>AT THE SAME TIME!<\/span>\u00a0<\/span>It’s a recipe for disaster, as the data cannot be recovered. Even i<\/span>f we turn off node A before fixing the connection between nodes, we will repair the cluster path \/ mirror path and just start node A, it will join the cluster and put its disks under the control of node B, as it manages the current Pool0. <\/span>The data from node A will be overwritten by the data from node B. Which is also a disaster but a minor one because we “only” lost data from node A and both nodes now have data belonging to node B. An analogical situation happens if node B is turned off.<\/span><\/p>\n

So what will cause our separation mechanism?<\/span><\/p>\n

As we already know, thanks to this mechanism each node was marked on its side as separated. <\/span>After fixing the cluster path \/ mirror path line between the nodes, they first check the state of the second node. <\/span>If both nodes are in separated mode, none of them will share their disks with the other node in order to prevent overwriting, and thus data loss. <\/span>Even if the node had been switched off before the communication line was repaired, and then the line was repaired and switched on again, it still has a separated state on its side. W<\/span>hen connecting to a cluster, it will also detect the separated mode on the remote node and will not share its disks, which means the data is secure.<\/span><\/p>\n

Okay, but what can the client do?<\/span><\/p>\n

Well, a lot actually. <\/span>The user should check the data on both nodes on his own and, for example make a copy of data from node A (the data which are not on node B). Next, he or she should force sharing disks from node A to node B and vice versa – this is possible thanks to a special functionality in the Open-E JovianDSS webGUI. <\/span>After this operation, node B will overwrite the data on node A, and the client can set up the data which node B did not have from the previously created backup.<\/span><\/p>\n

But where is this mechanism that prevents split brain? <\/span>And why it does not work in this case? <\/span>Can it be more secure?<\/span><\/p>\n

The answers to all these questions will be revealed in part 3 of this series!
\n<\/span><\/h3>\n","protected":false},"excerpt":{"rendered":"

Here it is, the second part of the cluster split-brain article series. In case you have missed the first part, you can read it here. The previous part finished with …<\/p>\n","protected":false},"author":2,"featured_media":49283,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[14,29,18,27],"tags":[146,150,154,191,198,205,428,469,621],"class_list":["post-49243","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-backup-storage-technology","category-data-protection","category-hardware","category-open-e-joviandss","tag-cluster","tag-cluster-split-brain","tag-clustering","tag-data-protection","tag-data-security","tag-data-synchronization","tag-node","tag-open-e-joviandss","tag-split-brain"],"acf":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/posts\/49243","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/comments?post=49243"}],"version-history":[{"count":1,"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/posts\/49243\/revisions"}],"predecessor-version":[{"id":55100,"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/posts\/49243\/revisions\/55100"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/media\/49283"}],"wp:attachment":[{"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/media?parent=49243"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/categories?post=49243"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.open-e.com\/blog\/wp-json\/wp\/v2\/tags?post=49243"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}