[Mondo-devel] DRBD with MondoRescue

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Mondo-devel] DRBD with MondoRescue

Devin Ortner-2
Recently I was backing up a Redhat 6.7 machine with DRBD. MondoRescue didn't recover the data correctly, and left a gap in the filesystem. I apologize for not being able to find the original email I sent to the list, I just wanted to give my resolution in case it might help someone in the future, or maybe lead to DRBD support in the future. 

First my setup:
2 Identical hardware systems, node1 and node2 
Software mirror running LVM, and DRBD setup on a LVM volume on both nodes.
The DRBD resource is also LVM, all a single volume.
I use Pacemaker to run the DRBD device which contains a PostgreSQL database. (ReplDataClone)

Prior to mondoarchive (this could probably be scripted, but I didn't bother):
Make node2 the Pacemaker master (doesn't matter which as long as its consistent); "pcs standby node1"
Stop the PostgreSQL service in Pacemaker
On the master Pacemaker node; tar.gz everything on the DRBD device to root; "/drbd-data.tar.gz"
Stop DRBD service in Pacemaker
Stop the DRBD service on both nodes; "service drbd stop"
Verify that there are no /dev/drbd* devices.
Run "mondoarchive" and create media as normal.

After mondorestore (I wrote the following script to restore the archive, it will certainly need to be adjusted before it will work for most situations, it also assumes SSH keys are installed for both nodes):

#!/bin/bash
#Script to restore the DRBD data post-baseline.

cd /
echo "Restoring DRBD Data..."
echo "Creating Metadata for r0..."
/usr/sbin/drbdadm create-md r0
ssh mgt1 "drbdadm create-md r0"
echo "Starting DRBD on node2..."
sleep 2
#Background the drbd start command:
nohup service drbd start 2>&1 &
echo "Starting DRBD on node1..."
nohup ssh node1 "/etc/init.d/drbd start" 2>&1 &
sleep 2
echo "Starting Sync..."
sleep 2
/usr/sbin/drbdadm -- --overwrite-data-of-peer primary r0
echo "Adjusting r0's Replication Speed..."
/usr/sbin/drbdadm disk-options --c-plan-ahead=0 --resync-rate=110M r0
#
#Wait for the initial sync to finish, display progress:
echo "Waiting for initial sync to complete..."
sleep 1
LINE=$(grep "sync" /proc/drbd)
while [ "$LINE" != "" ]; do
LINE=$(grep "sync" /proc/drbd)
echo $LINE
sleep 5
done
#
#Create the LVM Volumes:
echo "Setting up LVM..."
pvcreate /dev/drbd0
sleep 1
vgcreate vg_replicated /dev/drbd0
sleep 1
vgchange -a ay vg_replicated
sleep 1
pcs cluster standby node1
pcs resource enable ReplDataClone
sleep 4
lvcreate -an -l 95%VG --name lv_pgsql vg_replicated
sleep 1
vgscan
lvscan
pcs resource disable ReplDataClone
echo "Waiting for DRBD to stop..."
sleep 5
echo "Starting DRBD..."
pcs resource enable ReplDataClone
sleep 5
#
# At this point we need to wait to see if pacemaker
# will activate the volume.
echo "Waiting for lv_pgsql to activate..."
sleep 4
LINE=$(lvs | grep lv_pgsql)
while [ "$LINE" == "" ]; do
LINE=$(lvs | grep lv_pgsql)
echo $LINE
sleep 2
done
#Restart ReplDataClone again.
echo "Restarting DRBD again"
pcs resource disable ReplDataClone
echo "Waiting for DRBD to stop..."
sleep 5
echo "Starting DRBD..."
pcs resource enable ReplDataClone
sleep 8
echo "Current LVM Status:"
lvscan
#Next we need to format the new volume
echo "Formatting the new volume..."
sleep 2
mkfs.ext4 /dev/vg_replicated/lv_pgsql
#
#Restart DRBD to apply it
echo "Stopping DRBD Resource..."
pcs resource disable ReplDataClone
sleep 6
echo "Starting DRBD Resource to setup volume..."
pcs resource enable ReplDataClone
sleep 8
#
#Only Restore if the LVM mounted:
if [ -d "/db/pgdata/lost+found" ]
then
echo "Extracting DRBD Data..."
tar -xzf /drbd-data.tar.gz
echo "Starting Postgres..."
pcs resource enable Postgres
echo "Waiting for services to start..."
sleep 8
#
#Re-activate node1 and watch the DRBD status:
echo "Re-joining node1 to the cluster..."
pcs cluster unstandby node1
echo "Allowing sync to finish..."
LINE=$(grep "sync" /proc/drbd)
while [ "$LINE" != "" ]; do
echo $LINE
sleep 5
done
echo "Reverting r0's replication speed..."
/usr/sbin/drbdadm adjust r0
echo "Finished! DRBD data was restored successfully."
else
echo "DRBD Restore FAILED!"
echo "LVM Directory did not mount!"
fi

This was a quick-and-dirty solution to fix my problem, feel free to manipulate and use it if you need to use MondoRescue in a DRBD environment.

Thanks,
Devin



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Mondo-devel] DRBD with MondoRescue

Bruno Cornec-4
Devin Ortner said on Wed, Jan 04, 2017 at 07:56:21AM -0500:
>This was a quick-and-dirty solution to fix my problem, feel free to
>manipulate and use it if you need to use MondoRescue in a DRBD environment.

Thanks Devin for sharing. With your authorization, I'll use your material to at least document on the MR web site, and maybe more if I can.

Best regards,
bruno.
--
Open Source Profession, WW Linux Community Lead     http://www.hpintelco.net
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
FLOSS projects:    http://mondorescue.org         http://project-builder.org
Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Mondo-devel] DRBD with MondoRescue

Devin Ortner-2
That is fine Bruno, go ahead and use whatever you need. I'm always happy to give back to the community. 

-Devin 

On Wed, Jan 4, 2017 at 3:59 PM, Bruno Cornec <[hidden email]> wrote:
Devin Ortner said on Wed, Jan 04, 2017 at 07:56:21AM -0500:
>This was a quick-and-dirty solution to fix my problem, feel free to
>manipulate and use it if you need to use MondoRescue in a DRBD environment.

Thanks Devin for sharing. With your authorization, I'll use your material to at least document on the MR web site, and maybe more if I can.

Best regards,
bruno.
--
Open Source Profession, WW Linux Community Lead     http://www.hpintelco.net
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
FLOSS projects:    http://mondorescue.org         http://project-builder.org
Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel