Merge lp:~a.skwar/mylvmbackup/zfs into lp:mylvmbackup

Just some personal comments on Skwar's rev86 branch proposal:

* The idea here, as I understand it, is to be able to create a snapshot of all relevant data necessary to facilitate a "full backup". If you only backup the "DataDir", then you miss out on binlogs, error logs, relaylogs, innodb logs, etc. For us, its simply best to backup the entire "root" mysql dir, /var/lib/mysql/ Inside that we have better organized directories for each aspect of mysql.

* Changing the snapshot size to a percentage won't work for us. Our customers (over 200) all have varying sizes of HD partitions. Some are 2TB, some are 100GB. In all cases, we only allocate 10GB for snapshot space; there isn't any need for more space. Changing to a percentage would, on most servers, cause the script to fail as being unable to allocate space for the snapshot. Perhaps this setting could be changed so that if the last character in the string is '%' it does a percentage or 'G' and it does a fixed size?

* lines 848-859 of r86 confuse me. I've never used ZFS before so cannot speak on its behalf. But I have and do use LVM. You cannot create a snapshot of a specific directory under LVM. You can only snapshot entire logical volumes.The lines mentioned seem to attempt to create multiple snaps of individual directories which is not possible with LVM.

* same goes for lines 981-993. With LVM, there would only be 1 snapshot volume.

Hi.

Let me answer your questions by paragraph.

a) Sure, a full backup should be done. But as of right now, there's no way of finding out where the binlog and such are stored. For us, there is no "root" mysql dir. Here, the databases are at /data/mysql/$dbname and datadir=/data/mysql. You cannot do a backup of /data/mysql/.., as that would include non-mysql stuff.

b) It needs to be a % value, because how would you specify the size for all the different logical volumes holding mysql data that you might have? Just assume that you have more than 1 LV (eg. one for every database, or something like that). That's actually the most important change in the branch - it changes the logic so, that installations spanning multiple filesystems would work. Cf. https://blueprints.launchpad.net/mylvmbackup/+spec/mylvmbackup-multiple-fs

c) lines 848-859:

848 # Loop over all the directories and create snapshots
849 foreach my $dir (@dir_list) {
850 # Get the filesystem information (ie. name and type) for the current $dir.
851 my %fsinfo = fs_info($dir);
852 # Do we have a ZFS?
853 if ( $fsinfo{"type"} == "zfs" )
854 {
855 $retval += __create_zfs_snapshot($fsinfo{"name"});
856 } else {
857 $retval += __create_lvm_snapshot(lvm_info($dir));
858 }
859 }

In @dir_list, the list of all the directories holding mysql stuff is stored; eg. /data/mysql, /data/mysql/Wiki. In line 849, I begin to loop over that array.

In line 851, I call fs_info(). This returns the name of the filesystem where the directory is stored on and also the type of the filesystem. Ie. %fsinfo = fs_info("/data/mysql"); Then the following would be true:

  $fsinfo{"type"} => "zfs" (or maybe "ext3", "xfs" or whatever)
  $fsinfo{"name"} => "data/mysql" (or "/dev/mapper/data--bases-mysql--base" or "/dev/dbs/MySQL_Base")

In 855 or 857 (depending on ZFS: yes/no), I create a snapshot of the filesystem. In 857, I call lvm_info, passing along the $dir (eg. "/data/mysql") and lvm_info would return the "LVM information" (ie. name of volume group and name of logical volume). See lines 1222ff.

d) No, with LVM there might also be more than 1 snapshot, maybe. Even with LVM it's very much possible to have many logical volumes. The assumption that there's just one snapshot lv (or source lv) is very much disturbing and also not what might be there on the system of the user.

Hi.

Would like to add something to your 1st paragraph:

If directories are missed, then all that needs to be done is to enhance "sub mysql_list_directories()" in lines 679-761. The purpose of this sub is to return a list (an array) of all the directories holding mysql data. Right now, it only return the mysql datadir and all the database directories,

If that's not enough, then add code to that sub.

Regards,
Alexander

Reg. your 2nd paragraph:

You're right. Being able to specify the size of the snap lv as a fixed size might be useful.

I'll change the code so, that it uses the % calculation only if the last char of "lvsize" is a "%". Otherwise, if it is a number (0-9), then it'll use that as a value for meaning "number of logical extents" and if it matches [kKmMgGtTpPeE], then it'll be used to mean that size in [kKmMgGtTpPeE] (IOE: if it matches [kKmMgGtTpPeE], then --size will be used).

The lvsize can now be entered as described above. Have a look at mylvmbackup.pl.in (revision 88), sub __create_lvm_snapshot() in lines 778 - 847.

Where are you backing up InnoDB data? You keep referring to DataDir but that is only for MyISAM.

Here is how we layout our customers:

basedir = /usr
socket = /var/lib/mysql/mysql.sock
tmpdir = /var/lib/mysql/tmp
datadir = /var/lib/mysql/data
log-error = /var/lib/mysql/log/error.log
log-bin = /var/lib/mysql/binlog/binlog
relay-log = /var/lib/mysql/relaylog/relaylog
innodb-data-home-dir = /var/lib/mysql/innodb
innodb-log-group-home-dir = /var/lib/mysql/innodb

If we only back'd up the datadir, we'd be missing out on tons of data necessary to completely rebuild a customer's database from the ground up.

Hi,

> Where are you backing up InnoDB data? You keep referring to DataDir but that
> is only for MyISAM.

Not necessarily - in a default configuration, InnoDB stores its table spaces inside of
the datadir as well.

> Here is how we layout our customers:
>
> basedir = /usr
> socket = /var/lib/mysql/mysql.sock
> tmpdir = /var/lib/mysql/tmp
> datadir = /var/lib/mysql/data
> log-error = /var/lib/mysql/log/error.log
> log-bin = /var/lib/mysql/binlog/binlog
> relay-log = /var/lib/mysql/relaylog/relaylog
> innodb-data-home-dir = /var/lib/mysql/innodb
> innodb-log-group-home-dir = /var/lib/mysql/innodb
>
> If we only back'd up the datadir, we'd be missing out on tons of data
> necessary to completely rebuild a customer's database from the ground up.

That's true. I guess the documentation should elaborate on that. For Linux LVM,
you need to make sure that all data is located on the same file system in order to
achieve a consistent snapshot. If your InnoDB table spaces are located outside the
default data directory, you would have to modify mylvmbackup to take this into
account.