Home > Solaris > Solaris Backup Server using ZFS and rsync

Solaris Backup Server using ZFS and rsync

March 26th, 2009

I’ve released my backup script under the GPL. Basically it uses rsync to copy the configured paths to a central backup system and snapshots the filesystem once all of the rsyncs are completed. The really nice thing about it is that it can run multiple rsyncs in parallel and has a very simple configuration. We use this to back up all 3 of our datacenters at the same time, which makes our backup jobs complete much faster.

You can find more information on the ZFS Backup Server page.

You can use the comments section here to discuss, ask questions, or report problems.

Solaris , ,

  1. mark
    April 10th, 2009 at 09:01 | #1

    script looks good, I’ve been meaning to try out opensolaris and zfs. I think you’ve finally given me a good reason to do it.

    My only suggestion for the script is to add ‘–numeric-ids’ and ‘-H’ options to your rsync command line. I use those when doing full system migrations with rsync. otherwise, you can lose uid and gid mappings.

  2. branjo
    July 16th, 2009 at 09:41 | #2

    I am new in opensolaris, I like to run the ypass-backup.
    some parts are not clear to me.
    running Opensolaris 2009.06, add PHP (hopefully the right pakages)
    # pkginfo | grep php
    system SUNWapch22m-php52 PHP Server 5.2 module for Apache Web Server V2.2
    system SUNWapch22r-php52 PHP 5.2 module for Apache Web Server V2.2 (Root)
    system SUNWphp52d PHP Server Documentation
    system SUNWphp52r PHP Server 5.2 (Root)
    system SUNWphp52u PHP Server Version 5.2
    #

    I have set up the ZFS storage, mountpoints and others.
    config file is located under /usr/local/etc/backup.conf and customized.
    it seems the that the system does not realize that ypass-backup is a PHP script.

    first try was to modify the ypass-backup script and change the first line
    #!/usr/local/bin/php
    <?

    to #!/usr/php/bin/php
    but without any difference, second i copy the php binary to /usr/local/bin
    but nothing changed.

    any help are welcome
    branjo

  3. July 16th, 2009 at 14:05 | #3

    @branjo
    Try running it through the php interpreter directly:

    # php ./ypass-backup

    If that works, use ‘which’ to find out where php is:

    # which php

    …and change the first line. Without seeing the actual error that is output, it’s hard to say. Oh, and make sure the script is executable:

    # chmod 755 ypass-backup

  4. January 11th, 2010 at 00:23 | #4

    Firstly, great script, well thought out and simple to use. I do have one question, can it be made to rsync from an rsync server rather than using rsh/ssh? If so, can someone please post an example.

    Thanks

  5. Brandy
    February 8th, 2010 at 21:29 | #5

    Hello,

    I “was” very interested i the zfs backup system. I looked at the requirements. Looking at PHP there appears to be a long list of requirements for it also. Do you have a complete list of software required to implement this backup system? What is required on the clients also?

  6. February 10th, 2010 at 15:10 | #6

    @Brandy
    sunfreeware.com has a packaged version of PHP that you can try. The only module dependency in PHP is pcntl, which provides methods for forking.

    The only non-Solaris software required by the client is rsync.

  7. Brandy
    February 15th, 2010 at 18:18 | #7

    Eric,

    The Sunfreeware.com is where I found the php I also found it requires Apache 2.2.14 and libgcc-3.4.6 or gcc-3.4.6 what else?

    I’m working on customer owned machines and require approval to install FOSS products. Which can be long and painful. I need the full list of products that need to be installed for approval before I can start. I pointed the customer at this website and he was not very enthusiastic.

    Thank you, Brandy

  8. February 15th, 2010 at 19:07 | #8

    @Brandy

    I’m not sure what packages the sunfreeware.com package requires. We don’t use their packages since we compile our software from source. You can download PHP and compile it with GCC (or using the Sun C compiler if you don’t want GCC). You’ll need to do this if you don’t want the extra dependencies from the package. For example, PHP itself does not require Apache.

    You can install Solaris in a VM on your workstation if you need a test server.

  9. David
    February 17th, 2010 at 19:23 | #9

    Eric, thank you for the great script!

    In case it helps others, I had to do this to get it working nicely on OpenSolaris snv_132 w/SUNWphp52 installed:

    change first line to #!/usr/php/bin/php
    change version line to: define(‘VERSION’, ’1.0′);

    diff /etc/php/5.2/php.ini /etc/php/5.2/php.ini.orig
    132c132
    short_open_tag = Off
    350c350
    ;error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR
    354c354
    error_reporting = E_ALL

    Now, to my questions:

    I’m trying to scale this up quite a bit. One of the big advantages of ZFS is the ability to create, destroy, and otherwise operate on filesystem very cheaply. I’d like to have a separate filesystem for every host I back up.

    Before I spend too much time hacking on this, do you see any problems with this idea?

    I think I need to change your script’s mkdir to a zfs create poolname/filesystem and its snapshot to a snapshot -r, so the all the descendant ZFS filesystems will also get snapshotted (not just poolname).

    Does this sound reasonable?

    I’ve had lots of problems trying to get other ZFS snapshot and/or replication (zfs send|receive) scripts to work reliably, so I’m a bit wary of pouring a lot more time down a rabbit hole. :)

    Thanks, David

  10. February 18th, 2010 at 16:37 | #10

    @David
    I rarely use the recursive functionality in zfs. The recursive snapshot will definitely work, and as long as you’ve tested that zfs destroy works recursively for snapshots as well (the man page says it does), it shouldn’t be an issue.

    That’s not a bad idea anyway, since that’ll make it easier to completely remove a specific host’s backups. When you’re done hacking it up let me take a look at it and if it works well, I’ll update the script here.

  11. February 19th, 2010 at 17:03 | #11

    @David
    Actually, the more I thought about it this morning, the more I realized how awesome that idea is. A zfs filesystem per server also lets you have varying retention rates for each host, shows you exactly how much space each host is taking, as well as allowing you to nuke a specific hosts backups… all of which are really nice features.

    As such, I implemented your suggestion today with some minor changes. I make a ZFS filesystem for each host, and each host gets its own snapshot as soon as its backup is complete. I’ll need to test it for at least a week or so to make sure it works as expected, but if you want to test it along with me I can send you a new version.

  12. April 5th, 2010 at 17:43 | #12

    Eric, I found another feature you may want to add to your already extremely useful backup script.

    Currently, I added -x to your rsync options, so it won’t cross filesystem boundaries. This is because I have some servers with NFS mounts, and I don’t want to back them up multiple times.

    This has tripped me up on some critical filesystems that I DO need to back up, though.

    A feature which allows one to specify this by host, or even better, by filesystem would be great, if you ever have time to implement it.

    Thanks again for your great script!
    David

  13. Anonymous
    February 2nd, 2012 at 23:30 | #13

    @David Di Gioia

    David Di Gioia :
    Eric, I found another feature you may want to add to your already extremely useful backup script.
    Currently, I added -x to your rsync options, so it won’t cross filesystem boundaries. This is because I have some servers with NFS mounts, and I don’t want to back them up multiple times.
    This has tripped me up on some critical filesystems that I DO need to back up, though.
    A feature which allows one to specify this by host, or even better, by filesystem would be great, if you ever have time to implement it.
    Thanks again for your great script!
    David

  14. February 16th, 2012 at 17:24 | #14

    Hello,

    I would like to know if anyone has already tried this script on FreeBSD with ZFS as root.
    I am looking for a backup solution that would take advantage of ZFS… and your script seems to be a very good solution… 

    Which part might be specific to Solaris ?

    Thanks for your answer.

  1. No trackbacks yet.