Solaris Backup Server using ZFS and rsync
March 26th, 2009
I’ve released my backup script under the GPL. Basically it uses rsync to copy the configured paths to a central backup system and snapshots the filesystem once all of the rsyncs are completed. The really nice thing about it is that it can run multiple rsyncs in parallel and has a very simple configuration. We use this to back up all 3 of our datacenters at the same time, which makes our backup jobs complete much faster.
You can find more information on the ZFS Backup Server page.
You can use the comments section here to discuss, ask questions, or report problems.
script looks good, I’ve been meaning to try out opensolaris and zfs. I think you’ve finally given me a good reason to do it.
My only suggestion for the script is to add ‘–numeric-ids’ and ‘-H’ options to your rsync command line. I use those when doing full system migrations with rsync. otherwise, you can lose uid and gid mappings.
I am new in opensolaris, I like to run the ypass-backup.
some parts are not clear to me.
running Opensolaris 2009.06, add PHP (hopefully the right pakages)
# pkginfo | grep php
system SUNWapch22m-php52 PHP Server 5.2 module for Apache Web Server V2.2
system SUNWapch22r-php52 PHP 5.2 module for Apache Web Server V2.2 (Root)
system SUNWphp52d PHP Server Documentation
system SUNWphp52r PHP Server 5.2 (Root)
system SUNWphp52u PHP Server Version 5.2
#
I have set up the ZFS storage, mountpoints and others.
config file is located under /usr/local/etc/backup.conf and customized.
it seems the that the system does not realize that ypass-backup is a PHP script.
first try was to modify the ypass-backup script and change the first line
#!/usr/local/bin/php
<?
to #!/usr/php/bin/php
but without any difference, second i copy the php binary to /usr/local/bin
but nothing changed.
any help are welcome
branjo
@branjo
Try running it through the php interpreter directly:
# php ./ypass-backup
If that works, use ‘which’ to find out where php is:
# which php
…and change the first line. Without seeing the actual error that is output, it’s hard to say. Oh, and make sure the script is executable:
# chmod 755 ypass-backup
Firstly, great script, well thought out and simple to use. I do have one question, can it be made to rsync from an rsync server rather than using rsh/ssh? If so, can someone please post an example.
Thanks
Hello,
I “was” very interested i the zfs backup system. I looked at the requirements. Looking at PHP there appears to be a long list of requirements for it also. Do you have a complete list of software required to implement this backup system? What is required on the clients also?
@Brandy
sunfreeware.com has a packaged version of PHP that you can try. The only module dependency in PHP is pcntl, which provides methods for forking.
The only non-Solaris software required by the client is rsync.
Eric,
The Sunfreeware.com is where I found the php I also found it requires Apache 2.2.14 and libgcc-3.4.6 or gcc-3.4.6 what else?
I’m working on customer owned machines and require approval to install FOSS products. Which can be long and painful. I need the full list of products that need to be installed for approval before I can start. I pointed the customer at this website and he was not very enthusiastic.
Thank you, Brandy
@Brandy
I’m not sure what packages the sunfreeware.com package requires. We don’t use their packages since we compile our software from source. You can download PHP and compile it with GCC (or using the Sun C compiler if you don’t want GCC). You’ll need to do this if you don’t want the extra dependencies from the package. For example, PHP itself does not require Apache.
You can install Solaris in a VM on your workstation if you need a test server.
Eric, thank you for the great script!
In case it helps others, I had to do this to get it working nicely on OpenSolaris snv_132 w/SUNWphp52 installed:
change first line to #!/usr/php/bin/php
change version line to: define(‘VERSION’, ’1.0′);
diff /etc/php/5.2/php.ini /etc/php/5.2/php.ini.orig
132c132
short_open_tag = Off
350c350
;error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR
354c354
error_reporting = E_ALL
Now, to my questions:
I’m trying to scale this up quite a bit. One of the big advantages of ZFS is the ability to create, destroy, and otherwise operate on filesystem very cheaply. I’d like to have a separate filesystem for every host I back up.
Before I spend too much time hacking on this, do you see any problems with this idea?
I think I need to change your script’s mkdir to a zfs create poolname/filesystem and its snapshot to a snapshot -r, so the all the descendant ZFS filesystems will also get snapshotted (not just poolname).
Does this sound reasonable?
I’ve had lots of problems trying to get other ZFS snapshot and/or replication (zfs send|receive) scripts to work reliably, so I’m a bit wary of pouring a lot more time down a rabbit hole.
Thanks, David
@David
I rarely use the recursive functionality in zfs. The recursive snapshot will definitely work, and as long as you’ve tested that zfs destroy works recursively for snapshots as well (the man page says it does), it shouldn’t be an issue.
That’s not a bad idea anyway, since that’ll make it easier to completely remove a specific host’s backups. When you’re done hacking it up let me take a look at it and if it works well, I’ll update the script here.
@David
Actually, the more I thought about it this morning, the more I realized how awesome that idea is. A zfs filesystem per server also lets you have varying retention rates for each host, shows you exactly how much space each host is taking, as well as allowing you to nuke a specific hosts backups… all of which are really nice features.
As such, I implemented your suggestion today with some minor changes. I make a ZFS filesystem for each host, and each host gets its own snapshot as soon as its backup is complete. I’ll need to test it for at least a week or so to make sure it works as expected, but if you want to test it along with me I can send you a new version.
Eric, I found another feature you may want to add to your already extremely useful backup script.
Currently, I added -x to your rsync options, so it won’t cross filesystem boundaries. This is because I have some servers with NFS mounts, and I don’t want to back them up multiple times.
This has tripped me up on some critical filesystems that I DO need to back up, though.
A feature which allows one to specify this by host, or even better, by filesystem would be great, if you ever have time to implement it.
Thanks again for your great script!
David
@David Di Gioia
Hello,
I would like to know if anyone has already tried this script on FreeBSD with ZFS as root.
I am looking for a backup solution that would take advantage of ZFS… and your script seems to be a very good solution…
Which part might be specific to Solaris ?
Thanks for your answer.