RFC: Plesk backup to Amazon S3 – it works!

First thanks to Uli for the first step towards some idea’s and Armijn giving good guidelines how to use some variables together with commands.
cause I am a newbie on bin/bash scripting.. but with lots of testing tonight I was able to backup several ‘clients’ content towards buckets and objects on Amazon S3. Probably this is a first good step towards the usage of a simple script for the backup of your server’s content to Amazon S3

Needed: Python, S3cmd, data and the correct settings

note: this script is not finished yet as I am manually executing it and the removal of data from Amazon S3 is not well tested (it works, but I need to check it during the time the stuff is running.

#!/bin/sh

###################################
### ###
### BACKUP FOR  DOMAIN  ###
### ###
###################################

DOMAIN=put-here-a-client-name

BASEDIR=/var/lib/psa/backupscript

cd /$BASEDIR

# $ENVIRONMENTAL VALUE: use basedir for the backups (note my master reseller is called ‘beheer’)
DATADIR=/var/lib/psa/dumps/resellers/beheer/clients

# $ENVIRONMENTAL VALUE: save logfile
LOGS=”/var/lib/psa/backupscript/log`date +_%d%m`.log”

# $ENVIRONMENTAL VALUE: value to xx days, so keep earlier days, can be used daily and modified after testing
REMOVE=$(python s3cmd ls s3://bucketname | grep `date –date=’14 days ago’ +%d%m%Y` | cut -f 4 -d ‘/’)

# first backup the new stuff
python s3cmd put -r –progress -v $DATADIR/$DOMAIN/domains/ s3://bucketname/”$DOMAIN”_`date +%d%m%Y`/ >> $LOGS

echo $REMOVE date will be deleted

# delete the object older than (xx) days defined
python s3cmd del s3://bucketname/$REMOVE/*

rm -rf $DATADIR/$DOMAIN/domains/*

in this script I first define the environmental variables to make sure you only need to enter the correct path and ‘client’ name
than the content on your server is put into the bucked on S3, Then old backup removed from S3 and the actual backup deleted from your server to safe space.

Note: probably I need to use a ‘sync’ option after the put to make sure all data is correctly put onto the server (kind of validation?) if all is uploaded correctly
there is not an if than else statement, the backup content on server is ALWAYS removed when stuff fails ..

for a first script a nice try .. I know the bucketname could also be a ‘variable’ but did not choose for that (yet).

update 7/01/10

added
BASEDIR=/var/lib/psa/backupscript
cd /$BASEDIR

Otherwise s3cmd cannot work ;-)

MediaGiant News: Save your settings before firmware upgrade

Using a Conceptronic Mediagiant and you want to save your settings before a firmware upgrade? Here is how you can do it

A forced firmware upgrade is deleting all settings, but with a script it is possible to save the information. (and possible it will allow you to do more things).

1. Copy the shooting.sh file to an USB Flash Drive (use link to save the file)
2. Insert USB Flash Drive into the MediaGiant
3. Goto the Copy page  and focus on the shooting.sh file
4. Push on your remote the GOTO button and than push the REPEAT button  (you see 2x the invalid icon) the configuration is being backupped.
5. The device will be rebooting, after the reboot than you can do the firmware upgrade.
6. When firmware upgraded is finished, please go into  the Copy page and focus on shooting.sh again
7. push on your remote control the GOTO button thanpush REPEAT button the configuration is restored, device will reboot

Please be noticed: the script is creating a ‘VenusSetup.dat’ file on your HDD, you have to REMOVE  after your settings are restored.Cause if exist a next time it will use that file for your settings. Even if you have made changes afterwards

Not Good: My own FTP script

My backup FTP script is NOT GOOD.. it exhausted my memory ?? on the webserver caushing the server to ‘crash’ / not react for 4 hours, even connecting my monitor to the server: I could type my username, but login: nope ..

So message to self: how the hell can I FTP 40GB of data via a shell script without exhausting my memory .. <sigh> …

rebooting (fingers always crossed) and after 5 minutes the server was up again

My Simple FTP script

#!/bin/bash
cd /home/backup/server
if [ -e data ]
then
cd /home/backup/server/data/
HOST=’ftp-server-address’
USER=’username’
PASSWD=’password’
ncftp -u $USER -p $PASSWD $HOST << EOT
binary
cd /Volume_1
mkdir `date “+%d-%m-%y”`
cd `date “+%d-%m-%y”`
mput *
bye
EOT
fi
cd /home/backup/server
rm -rf data

with some help of Matthijs en of course myself and a little ‘maggi’ I created a bin/bash shell script to backup data (backup of the server) to one of my other FTP server(s)

it checks if the data folder is available and if so than the a ftp session is started, logged on, a folder with the todays date is made and the data is put in that folder. After that the data folder including the content will be removed. Yes I know there is no validation if A = B but I’ve no idea how to accomplish that, but this works in my test environment. This script is saving me about 4 Hours of datatransfer where FTP is showing me a nice 10MB/s where Samba / NFS is showing me a not satisfied performance at all .. (I believe the total amount of data is about 40GB I have to backup everytime … ) ..

hints and tips welcome to have a A=B comparison before removal of the data folder ;-)

Proftpd adding features to setup

I got a special request of a user:

allow a user to upload files, see the files but do not allow them to download any of the stored files. (So you can share 1 account to many but they are not allowed to download those files with that same account).

The master user of that vhost is allowed to control the whole upload folder.

It took me a good dinner to find an appropriate proftpd.conf setting. Some tests performed and it’s working now .

Find here the script

New Feature @ this Playground

I have implemented a new kewl feature today. I wanted to have ‘rotating’ images on every day of the hour for every day of the week. Through the #WordPress IRC channel (press only if you have IRC installed) I heard about the website of Dunstan Orchard. He implemented a rotating panorama header image with xml connection to the weatherforecast.

Okay I do not want to have it that advanced, but in a similar way it must be possible I thought. And Yes it is possible .. I am still experimenting, shooting images but slowly you will see every day of the week, every hour a different image displayed here @ the playground. It can be everything: the PlayGround, Something we did at a certain time a certain date. Let it be a unique PlayGround Project ..

What do I use? It’s not that difficult to find out. But it’s working.

Do you want to advertise: yeah it’s possible!