Backups

Vicidial Backups

*Backup is a normal process.*

It is recommended that all data and applications be backed up on all servers nightly. In fact, it is recommended that the backups be moved to another server (and duplicated at a remote location to guard against catastrophic events at the server location). But this is not related to Vicidial, it's a universal standard. So let's talk about how this is applied to Vicidial.

1) Data - The most obvious item that qualifies as data is the database. But also we have client audio recordings and the audio store/MOH recordings and configuration files for all applications. Then there is voicemail messages and the user VM audio files.

2) Applications - There are two "classes" of application to discuss. First is the executable applications (which include the OS and support programs) and services, then there are 'scripts' which are arguably data or applications depending on your point of view. The executable applications are NOT backed up at all, but the Vicidial scripts are.

Some "non-vicidial" backups systems actually back up the entire OS and all applications and data therein by performing a full disk backup (or image). Vicidial does not. Their method of restore does not require this as Vicidial is an "OS independent application". Which means it is not necessary to image the drive to be able to restore.

Vicidial's backup script will back up everything necessary to duplicate the system based on the options selected when executing the script. (I have not verified that the VM user audio files (or messages) are captured by the Vicidial backup system, but I've also never had a client restore result in a client saying "Hey! I had to set up my VM again cuz my VM files were missing!" So they could be backed up or not. I've never checked.)

When I say everything necessary, I do not include Executable Applications (or the OS). This is because of how the restoration process works (and doesn't work).

For all except the Client Audio recordings each server will:

1. Create a tar.gz file of each category/location in a tmp folder based on the command line options during execution.
2. Combine all those files into a single tar.gz with the IP address and Day of the Week in the file name.
3. Push that file to /var/log/astguiclient/archive for storage (overwriting the prior file for that day/IP if there was one from last week).
4. Optionally push a copy of that file to an FTP service (if so instructed and if FTP is properly configured)

For Client Audio recordings, if the 4th audio script in crontab is active (FTP!) each server will (every 3 minutes, based on the cron schedule):

1. Create a new folder on the ftp server for today's date (this will fail if the folder already exists, and the server will ignore the failure)
2. Copy each file, one at a time, up to 1000 files per run, to the FTP folder it just attempted to create.
3. Modify the URL in the recording_log table for this recording to point to the new location of the file.
4. Move the file from its present folder (GSM or MP3 in /var/spool/asterisk/monitorDONE) to the FTP folder in the same directory.
5. Optionally push the file to a 2nd FTP server to ensure there are always two copies, since this server will likely be set to delete anything over 30 days. In essence a "non-mirror" simple copy of all recordings requiring only a 2nd ftp server be available. This server need not have a web service running or be publicly accessible and not database URL changes will happen. Just a copy.

So now we have all our data, scripts, recordings, etc. backed up (audio possibly to two locations). We can set the cronjob that prunes the audio folders to delete everything in all folders after 30 days which will keep the drive usage on the dialers manageable with small HDs. The application/database backup files are large and will continue to grow, but there are only 7 of them (one for each day of the week) and even if they grow to 3G each they will still only take 21G of drive space. Most servers will not have 3G backup files until they have made a LOT of calls. It takes a while to exceed 1G, much less three.

*Now on to restoration ...*

• Unlike Backup: Restoration is NOT a normal process.* Let's face it, if you are restoring a Vicidial server, something unusual is happening. Your server died. Your hardware is About to die. Or you have to move to a new physical server because of some business reason. There is no "restore application" (well, not that we've tested, but we'll talk about that later ...).

The primary concept is that you must first have a functional (and certified) Vicidial system upon which to overlay your restore. If the dialer does not work as an autodialer and demonstrate that all the functions you use work Before you restore ... you have no reason to believe that they will work after. This is because you are not restoring "programs", but only data, configuration files, and "scripts" that rely on a great many things being perfect to operate correctly. So before you begin any restore process, test the server thoroughly. And remember: Anything you create during the test process (users/campaigns/leads/passwords) WILL be wiped out during the restoration of the database.

To restore the server:

1. Install a fresh copy of Vicidial on the server. Preferably the exact same SVN revision level (handy to know this in advance, keep a note, it is in the system settings these days).
2. If necessary, copy the backup file to the server
3. Unpack the backup file
4. Unpack the item you want to restore (since each category is still zipped)
5. For DB restoration: drop the asterisk database, create the asterisk database, and "apply" the .sql file to that database. Upgrade the DB using the instructions in the UPGRADE document in /usr/src/astguiclient/trunk/ if you find that your SVN revision level does not match the database (or downgrade the Vicidial to match the DB).
6. For all other restorations, each file would be restored to it's original location. This is done by using the tar command and instructing it to restore the file starting at "/" (root). This means the files would be put back precisely where they were when they were backed up, but on the new server. If your version does not match the original server, and you've already upgraded the DB to match THIS server, re-invoke the "install.pl" at this point to re-assert the svn revision level of Vicidial. Always say "Yes" to install the sample configuration files (never modify any of the .conf files directly, as this process overwrites them and is necessary to maintain compatibility when upgrading).
7. Reboot!

• Next: Vicibox 5.X has a restore system!* However, we have not yet tested it and have NO idea how it works as no one has asked us to look into it yet. Remembering that Restoration is not a "normal" process, this likely won't happen until someone who is actually planning ahead and has a budget for such things pays us to. Or until we have nothing better to do one day (not this year, we're booked until Christmas!). I would expect it to rely on the presence of a "standard" Vicidial backup set available from an FTP service during installation, though. And it likely makes life much easier during restoration. Most call centers, however, merely want the systems back online immediately, they have NO care whatsoever how "easy" it was for the technician to restore the system as long as it is actually restored and working SOON.

It is worth saying, I suppose, that most clients just have us perform these tasks. :)