Backup Exec Blues

I’ve stumbled across a problem with the Backup Exec remote agent. Under a certain set of circumstances, the agent will consume a huge amount of virutal memory and never release it unless it is restarted.

This affects Backup Exec 11d rev 7170.0 with no hotfixes or service packs and with service pack 2 with all current hotfixes (as of 21st April, 2008). So, I assume all minor versions between the base and the latest are affected.

The problem relates to a remote server – namely a server with a remote agent that is running on a Windows 2003 R2 Server 64bit with service pack 2. Also installed on this server is Exchange Server 2007 with service pack 1. The version of the agent is 7170.0 – the latest version at the time of writing this post.

There’s one more thing. The backup is being encrypted. When I disable the encryption the problem doesn’t reoccur. So it must have something to do with that.

In one case I saw the process beremote.exe holding 1.5GB of virtual memory, even though there wasn’t a backup running. Of course, this takes resources away from the system and services like the Exchange 2007 Information Store gets a wee bit upset about this.

In another example, with the same setup (Windows R2 sp2 with Ex2k7 with SP1), beremote was consuming 6.8GB of virtual memory. See image below.

So, what to do? Well here’s a technote from Symantec: http://seer.entsupport.symantec.com/docs/289695.htm

Here’s an extract from the article:

Details:
Each time a GRT backup of an Exchange server is performed, the memory used by the Backup Exec Remote Agent for Windows Servers ( RAWS ) process, beremote.exe, may increase. In this situation, the memory used by beremote.exe does not drop back down after backup but continues to rise with each mailbox backup performed.

Note, I have other systems set up in a similar way that are not affected by this problem. The only difference is that in these cases the backups are not being encrypted.

Symantec don’t offer a fix for this problem at the moment, just a workaround. The workaround is to restart the backup exec agent. This could be done via a scheduled task calling a batch file. Basically, the offered solution is a fudge, not a fix.

It is, however, very important to encrypt your backups, so turning encryption off is not the solution either – this just avoids the problem.

I don’t know if Backup Exec v12 is affected by this issue. I’ve not done enough testing in this version yet.

Symantec Tech Support can only offer the fudge and there’s no hint of a fix being released.

For now, the only way forward is to restart the agent when you’re sure the backup process is not in use. So here’s an idea, as an interim solution. Configure the backup job so that it runs a batch file that will restart the backup agent service. To do this, under Job Settings | Pre/Post commands, call a batch file that restarts the remote agent on the remote server. Tip, use the ‘SC.EXE’ command in the batch file. Details on the command are here. Here’s an example of the command in full:

sc \\REMOTE_SERV stop BackupExecAgentAccelerator & timeout /T 30 && sc \\REMOTE_SERV start BackupExecAgentAccelerator

(there’s no carriage return within the command, only at the end)

In the example above, change ‘REMOTE_SERV’ with the name of your remote server that is affected by this problem. Note, the ‘/T 30′ part of the command relates to the timeout required between sending the stop command and sending the start command. 30 seconds seems to work okay for me. You may want to increase this timeout if the service doesn’t start (because it hasn’t yet finished the stop command).

Hope this helps. Let me know if you’ve come across this problem.

BEBlue