Backup & Recovery manager for Avamar, Networker and Datadomain.

New EMC centralized monitoring tool for Avamar Networker and Datadomain only. 

how to find Avamar serial number for utility and storage nodes

For Avamar Datastore Generation 4 nodes

Avamar Datastore Generation 4 hardware consists of two different types of Dell nodes (R510 and R710). A new hardware information retreival system is used and has a different command that will work on all Gen4 nodes.

To retrieve the serial number on a node, login as root and issue the following command:

/usr/bin/ipmitool fru print 0 " grep "Product Asset Tag" " sed "s/^.*: *\(.*\)/\1/"

This command must be run as root user or else you will see an error such as:

"Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory"

The following mapall command is run from the utility node (run as admin with dpnid key loaded) will retrieve the serial numbers from all of the nodes in the grid.

mapall --noerror --user=root --nodes=all+ --quiet 'echo `hostname -i`": "`/usr/bin/ipmitool fru print 0 " grep "Product Asset Tag" " sed "s/^.*:\s\(.*$\)/\1/"`'

Sample output:

10.6.240.193: NNG05104310280_100-580-617_A04
10.6.240.194: NNG05104310279_100-580-617_A04
10.6.240.195: NNG05104310284_100-580-617_A04
10.6.240.196: NNG05104310287_100-580-617_A04

The node serials in the example above start with NNG up to the first underscore.

The 100-580-617_A04 part is the EMC part/revision number.

Avamar unix client side restore, add noticketrequired for the user created. Ldap

Admin@avamar> avmgr logn --id=rman_user@/rman --password=rman_user

1 Request succeeded

32817 privilege level (enabled,read,backup,mclogin)

2 block type (directory)

Admin@avamar>avmgr chgv --path=/rman  --u=rman_user --pv=’enabled,read,backup,mclogin,noticketrequired’ –ud=’avamar’

1 Request succeeded

36913 privilege level (enabled,read,backup,noticketrequired,mclogin)

2 block type (directory)

Man-In-The-Middle attack: “REMOTE HOST IDENTIFICATION HAS CHANGED!”

If you receive a message referencing a possible Man-In-The-Middle attack:
“REMOTE HOST IDENTIFICATION HAS CHANGED!”

Remove the IP address in question from "/home/admin/.ssh/known_hosts"

Redirected restore disabled for all non-administrators in Avamar 6.1 and above. You can enable by following this procedure.

Avamar 6.1, Redirect restore is disabled by default for non-administrator users and needs to be enabled from the avamar server.

1-      Login to avamar as root

2-      Run the commands in red without #

3-      # cd /usr/local/avamar/var/mc/server_data/prefs/

4-      # cp mcserver.xml mcserver.xml_orig

5-      Open the file mcserver.xml, you can use vi

6-      Search for the entries below

 entry key="admin_can_direct_restores" value="false" 

 entry key="restore_admin_can_direct_restores" value="false" 

7-      Change both entries to true

 entry key="admin_can_direct_restores" value="true" 

 entry key="restore_admin_can_direct_restores" value="true" 

8-      Save the file

9-      Need to restart the mcs (All backup and restore activities will be cleared)

10-   # su - admin

11-   # ssh-agent bash

12-   # ssh-add .ssh/dpnid

13-   # dpnctl stop mcs

14-   # dpnctl start mcs

15-   # dpnctl start

If checkpoints fails with DDR error

root@Avamar_server:~/#: cplist
cplist: ERROR: ddrmaint: <4750>Datadomain get checkpoint list operation failed.

cp.20120926170740 Wed Sep 26 12:07:40 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120926185155 Wed Sep 26 13:51:55 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120927170726 Thu Sep 27 12:07:26 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120927184726 Thu Sep 27 13:47:26 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120928170738 Fri Sep 28 12:07:38 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120928190034 Fri Sep 28 14:00:34 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120929170731 Sat Sep 29 12:07:31 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120929184944 Sat Sep 29 13:49:44 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120930170728 Sun Sep 30 12:07:28 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20120930184916 Sun Sep 30 13:49:16 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121001170739 Mon Oct 1 12:07:39 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121001184754 Mon Oct 1 13:47:54 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121002170739 Tue Oct 2 12:07:39 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121002190634 Tue Oct 2 14:06:34 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121003170734 Wed Oct 3 12:07:34 2012 invalid --- --- nodes 5/5 stripes 77852
cp.20121003185200 Wed Oct 3 13:52:00 2012 invalid --- --- nodes 5/5 stripes 77852
cp.20121004170736 Thu Oct 4 12:07:36 2012 invalid --- --- nodes 5/5 stripes 77852
cp.20121004190124 Thu Oct 4 14:01:24 2012 invalid --- --- nodes 5/5 stripes 77852
cp.20121005170734 Fri Oct 5 12:07:34 2012 invalid --- --- nodes 5/5 stripes 77854
cp.20121005190516 Fri Oct 5 14:05:16 2012 invalid --- --- nodes 5/5 stripes 77854
cp.20121006170757 Sat Oct 6 12:07:57 2012 invalid --- --- nodes 5/5 stripes 77860
cp.20121006185724 Sat Oct 6 13:57:24 2012 invalid --- --- nodes 5/5 stripes 77860
cp.20121007170750 Sun Oct 7 12:07:50 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121007185441 Sun Oct 7 13:54:41 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121008170750 Mon Oct 8 12:07:50 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121008185438 Mon Oct 8 13:54:38 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121009170757 Tue Oct 9 12:07:57 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121009190138 Tue Oct 9 14:01:38 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121010170754 Wed Oct 10 12:07:54 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121010190509 Wed Oct 10 14:05:09 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121011170740 Thu Oct 11 12:07:40 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121011185927 Thu Oct 11 13:59:27 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121012170652 Fri Oct 12 12:06:52 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121012185325 Fri Oct 12 13:53:25 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121013170656 Sat Oct 13 12:06:56 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121013185554 Sat Oct 13 13:55:54 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121014170505 Sun Oct 14 12:05:05 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121014192822 Sun Oct 14 14:28:22 2012 invalid --- --- nodes 5/5 stripes 77862

Solution:

The /usr/local/avamar/var/ddr_info file had been zero’d out.  need to recreate ddr_info file, and wait for the extra checkpoints to be removed after next cp and hfschecks.

to check DIMM errors on avamar grid.

login as admin

add ssh keys

admin@avamar_grid_name:~/>: mapall --all 'omreport chassis memory | egrep "^Connector|^Status"' 2>&1 | more Using /usr/local/avamar/var/probe.xml

To check Garbage collection status

userid@avamar_grid> avmaint gcstatus

To kill Garbage collection manually through command line

user@avamar_gridname> avmaint -ava gckill

if the avamar has Datadomain then kill this process after gckill ran and gc still running.


user@avamar_gridname> ps -ef|grep ddrmaint
and "kill -9 process_id"
-9 is to kill forcibly.

Avamar passwords

To find out Avamar passwords, goto


more /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml and grep for userid's like MCUser/root/dpn

Avamar 6.1 released at EMC World 2012 - Las Vegas - May 21, 2012

http://www.emc.com/about/news/press/2012/20120521-06-03.htm

replicating client level from command line

nohup replicate --flagfile=/usr/local/avamar/etc/repl_cron.cfg --timeout=0 --include=/clients/client_name --logfile=/tmp/replication3.log >> /tmp/replication3.log

To know the grids configuration about GC, CP, Readonly

admin@avamar_grid01:~/>: avmaint --ava config | grep disk

disknocreate="90"

disknocp="96"

disknogc="87"

disknoflush="94"

diskwarning="50"

diskreadonly="65"

disknormaldelta="2"

diskfull="30"

diskfulldelta="5"

balancelocaldisks="false"

admin@avamar_grid01:~/>

Avamar windows system state backup failure.

One of the work around for system state backups. First install the New 6.0.1-66 client - AvamarClientWindows-x86_x64-6.0.101-66._HF34148.msi or later versions on the windows client.


 
Symptom

System state backup failed with the following error message:
"System writer is not found"

In addition, when you run the “Vssadmin list writers” command to display VSS writers on the system, the System Writer is missing.


Cause
The system writer fails due to the fact that the trusted installer & system accounts are missing permissions to files in the directory %windir%\winsxs\filemaps\.

Resolution
To resolve this issue, please run the following commands from an administrative elevated Command Window:

CD c:\windows\system32
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant "BUILTIN\Users:(RX)"

Applies to
Windows Server 2003, Windows Server 2008

Avamar: Multiple stream replication

Note: Before applying these changes, should have Avamar admin knowledge atleast a year or more.

Modifying the "repl_cron" Script

1) As user root on the replication source utility node, copy the /usr/local/avamar/bin/repl_cron script to a different, preferably meaningful, name.

% cd /usr/local/avamar/bin

% cp -p repl_cron repl2_cron

2) As user admin on the replication source utility node, copy the /usr/local/avamar/etc/repl_cron.cfg file to a different name, preferably a name that is consistent with the name used for the repl_cron program.  NOTE: Do not modify the original repl_cron.cfg file in the /usr/local/avamar/bin directory.  Copy the in-use repl_cron.cfg in the /usr/local/avamar/etc directory.

% cd /usr/local/avamar/etc

% cp -p repl_cron.cfg repl2_cron.cfg

3) As user root, edit the /usr/local/avamar/bin/repl2_cron for modification:

The portion of code that is of interest on the original repl_cron looks like:

--BEGIN--  

sub init {

    dpn::add_legal("timeout:d", "configfile:s");

    dpncron::init("repl_cron", "replicate", 4600, "--infomsgs --verbose");

    $configfile = "$dpn::avamardir/etc/repl_cron.cfg";

    $configfile = $dpn::flags{configfile} if $dpn::flags{configfile};

    dpncron::tprint("repl_cron [$version]: configfile = '$configfile'\n") if $dpn::flags{verbose};

}

--END--

Using the repl2_cron examples as provided earlier modify the code to look like:

--BEGIN--  

sub init {

    dpn::add_legal("timeout:d", "configfile:s");

    dpncron::init("repl2_cron", "replicate2", 4600, "--infomsgs --verbose");

    $configfile = "$dpn::avamardir/etc/repl2_cron.cfg";

    $configfile = $dpn::flags{configfile} if $dpn::flags{configfile};

    dpncron::tprint("repl2_cron [$version]: configfile = '$configfile'\n") if $dpn::flags{verbose};

}

--END--

Notes about the modifications:

-          The new variable "repl2_cron" is the name of the program that is to be called.  In this case the program being called is /usr/local/avamar/bin/repl2_cron

-          The new variable "replicate2" is base name of the replicate log.  In this case the replicate log for repl2_cron will be in /usr/local/avamar/var/cron/replicate2.log

In the /usr/local/avamar/bin/repl2_cron file, search for all other instances of repl_cron and replace it with repl2_cron.  This will modify log print outs and provide consistency when making future modifications, etc.

Modify the Replication Configuration Files

1) In the original repl_cron.cfg and the newly created repl2_cron.cfg files use the appropriate flags such as: --dstaddr, --dstid, --dstpassword, --exclude, and --include patterns to exclude certain clients or groups.  At this point it should be just as as you'd expect when making any modifications to replication jobs.  The server names should be specified correctly etc.  You can refer to the replication documentation for additional flags etc.  NOTE: You can not use the EMS or the MCS to edit the repl2_cron.cfg file.

2) If you launch concurrent replication sessions to the same target grid, you need to avoid overlapping clients between the two configuration files.  The easiest way to do this is to use the "--include" flag in the repl2_cron.cfg file to limit the replication to ONLY the "special clients" and use the "--exclude" flag in the standard repl_cron.cfg file to exclude these "special clients".

NOTE: If both replication sessions attempt to replicate the backups for the same client, the replication session that started first will lock the hash cache file, and will cause the second replication session to generate an error on that client.  The replicator will terminate after a set number of errors.

Modify the dpn crontab

1) As user root on the utility node, to list the existing cron:

% crontab -u dpn -l

The existing crontab of an Avamar server that performs replication looks something similar to:

--BEGIN--

# <<< BEGIN AXION ADMINISTRATOR MANAGED ENTRIES -- DO NOT MANUALLY MODIFY >>>

0 10 * * * /usr/local/avamar/bin/cron_env_wrapper morning_cron_run

0 18 * * * /usr/local/avamar/bin/cron_env_wrapper evening_cron_run

0 0 * * * /usr/local/avamar/bin/cron_env_wrapper /usr/local/avamar/lib//mcs_ssh_add repl_cron

# <<< END AXION ADMINISTRATOR MANAGED ENTRIES >>>

--END--

2) As user root or dpn on the utility node, modify the crontab to add the additional replication.

To modify the dpn crontab as user root, use:

% crontab -u dpn -e

After the modifications, the crontab should look like:

--BEGIN--

0 1 * * * /usr/local/avamar/bin/cron_env_wrapper /usr/local/avamar/lib/mcs_ssh_add repl2_cron

# <<< BEGIN AXION ADMINISTRATOR MANAGED ENTRIES -- DO NOT MANUALLY MODIFY >>>

0 10 * * * /usr/local/avamar/bin/cron_env_wrapper morning_cron_run

0 18 * * * /usr/local/avamar/bin/cron_env_wrapper evening_cron_run

0 0 * * * /usr/local/avamar/bin/cron_env_wrapper /usr/local/avamar/lib//mcs_ssh_add repl_cron

# <<< END AXION ADMINISTRATOR MANAGED ENTRIES >>>

--END—

NOTE: The MCS manages all of the entries within the "Avamar Administrator Managed Entries" section.  Since the MCS is only able to control _one_ replication instance, the additional replication job(s) must go ABOVE the "# <<< BEGIN..." line.

NOTE: Due to bug 14879, when the MCS restarts, the MCS will not only over-write any entries within the "Avamar Administrator Managed Entries", but will also delete any entries BELOW this section.  Therefore, until we resolve this bug, be sure to add the additional crontab entry ABOVE the "Avamar Administrator Managed Entries" block.

Allowing the new Replication process to run:

1.)    As the root user we’ll need to add the new replication job to dpncron.pm

a.       cd /usr/local/avamar/bin/

b.      open dpncron.pm with your favorite editor

2.)    Look for the following section:

--BEGIN--

my %exclude_list =

    (

        'cp_cron' => [ 'cp_cron' ],

        'gc_cron' => [ 'gc_cron' ],

        'hfscheck_cron' => [ 'hfscheck_cron', 'hfscheck_kill' ],

        'hfscheck_kill' => [ 'hfscheck_kill' ],

        'repl_cron' => [ 'repl_cron' ],

        'metadata_cron' => [ 'metadata_cron' ]

    );

--END--

            3.) Copy the repl_cron line and paste it onto the next line, then change repl_cron to the name of the new replication job (above we used repl2_cron)

--BEGIN--

my %exclude_list =

    (

        'cp_cron' => [ 'cp_cron' ],

        'gc_cron' => [ 'gc_cron' ],

        'hfscheck_cron' => [ 'hfscheck_cron', 'hfscheck_kill' ],

        'hfscheck_kill' => [ 'hfscheck_kill' ],

        'repl_cron' => [ 'repl_cron' ],

        'repl2_cron' => [ 'repl2_cron' ],      

        'metadata_cron' => [ 'metadata_cron' ]

    );

--END--

            This will allow the process called repl2_cron to be carried out completely.

 IMPORTANT: When we've multiple concurrent replication sessions, we have always started the two replication sessions at least one hour apart.  We don't know if this is really required, but given that every replication session always starts with a series of avmgr commands that modify the GSAN accounts on the replication target, we've always been concerned about have two different replication sessions simultaneously attempting to modify the accounting information on the replication target.

Avamar product information.

You need to have powerlink account to login below link. it has complete information on Avamar and its supported products
https://support.emc.com/products/avamar

to check DIMM errors on avamar grid.

below command will provide DIMM status for each node.

login as admin

admin@avamar_grid_name:~/>: mapall --all 'omreport chassis memory | egrep "^Connector|^Status"' 2>&1 | more Using /usr/local/avamar/var/probe.xml

to check power supply and DIMM errors on Avamar grid

To check Power supply on grid:

admin@avamargridname:~/> omreport chassis pwrsupplies

Power Supplies Information

------------------------------------------

Main system Chassis Power Supplies: OK

------------------------------------------

To get the DIMM errors

admin@avamargridname:~/> omreport chassis memory | egrep "^Connector|^Status"

Avamar resources link

For any Avamar related materials(like vendor documents, support documents, supported backup datatypes, version support, etc....) goto below link.

https://support.emc.com/products/avamar

Note: you need to have EMC powerlink account in order to access above link.

Replication throughput

Avamar 5.0

admin@source_gridname:/usr/local/avamar/etc/>: iperf -c repl_target_grid -w 60k -t 30 -i 10
------------------------------------------------------------
Client connecting to repl_target_grid, TCP port xxxx
TCP window size: 120 KByte (WARNING: requested 60.0 KByte)
------------------------------------------------------------
[ 3] local xx.xx.xx.xx port xxxxxx connected with xx.xx.xx.xx port xxxx
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 27.3 MBytes 22.9 Mbits/sec
[ 3] 10.0-20.0 sec 28.4 MBytes 23.8 Mbits/sec
[ 3] 20.0-30.0 sec 28.4 MBytes 23.8 Mbits/sec
[ 3] 0.0-30.0 sec 84.1 MBytes 23.5 Mbits/sec
This shows we have a intra-site bandwidth of roughly 23.8 Mbits/sec. It is generally accepted that Avamar replication can use around 60-80% of the total link capacity. If we use 70% as an average figure this gives us 16.6 Mbits/second.
We therefore have 16.6 Mbits/second = 7.47 Gigabytes/hr
With a backup window of 20 hours (current setting) we would be able to replicate up to 149.4 per day.
The client xxxx itself is replicating more than 100GB of new data per day. So we need to focus on the network.
Please check your network settings and let me know for any concerns. Thanks.