If checkpoints fails with DDR error

root@Avamar_server:~/#: cplist
cplist: ERROR: ddrmaint: <4750>Datadomain get checkpoint list operation failed.

cp.20120926170740 Wed Sep 26 12:07:40 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120926185155 Wed Sep 26 13:51:55 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120927170726 Thu Sep 27 12:07:26 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120927184726 Thu Sep 27 13:47:26 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120928170738 Fri Sep 28 12:07:38 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120928190034 Fri Sep 28 14:00:34 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120929170731 Sat Sep 29 12:07:31 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120929184944 Sat Sep 29 13:49:44 2012 invalid --- --- nodes 5/5 stripes 77848
cp.20120930170728 Sun Sep 30 12:07:28 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20120930184916 Sun Sep 30 13:49:16 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121001170739 Mon Oct 1 12:07:39 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121001184754 Mon Oct 1 13:47:54 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121002170739 Tue Oct 2 12:07:39 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121002190634 Tue Oct 2 14:06:34 2012 invalid --- --- nodes 5/5 stripes 77850
cp.20121003170734 Wed Oct 3 12:07:34 2012 invalid --- --- nodes 5/5 stripes 77852
cp.20121003185200 Wed Oct 3 13:52:00 2012 invalid --- --- nodes 5/5 stripes 77852
cp.20121004170736 Thu Oct 4 12:07:36 2012 invalid --- --- nodes 5/5 stripes 77852
cp.20121004190124 Thu Oct 4 14:01:24 2012 invalid --- --- nodes 5/5 stripes 77852
cp.20121005170734 Fri Oct 5 12:07:34 2012 invalid --- --- nodes 5/5 stripes 77854
cp.20121005190516 Fri Oct 5 14:05:16 2012 invalid --- --- nodes 5/5 stripes 77854
cp.20121006170757 Sat Oct 6 12:07:57 2012 invalid --- --- nodes 5/5 stripes 77860
cp.20121006185724 Sat Oct 6 13:57:24 2012 invalid --- --- nodes 5/5 stripes 77860
cp.20121007170750 Sun Oct 7 12:07:50 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121007185441 Sun Oct 7 13:54:41 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121008170750 Mon Oct 8 12:07:50 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121008185438 Mon Oct 8 13:54:38 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121009170757 Tue Oct 9 12:07:57 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121009190138 Tue Oct 9 14:01:38 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121010170754 Wed Oct 10 12:07:54 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121010190509 Wed Oct 10 14:05:09 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121011170740 Thu Oct 11 12:07:40 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121011185927 Thu Oct 11 13:59:27 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121012170652 Fri Oct 12 12:06:52 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121012185325 Fri Oct 12 13:53:25 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121013170656 Sat Oct 13 12:06:56 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121013185554 Sat Oct 13 13:55:54 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121014170505 Sun Oct 14 12:05:05 2012 invalid --- --- nodes 5/5 stripes 77862
cp.20121014192822 Sun Oct 14 14:28:22 2012 invalid --- --- nodes 5/5 stripes 77862

Solution:

The /usr/local/avamar/var/ddr_info file had been zero’d out.  need to recreate ddr_info file, and wait for the extra checkpoints to be removed after next cp and hfschecks.

to check DIMM errors on avamar grid.

login as admin

add ssh keys

admin@avamar_grid_name:~/>: mapall --all 'omreport chassis memory | egrep "^Connector|^Status"' 2>&1 | more Using /usr/local/avamar/var/probe.xml

To check Garbage collection status

userid@avamar_grid> avmaint gcstatus

To kill Garbage collection manually through command line

user@avamar_gridname> avmaint -ava gckill

if the avamar has Datadomain then kill this process after gckill ran and gc still running.


user@avamar_gridname> ps -ef|grep ddrmaint
and "kill -9 process_id"
-9 is to kill forcibly.

Avamar passwords

To find out Avamar passwords, goto


more /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml and grep for userid's like MCUser/root/dpn

Avamar 6.1 released at EMC World 2012 - Las Vegas - May 21, 2012

http://www.emc.com/about/news/press/2012/20120521-06-03.htm

replicating client level from command line

nohup replicate --flagfile=/usr/local/avamar/etc/repl_cron.cfg --timeout=0 --include=/clients/client_name --logfile=/tmp/replication3.log >> /tmp/replication3.log

To know the grids configuration about GC, CP, Readonly

admin@avamar_grid01:~/>: avmaint --ava config | grep disk

disknocreate="90"

disknocp="96"

disknogc="87"

disknoflush="94"

diskwarning="50"

diskreadonly="65"

disknormaldelta="2"

diskfull="30"

diskfulldelta="5"

balancelocaldisks="false"

admin@avamar_grid01:~/>

Avamar windows system state backup failure.

One of the work around for system state backups. First install the New 6.0.1-66 client - AvamarClientWindows-x86_x64-6.0.101-66._HF34148.msi or later versions on the windows client.


 
Symptom

System state backup failed with the following error message:
"System writer is not found"

In addition, when you run the “Vssadmin list writers” command to display VSS writers on the system, the System Writer is missing.


Cause
The system writer fails due to the fact that the trusted installer & system accounts are missing permissions to files in the directory %windir%\winsxs\filemaps\.

Resolution
To resolve this issue, please run the following commands from an administrative elevated Command Window:

CD c:\windows\system32
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant "BUILTIN\Users:(RX)"

Applies to
Windows Server 2003, Windows Server 2008

Avamar: Multiple stream replication

Note: Before applying these changes, should have Avamar admin knowledge atleast a year or more.

Modifying the "repl_cron" Script

1) As user root on the replication source utility node, copy the /usr/local/avamar/bin/repl_cron script to a different, preferably meaningful, name.

% cd /usr/local/avamar/bin

% cp -p repl_cron repl2_cron

2) As user admin on the replication source utility node, copy the /usr/local/avamar/etc/repl_cron.cfg file to a different name, preferably a name that is consistent with the name used for the repl_cron program.  NOTE: Do not modify the original repl_cron.cfg file in the /usr/local/avamar/bin directory.  Copy the in-use repl_cron.cfg in the /usr/local/avamar/etc directory.

% cd /usr/local/avamar/etc

% cp -p repl_cron.cfg repl2_cron.cfg

3) As user root, edit the /usr/local/avamar/bin/repl2_cron for modification:

The portion of code that is of interest on the original repl_cron looks like:

--BEGIN--  

sub init {

    dpn::add_legal("timeout:d", "configfile:s");

    dpncron::init("repl_cron", "replicate", 4600, "--infomsgs --verbose");

    $configfile = "$dpn::avamardir/etc/repl_cron.cfg";

    $configfile = $dpn::flags{configfile} if $dpn::flags{configfile};

    dpncron::tprint("repl_cron [$version]: configfile = '$configfile'\n") if $dpn::flags{verbose};

}

--END--

Using the repl2_cron examples as provided earlier modify the code to look like:

--BEGIN--  

sub init {

    dpn::add_legal("timeout:d", "configfile:s");

    dpncron::init("repl2_cron", "replicate2", 4600, "--infomsgs --verbose");

    $configfile = "$dpn::avamardir/etc/repl2_cron.cfg";

    $configfile = $dpn::flags{configfile} if $dpn::flags{configfile};

    dpncron::tprint("repl2_cron [$version]: configfile = '$configfile'\n") if $dpn::flags{verbose};

}

--END--

Notes about the modifications:

-          The new variable "repl2_cron" is the name of the program that is to be called.  In this case the program being called is /usr/local/avamar/bin/repl2_cron

-          The new variable "replicate2" is base name of the replicate log.  In this case the replicate log for repl2_cron will be in /usr/local/avamar/var/cron/replicate2.log

In the /usr/local/avamar/bin/repl2_cron file, search for all other instances of repl_cron and replace it with repl2_cron.  This will modify log print outs and provide consistency when making future modifications, etc.

Modify the Replication Configuration Files

1) In the original repl_cron.cfg and the newly created repl2_cron.cfg files use the appropriate flags such as: --dstaddr, --dstid, --dstpassword, --exclude, and --include patterns to exclude certain clients or groups.  At this point it should be just as as you'd expect when making any modifications to replication jobs.  The server names should be specified correctly etc.  You can refer to the replication documentation for additional flags etc.  NOTE: You can not use the EMS or the MCS to edit the repl2_cron.cfg file.

2) If you launch concurrent replication sessions to the same target grid, you need to avoid overlapping clients between the two configuration files.  The easiest way to do this is to use the "--include" flag in the repl2_cron.cfg file to limit the replication to ONLY the "special clients" and use the "--exclude" flag in the standard repl_cron.cfg file to exclude these "special clients".

NOTE: If both replication sessions attempt to replicate the backups for the same client, the replication session that started first will lock the hash cache file, and will cause the second replication session to generate an error on that client.  The replicator will terminate after a set number of errors.

Modify the dpn crontab

1) As user root on the utility node, to list the existing cron:

% crontab -u dpn -l

The existing crontab of an Avamar server that performs replication looks something similar to:

--BEGIN--

# <<< BEGIN AXION ADMINISTRATOR MANAGED ENTRIES -- DO NOT MANUALLY MODIFY >>>

0 10 * * * /usr/local/avamar/bin/cron_env_wrapper morning_cron_run

0 18 * * * /usr/local/avamar/bin/cron_env_wrapper evening_cron_run

0 0 * * * /usr/local/avamar/bin/cron_env_wrapper /usr/local/avamar/lib//mcs_ssh_add repl_cron

# <<< END AXION ADMINISTRATOR MANAGED ENTRIES >>>

--END--

2) As user root or dpn on the utility node, modify the crontab to add the additional replication.

To modify the dpn crontab as user root, use:

% crontab -u dpn -e

After the modifications, the crontab should look like:

--BEGIN--

0 1 * * * /usr/local/avamar/bin/cron_env_wrapper /usr/local/avamar/lib/mcs_ssh_add repl2_cron

# <<< BEGIN AXION ADMINISTRATOR MANAGED ENTRIES -- DO NOT MANUALLY MODIFY >>>

0 10 * * * /usr/local/avamar/bin/cron_env_wrapper morning_cron_run

0 18 * * * /usr/local/avamar/bin/cron_env_wrapper evening_cron_run

0 0 * * * /usr/local/avamar/bin/cron_env_wrapper /usr/local/avamar/lib//mcs_ssh_add repl_cron

# <<< END AXION ADMINISTRATOR MANAGED ENTRIES >>>

--END—

NOTE: The MCS manages all of the entries within the "Avamar Administrator Managed Entries" section.  Since the MCS is only able to control _one_ replication instance, the additional replication job(s) must go ABOVE the "# <<< BEGIN..." line.

NOTE: Due to bug 14879, when the MCS restarts, the MCS will not only over-write any entries within the "Avamar Administrator Managed Entries", but will also delete any entries BELOW this section.  Therefore, until we resolve this bug, be sure to add the additional crontab entry ABOVE the "Avamar Administrator Managed Entries" block.

Allowing the new Replication process to run:

1.)    As the root user we’ll need to add the new replication job to dpncron.pm

a.       cd /usr/local/avamar/bin/

b.      open dpncron.pm with your favorite editor

2.)    Look for the following section:

--BEGIN--

my %exclude_list =

    (

        'cp_cron' => [ 'cp_cron' ],

        'gc_cron' => [ 'gc_cron' ],

        'hfscheck_cron' => [ 'hfscheck_cron', 'hfscheck_kill' ],

        'hfscheck_kill' => [ 'hfscheck_kill' ],

        'repl_cron' => [ 'repl_cron' ],

        'metadata_cron' => [ 'metadata_cron' ]

    );

--END--

            3.) Copy the repl_cron line and paste it onto the next line, then change repl_cron to the name of the new replication job (above we used repl2_cron)

--BEGIN--

my %exclude_list =

    (

        'cp_cron' => [ 'cp_cron' ],

        'gc_cron' => [ 'gc_cron' ],

        'hfscheck_cron' => [ 'hfscheck_cron', 'hfscheck_kill' ],

        'hfscheck_kill' => [ 'hfscheck_kill' ],

        'repl_cron' => [ 'repl_cron' ],

        'repl2_cron' => [ 'repl2_cron' ],      

        'metadata_cron' => [ 'metadata_cron' ]

    );

--END--

            This will allow the process called repl2_cron to be carried out completely.

 IMPORTANT: When we've multiple concurrent replication sessions, we have always started the two replication sessions at least one hour apart.  We don't know if this is really required, but given that every replication session always starts with a series of avmgr commands that modify the GSAN accounts on the replication target, we've always been concerned about have two different replication sessions simultaneously attempting to modify the accounting information on the replication target.

Avamar product information.

You need to have powerlink account to login below link. it has complete information on Avamar and its supported products
https://support.emc.com/products/avamar

to check DIMM errors on avamar grid.

below command will provide DIMM status for each node.

login as admin

admin@avamar_grid_name:~/>: mapall --all 'omreport chassis memory | egrep "^Connector|^Status"' 2>&1 | more Using /usr/local/avamar/var/probe.xml

to check power supply and DIMM errors on Avamar grid

To check Power supply on grid:

admin@avamargridname:~/> omreport chassis pwrsupplies

Power Supplies Information

------------------------------------------

Main system Chassis Power Supplies: OK

------------------------------------------

To get the DIMM errors

admin@avamargridname:~/> omreport chassis memory | egrep "^Connector|^Status"

Avamar resources link

For any Avamar related materials(like vendor documents, support documents, supported backup datatypes, version support, etc....) goto below link.

https://support.emc.com/products/avamar

Note: you need to have EMC powerlink account in order to access above link.

Replication throughput

Avamar 5.0

admin@source_gridname:/usr/local/avamar/etc/>: iperf -c repl_target_grid -w 60k -t 30 -i 10
------------------------------------------------------------
Client connecting to repl_target_grid, TCP port xxxx
TCP window size: 120 KByte (WARNING: requested 60.0 KByte)
------------------------------------------------------------
[ 3] local xx.xx.xx.xx port xxxxxx connected with xx.xx.xx.xx port xxxx
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 27.3 MBytes 22.9 Mbits/sec
[ 3] 10.0-20.0 sec 28.4 MBytes 23.8 Mbits/sec
[ 3] 20.0-30.0 sec 28.4 MBytes 23.8 Mbits/sec
[ 3] 0.0-30.0 sec 84.1 MBytes 23.5 Mbits/sec
This shows we have a intra-site bandwidth of roughly 23.8 Mbits/sec. It is generally accepted that Avamar replication can use around 60-80% of the total link capacity. If we use 70% as an average figure this gives us 16.6 Mbits/second.
We therefore have 16.6 Mbits/second = 7.47 Gigabytes/hr
With a backup window of 20 hours (current setting) we would be able to replicate up to 149.4 per day.
The client xxxx itself is replicating more than 100GB of new data per day. So we need to focus on the network.
Please check your network settings and let me know for any concerns. Thanks.

kill HFSCheck

Avamar 5.0

login to CLI and run below command

hfscheck_kill

GSAN capacity of avamar grid

Avamar version 5.0

avmaint nodelist|grep "fs-percent-full"

How to check HFS check through CLI.

Avamar Version 5.0

dumpmaintlogs --types=hfscheck --days=1

Avamar: to activate clients from client side

unix:

/client_pkg_installed/etc/avagent.d register avamar_gridname domain_name

ex:

/usr/local/avamar/etc/avagent.d register avamar01 /clients/unix

windows:

right click on Avamar icon and activate:

fillup avamar grid name and domain name.

Avamar provides fault tolerance at several different levels:

RAID and RAIN protects the data loss.

• RAID: Avamar ensures protection from disk and data corruption through the use of RAID (redundant array of independent disks). The type of RAID depends on the particular node type. The first thing you must know is that all storage nodes have 6 physical disks. Beginning with Avamar release 4.0, two sizes are supported: nodes with 1TB of licensable capacity or 2TB of licensable capacity. Single-node servers with a capacity of 1TB actually have higher performance disks than the 2TB flavor. The 1TB storage nodes are 6 300GB 15k SAS drives setup in RAID-5, configured into 4 Luns (virtual disks). If you opt for the 2TB capacity, the storage node will have 6 SATA disks configured with 3 RAID-1 Luns. Just FYI, the Utility node and NDMP accelerator nodes both only have 2 146GB physical disks configured with RAID-1.
• RAIN: In an multi-node system (1 utility node & 3 storage nodes or more), Avamar provides failover and fault tolerance across all nodes usingRAIN (redundant array of independent nodes). If a node failure occurs, the Avamar Server continues to function, during this time, backup data for recovery will be reconstructed on the remaining nodes using parity. Once the failed node is replaced (a spare node is always included in a RAIN configuration), the capacity across all disks can be rebalanced using a very simple process.

Avamar windows open files VSS bug.

<http://powerlink.emc.com/km/appmanager/km/secureDesktop?_nfpb=true&_pageLabel=homePgSecureContentBk>
EMC Solutions



EMC Solutions

Solution ID:
esg122338<http://solutions.emc.com/EMCSolutionView.asp?id=esg122338>
Product(s):
Avamar Client for Windows

Date Created:
06/14/2011
Category(ies):
Bug or Defect

Date Modified:
06/21/2011
Status:
Approved

Use Count:
0
Related Bugs:
26904



Subject:
ETA esg122338: Avamar Client for Windows: During Windows NTFS backups, the Avamar client software might fail to back up files, folders, and/or volumes and not report the exceptions

Solution:
EMC Technical Advisory
ETA esg122338: Avamar Client for Windows: During Windows NTFS backups, the Avamar client software might fail to back up files, folders, and/or volumes and not report the exceptions

Symptoms
If the Microsoft Volume Shadow Copy Service (VSS) fails during an NTFS backups of Windows clients running Windows 2003 SP1 or more recent Windows versions, then the Avamar Windows file system client software might fail to backup files, folders, and/or volumes without reporting errors for all the files, folders, and/or volumes that did not get backed up.

In some cases the backup might report a "completed" status, which means that no errors were reported during the backup, but might fail to back up an entire volume.

In other cases, the backup might report a "completed with exceptions" status, but entire folders might not be backed up, and not all the relevant exceptions are reported. As a result, the size of the folders that were not backed up is reported in the restore window to be 0 bytes.

In these cases, the Windows system "Event Viewer" will log one or more of the following VSS related errors or warnings during the backup (using volume C: as an example):

=================================

Event Type: Error
Event Source: VolSnap
Event Category: None
Event ID: 35
Date: 4/11/2011
Time: 1:00:21 AM
User: N/A
Computer: CORP-AVAMAR1
Description:
The shadow copies of volume C: were aborted because the shadow copy storage failed to grow.

=================================

Event Type: Error
Event Source: VolSnap
Event Category: None
Event ID: 13
Date: 4/11/2011
Time: 12:59:44 AM
User: N/A
Computer: CORP-AVAMAR1
Description:
The shadow copy of volume C: could not grow its shadow copy storage on volume F:.

=================================

Event Type: Error
Event Source: VolSnap
Event Category: None
Event ID: 25
Date: 4/7/2011
Time: 1:55:22 AM
User: N/A
Computer: CORP-AVAMAR1
Description:
The shadow copies of volume C: were deleted because the shadow copy storage could not grow in time. Consider reducing the IO load on the system or choose a shadow copy storage volume that is not being shadow copied.

=================================

Event Type: Warning
Event Source: VolSnap
Event Category: None
Event ID: 24
Date: 4/13/2011
Time: 4:05:24 PM
User: N/A
Computer: CORP-AVAMAR1
Description:
There was insufficient disk space on volume C: to grow the shadow copy storage for shadow copies of C:. As a result of this failure all shadow copies of volume C: are at risk of being deleted.

Affected Environments
All supported Avamar Windows file system plug-in versions up to, and including, v5.0.106-28 are affected.

Impact
Files, folders, and/or volumes on the Windows client's file system might not be backed up due to this issue and there might not be errors reported for files, folders, and/or volumes that are not backed up.

Cause
The Avamar backup software invokes VSS to create a snapshot ("shadow copy") of the data on each NTFS volume being backed up. The Avamar client software will then back up the files and folders on each snapshot. If VSS encounters an issue on a volume, then the files and folders on the snapshot of the volume will no longer be available to be backed up. If the Avamar client software had not yet started to process a folder when the snapshot disappears, then the folder might fail to be backed up without an error message ("exception") noting that the folder was not backed up. If, while backing up one volume, VSS encounters an issue on another volume, then the Avamar client software will not back up any files or folders on the affected volume because the snapshot of that volume disappears, and no error message is generated to note that the volume did not get backed up.

NOTE: If VSS fails at the beginning of the backup, then no snapshot is created, and the Avamar client software will attempt to back up the files/folders on each volume. Any files that are "locked" by another process will not be backed up, and the Avamar software will report an error for each file that could not be backed up.

Resolution
1. Upgrade to Windows file system client plug-in version 5.0.106-28 and then apply Hotfix 26904.
Or
2. Upgrade to Windows file system client plug-in version 6.0.100-592 or later.

These upgraded versions will provide improved visibility to the files, folders, and/or volumes that are not backed up when the snapshot disappears.


ETAs are Knowledgebase solutions that address:
*

Critical problems that cannot be prevented or controlled through proactive tools and processes.
* Product defect that might cause unexpected behavior or any kind of data loss.
Use the information in this EMC Advisory to avoid any situation that might arise from the problems described in the Knowledgebase solution. Knowledgebase solutions are updated as new information becomes available until a permanent solution exists. For questions regarding this advisory, contact EMC CSS Remote at 1-877-534-2867 .

Back<http://solutions.emc.com/emcsolutionview.asp?id=esg122338>

Avamar deduplication.

Introduced at our client location since 2009. and its one of the best dedup and efficient backups for structured and remote backups.