I've found that Exchange 2010 database copies (in a DAG group) have a tendency to fail if the DAG cluster is geographically distributed, and the WAN goes down occasionally. Microsoft specifies that connectivity should be redundant,etc, but that isn't a reality for some organizations who want to don't want the time or expense of maintaining a highly redundant infrastructure, when a slightly less redundant one will work 95% as well. That being said, we need a few scripts to mitigate the risks and address the issues in these scenarios. 

 

The simple scenario is this:

• Two Exchange 2010 servers, each running all roles (except UM)
• Each server is in a different datacenter, and has access to a local domain controller & GC
• The servers are running in a DAG and are replicating ALL databases between each other
• There are no redundant networks; each Exchange server has a single NIC, and they replicate over a VPN tunnel
• We specify which server we "prefer" each database to be on (with activation preference)
• There is a separate DNS record for each server externally (e.g., mail1.domain.com, mail2.domain.com)
• There is a single CNAME record for CAS access, which we always want to point to the CAS with the active database copy (e.g., mail.domain.com --> mail1.domain.com)

Problems

• Although auto-discover is great in some scenarios, it doesn't work well for Outlook Anywhere (RPC over HTTP), and hence the failover between CAS servers requires manual DNS intervention.
• Database copies fail occasionally and don't automatically reseed and remount themselves (update function)
• Database failovers take place occasionally, but there is no facility for failback to the preferred server, which we want because the 2nd server is really just for disaster recovery

 

Therefore we need three pieces of functionality that Microsoft does not provide:

1. DNS - We need our DNS for CAS to follow our database activation, automatically
2. Database copy resiliency - We need our databases to attempt to update themselves (reseed) and remount if they fail, automatically
3. Database Activation - We need our database copies to failback to any server marked with a better activation preference, automatically

 

The term "automatically" is key; although we could tie this to event logs or other types of triggers, we want to keep it simple and reliable, so we opted to use task scheduler to just run these Powershell scripts once every 15 minutes:

 

 

[ 1. ]

cd\

cls

#must have the DnsShell module from http://code.msdn.microsoft.com/dnsshell 

Import-Module DnsShell

#load exchange2010 mgmt snapins

Add-PSSnapin Microsoft.Exchange.Management.Powershell.Support

Add-PSSnapin  Microsoft.Exchange.Management.PowerShell.E2010

#input global variables and declarations

$CnameName = "Cname1.domain.com"

$CnameName2 = "Cname2.domain.com"

$cnames = $CnameName,$CnameName2

$DBname = "Database 1"

$DBname2 = "Database 2"

$DBs = $DBname,$DBname2

$Cname2DBArray = $DBname,$CnameName,$DBname2,$CnameName2

$MailServer01DNS = "mail1.domain.com."

$MailServer02DNS = "mail2.domain.com."

$Arecords = $MailServer01DNS,$MailServer02DNS

$server1 = "MailServer01"

$server2 = "MailServer02"

$servers = $server1,$server2

$EXCHarray = $server1,$MailServer01DNS,$server2,$MailServer02DNS

$DnsServerVar = "dns-server.domain.com"

$startstring="Start script run at:  "

$startendtime=date

$startannounce=$startstring+$startendtime

#end input global variables

#begin functions

#function to write event to Windows if DB is moved

function writeEvent1111([string]$eventChangeMade)

{

$evt=new-object System.Diagnostics.EventLog("Application")

$evt.Source="Exchange DNS follow DB"

$infoevent=[System.Diagnostics.EventLogEntryType]::Warning

$1stpart="DNS follows Exchange DB Event: "

$2ndpart=" , see log for more details at C:\Program Files\Microsoft\Exchange Server\V14\Logging\Failback_logs"

$eventStringfull=$1stpart+$eventChangeMade+$2ndpart

$evt.WriteEntry($eventStringfull,$infoevent,1111)

}

#Gets the current server name with the specified database active

function getCurrentServerName([string]$feedMeDB)

{

$currentServerinfo = Get-MailboxDatabase -identity $feedMeDB

Write-Output "Getting DB server list from Exchange..."

Write-Output $currentServerinfo

Set-Variable -name CurrentServerName -value $currentServerinfo.server.name -scope global

}

#gets the current DNS name assocated with the CNAME record (what it's pointing to)

function getCurrentDNS([string]$currentCNAME)

{

Write-Output "Getting DNS records(s) from Domain Controllers..."

$dnsCMD1 = Get-DnsRecord $currentCNAME -Server $DnsServerVar

Set-Variable -name DNSget -value $dnsCMD1 -scope global

Set-Variable -name DNSrecord -value $DNSget.recordData -scope global

Set-Variable -name CnameID -value $DNSget.identity -scope global

}

#Make a change to the CNAME record with the new pointer passed to this function frm the if clause below that matches where it should be

function setDNSchange([string]$newHostName)

{

Write-Output "Making DNS change... "

set-dnsrecord -server $DnsServerVar -identity $CnameID -Hostname $newHostName

}

#exit the program cleanly, announcing the change that was made (function called only if a change was made)

function exitClean([string]$changeMade)#should always exit program here

{

$text1 = "The DNS record: "

$text2 = " was changed from: "

$text3 = " to: "

$finalConcat = $text1+$tempCNAME+$text2+$DNSrecord+$text3+$changeMade

Write-Output $finalConcat

writeEvent1111 $finalConcat

}

#end functions

#make sure you create this folder

Start-Transcript -Append -Force -Path 'C:\Program Files\Microsoft\Exchange Server\V14\Logging\Failback_logs\DNSfollowsExchDB.log'

$startannounce

" "

#begin main program

#step 1 - call the function to get the active server name and display the results

FOREACH ($db in $DBs)

{

getCurrentServerName $db;

$ServerNamePosition = 0..($EXCHarray.length - 1) | where {$EXCHarray[$_] -eq $CurrentServerName}

$DNSnamePosition = $ServerNamePosition + 1

$correctDNS = $EXCHarray[$DNSnamePosition]

$dbNamePosition = 0..($Cname2DBArray.length - 1) | where {$Cname2DBArray[$_] -eq $db}

$CnamePosition = $dbNamePosition + 1

$tempCNAME = $Cname2DBArray[$CnamePosition]

#step 2 - call the function to get the active DNS record and display the results

getCurrentDNS $tempCNAME

Write-Output "Current DNS Record is: "

Write-Output $DNSget

Write-Output "Current WMI location of this record is: "

Write-Output $CnameID

#step 3 - makes changes as needed

If ($DNSrecord -ne $correctDNS)

{setDNSchange $correctDNS ; exitClean $correctDNS}

else{Write-Output "dns is correct for " $db}

}

#end main program

" "

stop-transcript

#exit default if no changes made

exit

[ 2. ]

#this script should be run from the preferred server, which implies it is functional

#  first subroutine to get and display status of each DB; if db is activated on the preferred server, the metric is in Exchange activation preference

#  second subroutine to move each DB to the preferred server, the metric is in Exchange activation preference

## INCREASE WINDOW WIDTH #####################################################

#prepare evenlog but only use if db is moved

function WidenWindow([int]$preferredWidth)

{

  [int]$maxAllowedWindowWidth = $host.ui.rawui.MaxPhysicalWindowSize.Width

  if ($preferredWidth -lt $maxAllowedWindowWidth)

  {

    # first, buffer size has to be set to windowsize or more

    # this operation does not usually fail

    $current=$host.ui.rawui.BufferSize

    $bufferWidth = $current.width

    if ($bufferWidth -lt $preferredWidth)

    {

      $current.width=$preferredWidth

      $host.ui.rawui.BufferSize=$current

    }

    # else not setting BufferSize as it is already larger

    

    # setting window size. As we are well within max limit, it won't throw exception.

    $current=$host.ui.rawui.WindowSize

    if ($current.width -lt $preferredWidth)

    {

      $current.width=$preferredWidth

      $host.ui.rawui.WindowSize=$current

    }

    #else not setting WindowSize as it is already larger

  }

}

WidenWindow(120)

function exitClean([string]$DBmoved)#should always exit program here

{

$text1 = "The DB: "

$text2 = " was moved from server: "

$text3 = " to server: "

$finalConcat = $text1+$DBmoved+$text2+$xNow+$text3+$dbown.key.name

Write-Output $finalConcat

writeEvent1112 $finalConcat

}

#function to write event to Windows if DB is moved

function writeEvent1112([string]$eventChangeMade)

{

$evt=new-object System.Diagnostics.EventLog("Application")

$evt.Source="Exchange DB Failback"

$infoevent=[System.Diagnostics.EventLogEntryType]::Warning

$1stpart="DB Failback Event for Exchange DB, "

$2ndpart=" , see log for more details at C:\Program Files\Microsoft\Exchange Server\V14\Logging\Failback_logs"

$eventStringfull=$1stpart+$eventChangeMade+$2ndpart

$evt.WriteEntry($eventStringfull,$infoevent,1112)

}

$startstring="Start script run at:  "

$startendtime=date

$startannounce=$startstring+$startendtime

#start Exchange Powershell snapins

cd\

Add-PSSnapin Microsoft.Exchange.Management.Powershell.Support

Add-PSSnapin  Microsoft.Exchange.Management.PowerShell.E2010

#start FailbackExchDBtoPreferredServer

Start-Transcript -Append -Force -Path 'C:\Program Files\Microsoft\Exchange Server\V14\Logging\Failback_logs\failback.log'

$startannounce

" "

"Status"

Get-MailboxDatabase | Sort Name | FOREACH {$db=$_.Name; $xNow=$_.Server.Name ;$dbown=$_.ActivationPreference| Where {$_.Value -eq 1};$quoteon=" on ";$quotesb=" Should be on ";If ( $xNow -ne $dbOwn.Key.Name){$stat=" WRONG"; }ELSE{$stat=" OK" };  $OutP=$db+$quoton+$xNow+$quotesb+$dbOwn.Key.Name+$stat; write-output $OutP}

" "

"Moves (if any)"

Get-MailboxDatabase | Sort Name | FOREACH {$db=$_.Name; $xNow=$_.Server.Name ;$dbown=$_.ActivationPreference| Where {$_.Value -eq 1};$quoteon=" on ";$quotesb=" Should be on ";If ( $xNow -ne $dbOwn.Key.Name){$stat=" MOVING..."; }ELSE{$stat=" OK" };  $OutP=$db+$quoton+$xNow+$quotesb+$dbOwn.Key.Name+$stat; write-output $OutP; If ( $xNow -ne $dbOwn.Key){Move-ActiveMailboxDatabase $db -ActivateOnServer $dbown.key.name -Confirm:$False; exitClean $db }}

" "

stop-transcript

#end FailbackExchDBtoPreferredServer

exit

[ 3. ]

cd\

cls

#load exchange2010 mgmt snapins

Add-PSSnapin Microsoft.Exchange.Management.Powershell.Support

Add-PSSnapin  Microsoft.Exchange.Management.PowerShell.E2010

#input global variables and declarations

$startstring="Start script run at:  "

$startendtime=date

$startannounce=$startstring+$startendtime

#end input global variables

#begin functions

#function to write event to Windows if DB is moved

function writeEvent1113([string]$eventChangeMade)

{

$evt=new-object System.Diagnostics.EventLog("Application")

$evt.Source="Exchange attempt to update DB copy"

$infoevent=[System.Diagnostics.EventLogEntryType]::Warning

$1stpart="Exchange attempt to fix DB with Update: "

$2ndpart=" , see log for more details at C:\Program Files\Microsoft\Exchange Server\V14\Logging\Failback_logs"

$eventStringfull=$1stpart+$eventChangeMade+$2ndpart

$evt.WriteEntry($eventStringfull,$infoevent,1113)

}

#exit the program cleanly, announcing the change that was made (function called only if a change was made)

function exitClean([string]$changeMade)

{

$text1 = "The Database Copy: "

$text2 = " was attempted to be updated and re-mounted "

$finalConcat = $text1+$changeMade+$text2

Write-Output $finalConcat

writeEvent1113 $finalConcat

}

#attempt to update the failed database

function updateDBnow([string]$passedID)

{

Update-MailboxDatabaseCopy -Force -Identity $passedID

}

#end functions

Start-Transcript -Append -Force -Path 'C:\Program Files\Microsoft\Exchange Server\V14\Logging\Failback_logs\DNSfollowsExchDB.log'

$startannounce

" "

#begin main program

#identify any database copies in the failed state and pass their identity to the updateDBnow function

Get-MailboxDatabase | Sort Name | FOREACH{$db=$_.Name; Get-MailboxDatabaseCopyStatus -Identity $db | FOREACH{If($_.Status -eq "FailedandSuspended"){$PassID = $_.Identity;updateDBnow $PassID;exitClean $PassID}ELSE{write-host "Database: " $_.Name " is: " $_.Status "  Mounted on: " $_.MailBoxServer}}}

#end main program

" "

stop-transcript

#exit default if no changes made

exit