Fixing Information Store Error -2147221213 (ecTimeSkew) – Exchange 2007 / 2010 / 2013

(Late) Sunday nights are not a fun time to get paged about a customer down…

Situation

Engineer reboots an Exchange 2010 server that is apart of a two node DAG, but when the server comes back online the MS Exchange Information Store service is stuck in the “Starting” phase within the Services MMC, and the Microsoft Exchange System Attendant will just not start. All other services are online. What gives?

Environment

Another pretty simple buildout, Exchange 2010 SP3UR6 running on top of Server 2008 R2 with all patches (security / hotfixes) that are available. Both servers are multiple-role (CAS/HUB/Mailbox) which also has a CAS array and a two node DAG. Two copies of each database in the environment.

Troubleshooting

The first thing I did was confirm that the only two services that were boinked at the time were the Information Store and System Attendant. Best way to actually check this is by running Test-ServiceHealth if you are running Exchange 2010 / 2013, which showed exactly that.

When I looked in the Services MMC I did see the Exchange Information Store stuck in the starting state. I looked in the Event Viewer: Application logs to find this error:

Log Name:      Application
Source:        MSExchangeIS
Date:          8/24/2014 11:51:29 PM
Event ID:      5003
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EXCH01.company.com
Description:
Unable to initialize the Information Store service because the clocks on the client and server are skewed. This may be caused by a time change either on the client or on the server, and may require a restart of that computer. Verify that your domain is correctly configured and  is currently online.

The first thing I did was make sure the Windows Time service was properly pulling the right time source..

w32tm /query /source

Since that looked proper (it was the PDC) I moved on and physically checked the time between the two Exchange servers, which looked fine. Confused, I decided to try a force sync of the Windows Time service..

w32tm /resync

Afterwards, performed a reboot and avail.. nothing. Back to square one with the databases showing a state of “Failed” and but this time the Information Store service did not get stuck in “starting”, it was stopped all together. I attempted to start the service with the following error message:

“Unable to start service. Error -2147221213. Please contact your support provider or the vendor”

This is a hex error. Luckily enough if you know the internals a bit to Exchange, sometimes you can look up these errors using the Err.exe tool MSFT provides (link).

That then shows us this..

# for decimal -2147221213 / hex 0x80040123 :
  ecTimeSkew                                                    ec.h
  MAPI_E_INVALID_ACCESS_TIME                                    mapicode.h

We now know its a windows time issue, so now what?

Resolution

There are two routes you can go with this:

- Manually set the Windows Time service to sync off the PDC (which is what I did)

- Completely reset the Windows Time service

Manually set Windows Time service

This is pretty easy, and all you have to do is run this command

net time \\PDCName /Set

It will ask you if you want to confirm, click Y and then confirm that the time is now synced. Afterwards, restart the MS Exchange Active Directory Topology service which will restart all the other Exchange services, and hopefully its fixed for you.

If that doesn’t work try resetting Windows Time all together, and then set it to point to the PDC (sometimes required)..

Reset the Windows Time service

Open up a command prompt and then each of the commands below (each line break is a new command to run)..

net stop w32time
w32tm /unregister
w32tm /register
net start w32time

Once this is done, you should check the actual provider to make sure it is either the DC that is hosting the PDC FSMO role (recommended) or another domain controller that is local to Exchange (e.g: within the same AD site)

w32tm /query /source

If it is NOT the DC holding the PDC FSMO role or even a DC (it might show as “CMOS”, which means your BIOS) then I would set it to either one of the options I said above..

net time \\PDCName /Set

At this point restart the MS Exchange Active Directory Topology service and make sure everything starts.

Once either one of these steps are done, you should be fully working again.

 

TL;DR: Windows Time sync is required for domain controller authentication on a domain joined machine, along with authentication requests. If windows time breaks (or Windows thinks its broken when the time is actually synced), now you know how to fix it!

Leave any questions and comments below.

 

Cheers!

 

- Adam F

 

 

 

 

 

 

 

Exchange 2010 / 2013: Address Book Service, EMC SourceOne and E_MAPI_FAILURES

So I was recently tasked with a pretty interesting case escalation this week, which was in regards to an address book sync failure from a third party application (EMC SourceOne).

Environment

For the sake of this blog, the environment was pretty simple. A single Exchange 2010 SP3UR5 CAS / HUB combination with an Exchange 2010 SP3UR5 Mailbox role. No HA on the DAG and no CAS Array. The EMC SourceOne server is in the same AD site. Multiple global catalog servers and domain controllers utilizing Kerberos.

The Issue

This was a fun one, mainly due to the lack of information provided. What I understood is that the EMC SourceOne application would utilize a local MAPI profile (Control Panel > Mail (32)) to connect to the Exchange role over MAPI / TCP, then start the Address Book Sync through the CAS to the Address Book Service. This is similar to the way RIM BlackBerry Enterprise Server does it (well.. I can speak for BES 4 / 5), and also how both applications do further lookups.

When the EMC SourceOne server would reach out to the CAS’s Address Book Service it would fail out with the error E_MAPI_FAILURE. Not very helpful, since that would be a few hundred things.. lets dive into troubleshooting.

Troubleshooting

The first thing that ran into my head was “is this being throttled”? If you don’t know much about client access throttling, I would recommend reading up on it. It does change a bit in Exchange 2013, but I can write a blog about this later if there is interest. The best way to check throttling is to go directly to the Exchange server, open the Event Viewer: Application logs and filter them for “ADACCESS Event ID 2915”, which is client throttling. You can look at the SID~ within the log to see which account is being throttled..

The Event ID looks something like this:

ID: 2915
Level: Error
Source: MSExchange ADAccess
Message: Process Microsoft.Exchange.RpcClientAccess.Service.exe (PID=5372). User 'Sid~DOMAIN\SERVICEACCOUNT~RCA~false' has gone over budget '263' times for component 'RCA' within a one minute period. Info: 'Policy:[Fallback], Parts:MaxConcurrency:262;'. Threshold value: '100'.

Ironically, NOBODY went other throttling as per the Event Viewer: Application logs this month. Cool, lets move on.

If it’s not client throttling, then what could it be.. it could be NTLM token bloat if NTLM was actually enabled, but they are using Kerberos so that’s not the case. The mailbox does exist for the service account, and the permissions are setup properly. The next logical step is to take a deeper look at the address book service logs (which are located by default at C:\Program Files\Microsoft\Exchange Server\V14\Logging\Address Book Service\)

Looking at the logs using this wonderful tool (TextAnalysisTool.net – really nifty tool everyone should use if you want an easy to use log searching tool) and filtering for the service account (SourceOne-Service) I found the following..

10666 2014-08-20T17:18:04.826Z,165558,1221,/o=COMPANY/ou=First Administrative Group/cn=Recipients/cn=SourceOne-Service,,10.12.4.178,COMPANYMBX1,ncacn_ip_tcp,DNToEph,80004005,0,,Throttled,,15000
10689 2014-08-20T17:18:20.201Z,165558,1222,/o=COMPANY/ou=First Administrative Group/cn=Recipients/cn=SourceOne-Service,,10.12.4.178,COMPANYMBX1,ncacn_ip_tcp,ResolveNames,80004005,0,,Throttled,,15000
10743 2014-08-20T17:18:35.108Z,165558,1223,/o=COMPANY/ou=First Administrative Group/cn=Recipients/cn=SourceOne-Service,,10.12.4.178,COMPANYMBX1,ncacn_ip_tcp,Unbind,,0,60395,,,14906

So what you see above is within the Address Book Service logs that during the address book sync the service account for SourceOne was throttled during an NSPI call to the GAL. Afterwards, that connection was unbound since it went over its budget and sub-sequentially dropped, which would have caused the E_MAPI_FAILURE error mid-sync.

Now that we actually have the error, and know the issue how do we fix this? (thank you for asking kind voice, let me explain that part)

The Fix

This one had me scratching my head at first until I looked at my notes. Originally in Exchange 2010 RTM the Address Book Service would throttle the amount of connections concurrent connections to 50 maximum, and any new connection (when it hit its maximum of 50) would be dropped. This setting use to exist within the exchange-addressbook.service.exe.config file, but was move in Exchange 2010 SP1+ to the registry along to the Client Access throttling policies (RPC Client Access, also known as RCA). What confused me originally is that the CAS does NOT show any ADACCESSS Event ID 2915 errors at all for this account, but I will digress.

To fix this, you should actually create a new throttling policy and either remove the RCA limits or up them to the limit you know you are hitting (I would speak to the vendor to figure out the desired limit, they are the best resource).

1. Create a new throttling policy

The creation of the throttling policy is pretty simple. All you have to do is open Exchange Management Shell (as an account with Org Admin or Org Management rights) and run the following:

New-ThrottlingPolicy "EMC SourceOne"

 2. Once the throttling policy is created, you would need to set the RCA* values within it to something higher than the defaults or simply remove them.

In my scenario, I know the EMC SourceOne application is going to only run the sync over a few minutes in the AM, so I chose to remove the RCA* values completely (e.g: setting them to $NULL)

Set-ThrottlingPolicy "EMC SourceOne" -RCAPercentTimeInAD $null -RCAMaxConcurrency $null -RCAPercentTimeInCAS $null -RCAPercentTimeInMailboxRPC $null

 You then can review the policy to make sure the RCA* values within the throttling policy are shown as blank (meaning there is nothing actually set)

Get-ThrottlingPolicy -Identity "EMC SourceOne" | Select RCA*

3. Apply the throttling policy to the service account

Set-Mailbox -Identity SourceOne-Service@company.com -ThrottlingPolicy "EMC SourceOne"

You can then view the mailbox settings itself to make sure the throttling policy change from the default (or whatever you had previously) to the EMC SourceOne policy..

Get-Mailbox -Identity SourceOne-Service@company.com | Select ThrottlingPolicy

4. Test and review the ABS (Address Book Service) logs. It should not work.

TL;DR: for some reason the address book service throttling does not show up within the actual Event Viewer: Application log under Event ID 2915 when the service account goes over budget. To fix this create a new throttling policy and then up (or remove) the limit for the RPC Client Access (RCA*) attributes. I do not recommend completely removing the limit unless you are sure it will NOT affect production in a negative way, so contact the vendor for there best practices (in this case, EMC Source One would be EMC).

Any other questions, comments or anything like that drop them below in the comments.

Cheers!

- Adam F

Exchange 2010, /PrepareSchema and a Cryptic Error

(note: TL;DR at the bottom in bold)

 

Its 11PM after a long day and I am here trying to rebuild my lab. I sit back, copy over the VHD templates for the VM I need (Exchange 2010 is what I was up too) and got all the prerequisites installed for Exchange 2010. I can get pretty lazy (remember this part, as this will bite me in the rear in a minute) and honestly I find PowerShell to be the lazy mans answer. I used the following to install the required prereqs for Exchange 2010 SP3 on Windows Server 2012 (works for 2012 R2)…

 

Add-WindowsFeature NET-Framework-Features,NET-HTTP-Activation,RPC-over-HTTP-proxy,RSAT-Clustering,Web-Mgmt-Console,WAS-Process-Model,Web-Asp-Net,Web-Basic-Auth,Web-Client-Auth,Web-Digest-Auth,Web-Dir-Browsing,Web-Dyn-Compression,Web-Http-Errors,Web-Http-Logging,Web-Http-Redirect,Web-Http-Tracing,Web-ISAPI-Ext,Web-ISAPI-Filter,Web-Lgcy-Mgmt-Console,Web-Metabase,Web-Net-Ext,Web-Request-Monitor,Web-Server,Web-Static-Content,Web-Windows-Auth,Web-WMI -Restart

Still awake? Good.. lets move on to the actual reason of me writing this..

So I am now at the point of preparing the schema and active directory. As a helpful reminder you will first prepare the schema and then active directory, which has not changed since Exchange 2007. You will also be required to use a user account that is in the following groups:

  • Schema Admins
  • Domain Admins
  • Enterprise Admins

A little piece of hidden advise for users who are doing this in a multi-site production / lab build, is that if you are preparing the schema or extending active directory the schema master FSMO role needs to be in the AD site you are actually preparing.

I download Exchange 2010 SP3 onto my brand new Server 2012 R2 machine which is domain joined, extract the package and run the “Setup.com /PrepareSchema” to get the following error thrown back in my face…

SchemaFailure

 

The snipplet is small due to the size of my monitor but you can click on it to expand it. The error code is also copied below..

 

The following error was generated when "$error.Clear(); install-ExchangeSchema -LdapFileName ($roleInstallPath + "Setup\Data\"+$RoleSchemaPrefix + "schema0.ldf")" was run: "The system could not find the file specific".

*sigh* .. Really? I went ahead and bing’ed and google’ed parts of these errors which came back with nothing. Great.. further examination of the logs (C:\ExchangeSetupLogs\) showed the following..

 

ExchangeSetup.log

[08/11/2014 02:59:42.0175] [1] Executing: 
    install-ExchangeSchema -LdapFileName ($roleInstallPath + "Setup\Data\"+$RoleSchemaPrefix + "schema0.ldf")


[08/11/2014 02:59:42.0175] [2] Active Directory session settings for 'Install-ExchangeSchema' are: View Entire Forest: 'True', Configuration Domain Controller: 'DC-01.extest.com', Preferred Global Catalog: 'DC-01.extest.com', Preferred Domain Controllers: '{ DC-01.extest.com }'
[08/11/2014 02:59:42.0175] [2] Beginning processing install-ExchangeSchema -LdapFileName:'Setup\Data\PostWindows2003_schema0.ldf'
[08/11/2014 02:59:42.0175] [2] Running <C:\Windows\system32\ldifde.exe> with arguments <-i -s "DC-01.extest.com" -f "C:\Windows\Temp\ExchangeSetup\Setup\Data\PostWindows2003_schema0.ldf" -j "C:\Users\administrator.EXTEST\AppData\Local\Temp\1" -c "<SchemaContainerDN>" "CN=Schema,CN=Configuration,DC=extest,DC=com">.
[08/11/2014 02:59:42.0175] [2] [WARNING] An unexpected error has occurred and a Watson dump is being generated: The system cannot find the file specified
[08/11/2014 02:59:42.0175] [2] [ERROR] The system cannot find the file specified

 

ExchangeSetupWatson.log

[08/11/2014 02:58:43.0366] [1] The following 1 error(s) occurred during task execution:
[08/11/2014 02:58:43.0366] [1] 0.  ErrorRecord: The system cannot find the file specified
[08/11/2014 02:58:43.0366] [1] 0.  ErrorRecord: System.ComponentModel.Win32Exception: The system cannot find the file specified
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at Microsoft.Exchange.Management.Deployment.InstallExchangeSchema.RunProcess(String fileName, String arguments, WriteVerboseDelegate writeVerbose)
   at Microsoft.Exchange.Management.Deployment.InstallExchangeSchema.ImportSchemaFile(String schemaMasterServer, String schemaFilePath, String macroName, String macroValue, WriteVerboseDelegate writeVerbose)
   at Microsoft.Exchange.Management.Deployment.InstallExchangeSchema.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.ProcessRecord()

 

Reading the error provided above was a bit much. I know its not permissions since I was using the “Administrator” account as its a brand new domain (as I just created it) and it could not be replication as its a single DC. I went back and then checked the actual prerequisite script I ran, and realized that it did not include the RSAT-ADDS toolset. Because these tools are not installed, and I am not preparing the schema on a domain controller (which has these tools installed by default) then the Exchange installation has no way to actually contact AD Schema to prepare it. Doh!

The simple solution here was to install the RSAT-ADDS module. The PowerShell command for each of these are below…

 

  • Server 2008 / 2008R2: ServerManagerCmd -i RSAT-ADDS
  • Server 2012 / 2012R2: Add-WindowsFeature RSAT-ADDS

 

Once this was done (no reboot is required) I reran the Setup.com /PrepareSchema and it worked without an issue:

 

PS working

 

I really wished there would be a normal error here, instead of outputting a bunch of nonsense that a normal admin, engineer or architect would not realize. Seeing something like that makes you think “oh man, there HAS to be permissions issues or something”, which there is not.. well at least not in this case.

 

Then again, I think MSFT read my mind as I purposely didn’t install the RSAT-ADDS module on my other Exchange server (which is now running Exchange 2013 CU5) and I got the following error:

 

Exchange 2013 error

 

At least they threw something non-cryptic in there :]

 

TL;DR: If you are preparing the schema or active directory on a server that is a non-domain controller, make sure you actually install the RSAT-ADDS module. Its required, and will throw a cool cryptic error at you if you don’t.

 

Hope this helps out more folks out there, and as always if you have any questions feel free to leave them in the comments below.

 

Cheers!

- Adam F

 

Setting up VyOS for Hyper-V 2012 (R2) or Windows 8 (8.1)

So it’s been a while, so as my “comeback” post I decided to make a post on something we all can relate to … creating a home lab. In this first post I will be going over the networking aspect within Hyper-V along with setting up a virtual router (free, glorious virtual router to be exact).

TL;DR: this is long, but a full guide on setting up two sites with routing within Hyper-V using VyOS.

The Setup

The lab setup is pretty simple, but instead of me explaining it pictures do speak a thousand words..

Network Config

The subnets are broken up into three sections:

  •  Home LAN (Lab DMZ) – 192.168.1.0/24 (default gateway on the home LAN is 192.168.1.1)
  •  Newark site (Exchange 2010) – 192.168.2.0/24 (default gateway = 192.168.2.254)
  •  NYC site (Exchange 2013) – 192.168.3.0/24 (default gateway = 192.168.3.254)

For this lab the VyOS router will be connecting to the appropriate vSwitch, which will serve the site directly. Since I want internet access there will be no ACLs or insane NAT rules within the VyOS, which is acting as the core router.

 


 

Different Types of Hyper-V Switches

I am not going to rewrite an excellent blog by John Howard (Sr. PM of the Hyper-V team) so check this out.

What is VyOS, and why is it the best thing since sliced bread?

VyOS is a branch off of the popular open source router Vyatta. Back in April of 2012 Brocade purchased Vyatta and basically killed off the open source version of this. Although the website (vyatta.org) recently went to the old landing page, it now goes directly to the brocade website.

What occurred is most of the devs took what they had / new and created a new project, which is VyOS (vyos.net). Although this is an awesome, linux based open source router I found very little available for setting up / configuring it within Hyper-V with a basic configuration…. which prompted me to create this blog for all of you J

Here are the download links:

Before continuing, make sure you download the ISO for the version of Windows you are running (either 32 or 64 bit).


 

Preparing Hyper-V for VyOS and your networks

We will start off with Hyper-V first so we can ensure the connectivity to each site is setup correct.

  • Open up the Hyper-V Manager MMC (you can search for the term “Hyper-V” and select the Hyper-V Manager in Server 2012, 2012 R2 and Windows 8, 8.1)
  • Within the actions panel (far right) click on “Virtual Switch Manager”

hyper-v_1

When you originally setup Hyper-V you should have bound a physical NIC to a virtual switch. The purpose of this will allow any VM connecting into that virtual switch access to the LAN, and if your network policies / ACL’s match up internet access. It should look like this, but we will discuss how to set this up in the event you didn’t configure it for whatever reason..

To create a vSwitch that will bind to a physical NIC that is plugged into your network..

  • Click on “New virtual network switch”
  • Select “external” and then “Create Virtual Switch”
  • Make sure the items highlighted above are the same

              External: this allows any VM attached to this switch external access, in this case LAN access

              The physical NIC on the LAN is selected

             Allow management operating system to share this network adapter (this bridges off the connection, so both your VM’s and the OS running Hyper-V can utilize a single NIC).

  • NOTE: If you have multiple network adapters connected to your LAN, you do not have to select this option. What this will do is you will dedicate one network adapter for Hyper-V VM’s, and one for the management OS (where Hyper-V is installed).

hyper-v_2

 

Moving on, creating the virtual switches for each site is the same principle…

  • Click on “New virtual network switch”
  • Select “Internal”
  • Select the new virtual switch that was created, and name it something fancy that you will remember
  • Select Apply, and that will create the vSwitch for you
  • Rinse and repeat for the number of vSwitches you would require

 

hyper-v_3

I have four virtual switches created for the following purposes:

  • LAN vSwitch: allows my VyOS router access to the LAN to provide internet connectivity
  • Exchange 2013 vSwitch: this vSwitch connects the VyOS Eth1 port and the NYC site VM’s together, allowing internet connectivity and routing to the Newark site
  • Exchange 2010 vSwitch: this vSwitch connects the VyOS Eth2 port and the Newark site VM’s together, allowing internet connectivity and routing to the NYC site
  • Exchange 2010 Repl vSwitch (OPTIONAL): this is for the replication network in Exchange 2010, and is a PRIVATE vSwitch

Lets move onto the configuration of the VyOS image for Hyper-V..


 

Create and Configure the VM:

  • In the “Actions” pane, click on New > Virtual Machine…

Vyos1

  • Name your VM, and choose the location. I have change the VM default locations, so I am keeping what I have BUT you can click the checkbox below the name, and select a different VM path.

vyos-2

  • Keep the assigned memory default, which is 512mb (it’s a router, it doesn’t need that much). Click next afterwards.

vyos-3

  • Under the networking section select the LAN vSwitch. We will configure internet access / lan access for your core router first and then move on from there. Once this is selected click next.

vyos-5

  • The next section has a few options..

o   Name your VM, make it something you will remember. This will be the name of your VHDx file, so make sure it’s something you will remember and recognize.

o   Select the location if the default doesn’t work for you

o   Change the size to 5GB. That is more than enough for this VM, as when I fully configured / installed it is only taking up 1.2GB of storage.

o   Click Next once you have the settings above the way you want them

  • On the “Installation Options” section, you need to select the ISO you downloaded above. Click on “Install an operating system from a bootable CD/DVD ROM” and select “Image File (.ISO)” then Browse.. select your VyOS ISO file and then select Finish.
  • Start your VM by double clicking it which will bring up the console. Click on the “Start” button to boot up the VM and don’t click on anything until you get to the logon prompt.

 


 

Installing VyOS on your router

  • Once the VM boots, login. The username and password are both vyos
  • Start the image installation by typing in Install Image and then press Enter
  • It will ask you if you want to install this on the local hard drive, press Enter
  • It will then ask you if you want to create a RAID, type N and select Enter
  • It will ask you to about partitioning, select A for Auto and then select Enter
  • Next will be the partition in which it is installed. By default it should be SDA. If [SDA] is listed, press enter. If not, type in SDA and press enter.
  • Formatting of the VHDx file will happen next, type Y and press Enter
  • The size of the root partition is next, and simply press enter to select the full partition
  • You need to name the router. Name it something you will remember and like (I named mine “Core_Router_Lab”)
  • Next will ask for the location of the config.boot. The default location is fine, so press Enter
  • Select the partition for GRUB to be installed. Default location is fine, so press Enter
  • You should get no errors and then installer exits.
  •  Next you will turn off the router and eject the ISO file, so do the following:

o   Type the command poweroff and press enter
o   When the machine is fully off, in the same hyper-v console you have open for the VM click on the media > DVD drive > eject
o   Boot the machine up and do not touch anything until you are at the logon screen


 

Configuring the VyOS Router VM for your networks

FINALLY! The meat and potatoes of what we want to do!

As you remember above, we have two AD sites within our lab under a single forest called Contoso.com:

  • AD Site: NYC

o   Subnet 192.168.3.0/24

  • AD Site: Newark

o   Subnet: 192.168.2.0/24

We also have our LAN (192.168.1.0/24) which is our internet access point. Since we created a vSwitch for each subnet / site above, we are going to configure the VyOS router to be the default gateway for each of my site subnets listed above.

The first thing we are going to do is configure the VyOS Eth0 port, as that is my LAN port which will provide internet access.

  • Log into the VyOS device using the username vyos and the password you configured during the setup
  • View the interfaces attached to confirm that you see eth0, which should be on the LAN vSwitch

Show interface

  • Once you have confirmed that the eth0 interface is online, lets configure it. Enter into configuration mode

Configuration

  • Once in configuration mode, you will setup the IP address and subnet of the interface. Since the LAN network segment is 192.168.1.0/24, I will make my IP address 192.168.1.254 (it’s a hop in the network, so we need an address assigned to it).

Set interfaces Ethernet eth0 address 192.168.1.254/24

  • We should also setup the default gateway for the LAN network, which is 192.168.1.1

Set system gateway-address 192.168.1.1

  • Oh yea, dont forget about the name server

Set system name-server 192.168.1.1

  • PROTIP: you need to commit and save the running config to the router. Changes will not take place until you commit them, and will not save to the vram dedicated within the VHDX file until you save them. NEVER turn off the router, and do not exit configuration mode without committing the changes

Commit

Save

  • Once the configuration is committed and saved, lets exit configuration mode and take a look:

Exit

Show int

  • When you run the show int, you should see the following:

Eth0 – IP Address: 192.168.1.254/24

There you have it. Depending on the way your LAN router is, you may need to throw routes in there to make sure you can route out but now the VyOS routers eth0 is setup for the LAN connections.

How about the other sites? Easy…

  • Shutdown the VyOS router by typing out the following:

Shutoff

Once the VM is fully offline, right click it within the Hyper-V Manager MMC and select “settings”

  • Select “Add Hardware” and select “Network Adapter”
  • Select the network adapter you added, and chose the vSwitch you wish to connect to it. Click apply, close out the Settings window and start your VyOS VM up again.
  • Within the console itself, configure the new Ethernet interface for the proper subnet.
  • Repeat as many times as needed.

 

That’s it. When you now add VM’s to your lab, you will attach them to the vSwitch that is appropriate for the subnet / site. The default gateway will be the Ethernet address you assigned within VyOS (for example, the NYC site is Eth2 on my VyOS router. I have it setup as 192.168.3.254/24 so the default gateway is 192.168.3.254).

 

 

 

 

 

 

 

 

 

 

How to solve ESE -566 Errors (JET_errDbTimeTooOld)

Hello all,

After having this issue at a customer, and finding the information on the internet to be a bit old (and haste) I wanted to post what occurred and how to solve it.

So over the weekend I had DR testing at a customer. They are running Exchange 2007 SP3RU8 on all servers, and for HA they have a two node Single Copy Cluster.

We did some maintenance on the cluster, and then attempted to fail over. After waiting about 3 minutes I decided to check (as a healthy cluster should fail over within 60-90 seconds). Afterwards, I saw three DB that refused to mount.

I reviewed the Event Viewer: System logs and saw this..

Event Type: Warning
Event Source: PartMgr
Event Category: None
Event ID: 59
Date:
Time:
User:
Computer: SCC-Node2
Description:
Disk 75 will not be used because it is a redundant path for disk 0.

Not really a warm fuzzy feeling, so I decided to fail back over to SCC-Node1. When that occurred I attempted to mount both of the databases that refused to mount (within Cluster Admin utility by bringing the IS Instance online) and failed. Once I looked within the Event Viewer: Application logs I saw a slew of these error messages:

Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 516
Date:
Time:
User: N/A
Computer: SCC-Node1
Description:
MSExchangeIS (6312) StorageGroup1: Database F:\database1\database1.edb: Page 231 (0x000000e7) failed verification due to a timestamp mismatch. The expected timestamp was 0xbedab929 but the actual timestamp on the page was 0xbedab403. Recovery/restore will fail with error -566. If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware “losing” one or more flushes on this page sometime in the past. Please contact your hardware vendor for further assistance diagnosing the problem.

and then a few of these:

Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 454
Date:
Time:
User: N/A
Computer: SCC-Node1
Description:
MSExchangeIS (4408) StorageGroup1: Database recovery/restore failed with unexpected error -566.

The first thing I checked was the health of the actual DB. To do this you should run the following:

  1. Open the Command Prompt
  2. Change directory to %programfiles%\Microsoft\Exchange Server\Bin
  3. Run the ESEUTIL command (eseutil /mh <DriveLetter>:\Path_To_DB)

This command dumps the DB headers, and will give you two pieces of critical information you are looking for:

  • State
  • Log Required

Here is the output from the command I ran:

Initiating FILE DUMP mode…
Database: D:\Database1\Database1.edb

File Type: Database
Format ulMagic: 0x89abcdef
Engine ulMagic: 0x89abcdef
Format ulVersion: 0x620,12
Engine ulVersion: 0x620,12
Created ulVersion: 0x620,12
DB Signature: Create time:09/15/2010 16:58:36 Rand:2592704934 Computer:
cbDbPage: 8192
dbtime: 3888704209 (0xe7c8ead1)
State: Dirty Shutdown
Log Required: 1829641-1829643 (0x1beb09-0x1beb0b)

At this point, I know I am going to need to do some type of recovery that is going to require the logs in the “Log Required” section.

Before I continue, lets discuss what actually happened here based off what I see from my troubleshooting..

What would cause this: 

Before fully understanding why this occurred  you should understand how dbTime works and how the Information Store works. There is a good article here about the Information Store process, but to sum it up there is a Log Buffer that is within the memory. To help reduce I/O on your disk subsystem, transactions are written to the Log Buffer. The Log Buffer (being 1MB in size, since transaction logs are that size) then commits to a log when one of two things happen:

  1. The DB is failing over, and needs to commit the transaction to memory. This could be due to node failure, turning the server off or HA testing
  2. A hard transaction hits the DB (send / receive of email, ect)

When the log buffer needs to write, it writes using the Log Writer process.

To summarize, transactions are not written to the DB immediately but instead to the log buffer and then from the log buffer written from the log buffer utilizing the log writer to the DB.

As for the dbTime, when a new log is generated a unique database time is generated based off the log generation time. The dbTime can be higher than the actual current log time, which is normal. The dbTime is utilized more as a counter than an actual time stamp.

Why does this occur? When we perform DR (hard / soft recoveries) on a DB, the dbTime is examined to see if the log has been replayed or not.

Now to the actual root cause..

What it looked like was that when SCC-Node1 failed over to SCC-Node2, the disk subsystem was overwhelmed along with the network backbone for the SAN, thus causing a mismatch in the time. If you look at Error -566 utilizing the Exchange Error Tool you would see that this theory does match up to the actual error:

JET_errDbTimeTooOld esent98.h
# /* dbtime on page smaller than dbtimeBefore in record */

The question now is how to fix it..

The Fix: 

Although the article above states you should run “ESEUTIL /P” I highly disagree with this. ESEUTIL /P should be ran only as a last resort. I was able to fix this quite quickly using ESEUTIL /R.

From our ESEUTIL /MH content above we know the DB is in a dirty shutdown state, and we wont be able to mount this until it is in a clean state. We see that he log required was 1829641-1829643 (0x1beb09-0x1beb0b). If you look at the log folder, you might be able to find these logs (they should exist – they were created, but the dbTime is “too old”). Look at the first three letters that start off on the log file. In my situation was E070. Remember that. 

Lets go back to our command prompt that is already in the /Exchange Server/BIN folder. From there we would run the following command:

ESEUTIL /R E07 /l<driveletter>:\Log_Location

The reason I am selecting to replay the logs starting with E07 is because the formatting of the command (ESEUTIL /R Enn – Enn = the base log that needs to be replayed). Because I know Log 0x1beb09 = E0700001beb09, I should replay the logs for E07.

Monitor the process as it runs..

Initiating RECOVERY mode…

Logfile base name: E07

Log files: L:\LogLocation

System files: <current directory>

Performing soft recovery…

Restore Status (% complete)

0    10   20   30   40   50   60   70   80   90  100

|—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|

……………………………………………

Once this is completed you should have no errors. Check out the status of your DB utilizing ESEUTIL /MH. The db should show a clean shutdown state with 0x0 logs required (meaning there are no logs that are required for the db to replay):

Initiating FILE DUMP mode…

Database: F:\NAData21\NAMailboxDatabase21.edb

File Type: Database

Format ulMagic: 0x89abcdef

Engine ulMagic: 0x89abcdef

Format ulVersion: 0x620,12

Engine ulVersion: 0x620,12

Created ulVersion: 0x620,12

DB Signature: Create time:09/15/2010 16:58:36 Rand:2592704934 Computer:

cbDbPage: 8192

dbtime: 4104664568 (0xf4a835f8)

State: Clean Shutdown

Log Required: 0-0 (0x0-0x0)

Once you have also replayed the logs, the dbTime log sequencing is also updated so that logs will be stamped once the DB is mounted.

Attempt to mount your database again and you should be good.

Any questions leave them below in the comments section.

(and for the Redditors who read this)

TL;DR: Don’t run ESEUTIL /P as your first step! /P = Panic = :( data loss! Run ESEUTIL /MH to get the status of the DB and then run ESEUTIL /R if you are able to find the logs / restore the logs.

-Adam F

Troubleshooting Intra-Org SMTP Traffic Issues (and disabling Cisco ASA ESMTP Inspection)

I had a particular issue that is not well documented on the Technet site so I decided I would blog about it and share my experience.

Today I was doing an Exchange 2003 to 2010 upgrade for a customer. Their AD setup was quite typical, two Active Directory sites under a single forest interconnected by a Cisco ASA in each site doing IPSEC. They wanted an Active / Active configuration (one site will be doing SMTP and CAS traffic, which is NY and the FL site will be doing just CAS traffic) which was fine. Here comes the problems.

We tested mail flow from the NY site both external (inbound/outbound) to a user in the NY site along with the Exchange 2003 setup. All worked without a problem. When we tried to send an internal or external message to a user within the FL site, it would get stuck in the messaging queue with the status of “Retry”.

The first thing when troubleshooting intra org SMTP traffic is to review the Receive connectors. By default, you would have two…

  • Client <servername>
  • Default <servername>

The default receive connector is used for SMTP traffic, and is listening over TCP 25. The client receive connector is used for ESMTP, as it is listening over TCP 587.

You should look at several things:

1)      What are the subnets allowed within the Network’s tab on my Default <servername> connector? By default, you should be allowing all subnets to utilize this connector to send message to this server.

Image

2)      What is my authentication set to for this receive connector? I have seen this one a lot, and where people think it’s an awesome idea to remove all types of authentication. Its not, don’t do it. By default is what you see below, and should be kept this way. If you need to change this, I would recommend making a new SMTP connector with the values desired

Image

3)      Who is actually allowed to use this connector? The values for the defaults are below (Exchange 2010 SP3) but should be kept alone. If you want to select “Anonymous” I would recommend setting up a new Receive connector and setting up the network section to allow only the servers that would need to relay off of this server. This prevents your Exchange 2010 servers from being a generic SMTP relay out in the interwebs for anyone to use and abuse.

Image

Since we are troubleshooting Exchange 2010 intra-org SMTP connectivity, we want to make sure that we keep the Exchange servers checked within the Authentication and Permissions group tabs. If this is unchecked, check it off and restart the “Microsoft Exchange Transport Service” (anything changed in here should have a restart of this service).

If this is fine, we should move onto actually checking if port 25 is available between the two sites. You can check this by running the following from the command line..

telnet <Name of the HUB server in the other site) 25

The response should look similar to below

Image

If we are able to telnet to this without a problem, then check out the other ports listed here. You should have these also open and available to for intra org SMTP communications and also for the HUB Transport services to work properly.

The best troubleshooting in my opinion would be to turn up the IntraOrgProtocolLoggingLevel to Verbose. This should be done on all HUB transport servers when troubleshooting, and can be done like this:

      Get-TransportServer | Set-TransportServer –IntraOrgProtocolLoggingLevel Verbose

Once this is done, attempt to send another test message. Give it about 5 minutes and then check the logs from the sending HUB transport server. These logs can be located (by default) at %PROGRAMFILES%\Microsoft\Exchange Server\V14\TransportRoles\Logs\ProtocolLogs\SmtpSend

From there review the errors. There are usually typically a handful that can be there, but the one distinct one I saw today was “cannot achieve Exchange server authentication – failed”.

When reviewing further I noticed the following error within the queue viewer:

“451 5.7.3 – Cannot achieve Exchange Server authentication”

The Cisco ASA does protocol level filtering, the 250 STARTTLS reply is not established and dropped. To get IntraOrg SMTP Traffic to work, we would need to turn off the ESMTP inspection which can be done as shown below (thank you Lazy Network Admin for these steps below)

      CiscoASA# config t 
      CiscoASA(config)# policy-map global_policy 
      CiscoASA(config-pmap)# class inspection_default 
      CiscoASA(config-pmap-c)# no inspect esmtp 
      CiscoASA(config-pmap-c)# exit 
      CiscoASA(config-pmap)# exit 
      CiscoASA(config)# exit 
      CiscoASA# wr me

Once this is done retest, and you should be receiving emails from the other site. If you did turn on the intra organization protocol logging, I would recommend turning it off.

       Get-TransportServer | Set-TransportServer –IntraOrgProtocolLoggingLevel None


Now, these are not the only troubleshooting steps in the world. If you continue to have issues with intra-organization mail delivery I would also check within the Exchange Toolbox the routing configuration, and run through the Mail Flow Troubleshooting wizard.

Hope this is helpful to all! Any questions please let me know below.

Adam F

Exchange Proxy and Redirection (Exchange 2007 and 2010) Explained

There has been a lot of discussion recently around proxy/redirection with Exchange 2007/2010 so I wanted to clear the air about what each one does, and how they work with CAS services.

What is the real definition of Proxying and Redirection?

To understand proxying and redirection, you should understand the meaning of both. Later in this article I will cover when/why proxy or redirection happens.

Proxying occurs when a CAS role sends traffic to another CAS role in particular situations (read below for more about that). For example here are two common situations:

  • CAS to CAS communication between two AD sites
  • CAS to CAS communication between Exchange 2010 and 2007 or 2003

Redirection occurs when there are two AD sites with Exchange 2010 / 2007 / 2003 and both are internet facing.

When do I proxy or redirect a CAS connection?

This is a little more complex. The following CAS protocols / services are proxy enabled:

  • Outlook Web App (OWA) and Exchange Control Panel (ECP)
  • Exchange ActiveSync (EAS)
  • Exchange Web Services (EWS) and the availability service (part of EWS)
  • POP3 / IMAP

And then the only three services that will actually redirect are OWA/ECP and EAS.

On to the fun part..

When do I actually proxy my connections from one AD site to another?

Simple, and we will use OWA as the example.

Adam’s mailbox is in the EU AD site, which has an ExternalURL set to $NULL (no value) for OWA but the InternalURL is set to euemail.exchangelaboratory.com. My internet facing site is the US AD Site, and we have an ExternalURL/InternalURL of naemail.exchangelaboratory.com.

Since OWA does have the capability of Proxy and Redirection, when the initial authentication request goes to the UAG or CAS (depending on your deployment) exchange is looking for three items from the GCS in a LDAP/RPC request, which is

  • My homeMDB (mailbox location)
  • My msExchangeVersion (the version of Exchange the user mailbox is on)
  • Authentication Token to fulfill my request

If it notices that my homeMDB is in another site, then it will query that CAS for the InternalURL and ExternalURL of the services in question (in this case, OWA). Since the ExternalURL is set to $NULL in the EU AD Site CAS, then the only option we have will be to proxy.

What occurs at this point is that the request gets proxied from the source server (cashub1.exchangelaboratory.com) to the target server (cashub2.exchangelaboratory.com), the CASHUB2 server will contact the Mailbox server for the appropriate data from the user’s mailbox and then proxy the connection back to the source server, which then delivers the content to the user. Beautiful thing.

Image

Now for redirection:

In this situation, we now have an “Active / Active” configuration as in both the US AD Site and the EU AD Site are internet enabled. My ExternalURL for OWA in the EU AD Site is euemail.exchangelaboratory.com, and my ExternalURL for OWA in the US AD Site is naemail.exchangelaboratory.com. We will use Adam again as our example mailbox.

When Adam attempts to authenticate to the URL naemail.exchangelaboratory.com the Exchange CAS (or UAG depending on your configuration) will perform the auth request along with the Mailbox Location and Version lookup. Exchange will realize that Adam’s mailbox is actually in the EU AD Site, and will send a HTTP 451 request back to the client to redirect to the proper internet facing URL (in this situation, euemail.exchangelaboratory.com). Once this is complete, it will either be a silent redirection or the user will have to authenticate again.

Image

ActiveSync and OWA Virtual Directory permissions to make redirection and proxying to work?

To make sure we can successfully redirect Exchange ActiveSync and OWA requests, we need to ensure that the proper permissions are setup on the IIS Virtual Directories.

Exchange 2003 (non-internet facing) and Exchange 2010:

In this situation, our endpoint for client access is Exchange 2010 as this is internet facing and the most up to date version of Exchange. The Exchange 2003 site is not internet facing, so the Exchange ActiveSync protocol would have to proxy from Exchange 2010 to 2003 for any users within the Exchange 2003 environment.

To allow this to work, you should enable IWA (Integrated Windows Auth) on the IIS Virtual Directory Microsoft-Server-ActiveSync on the Exchange 2003 server. This should not be done through the IIS Manager, as the System Attendant in Exchange 2003 will overwrite these changes and break proxying. You can do this through the steps available here.

Exchange 2007 (non-internet facing) to Exchange 2007

In this situation, we have two AD sites both running Exchange 2007. One site is internet facing and the other is not, thus if a user is within the non-internet facing site we have to proxy the connection from the internet facing CAS to the non-internet facing CAS.

To allow this to work we need to enable IWA on the non-internet facing Exchange 2007 CAS. To do this we will need to go through IIS Manager, and the steps can be found here.

Single Sign On for OWA – Exchange 2010 SP2?

A new feature the Exchange Team at Microsoft implemented in Exchange 2010 is called silent redirection. I would recommend setting this up, the reason for this is because you now eliminate the need for a double authentication when doing a redirection within OWA on Exchange 2010 SP2.

This does get a little lengthy, and there is an excellent blog from the Exchange Product Group located here.

 

Well, that’s all I have for now. Leave your questions below if you have any. Thanks for reading!

Adam F