CUCM 8.0.3 to 10.5.2 Migration thru PCD 10.5.3

Scenario:
Cluster with 5 servers running on MCS. Trying to migrate to 10.5 virtual environment via PCD. Doing a parallel migration.
- CUCM PUB
- CUCM SUB1
- CUCM SUB2
- CUCM SUB3
- CUCM SUB4
- IM&P

Problem Summary:
Migration task of CUCM 8.0.3 to 10.5.2 thru PCD is interrupted with a Critical Error on CUCM PUB (see below)



Cause:
- Orphaned Phone Button Template

Solution:
- Generic Solution: Follow the dump file collection steps shown below and determine the config that's causing the conflict and delete that item from your current cluster and re-run the PCD Migration task.

See detailed steps below...

Attempt 1: Day 1
OVAs & Images used:
CUCM OVA: cucm_10.5_vmv8_v1.8.ova
CUCM ISO: Bootable_UCSInstall_UCOS_10.5.2.10000-5.sgn.iso
CUP OVA: cucm_im_p_10.5_vmv8_v1.0.ova
CUP ISO: Bootable_UCSInstall_CUP_10.5.2.21900-4.sgn.sgn.iso

- Discovered current cluster
- Defined Migration Task by selecting the discovered cluster and creating the New Cluster with new Hostnames & IP addresses. PCD supposedly makes it 'easier' to migrate with hostname and IP address change.
- Executed Migration task
- First task in the list of 12 tasks went thru fine - which is to take a backup of all the servers in the cluster - took around 2hrs (consider WAN data transfer).
- Task 2 in the list was to install the new PUB and restore the data from current PUB (that was done part of Task 1)
- Installation failed with Critical error...
- Meanwhile PCD will show status of 'Running' for about 5hrs and then come up with a 'Failed' status (refer Bug: CSCu000851)

- Encountered an issue: VM was inadverdently moved from the ESXi Host as the rule to keep on the destined host was deleted by mistake.

Attempt 2: Day 2
- Deleted CUCM. Recreated CUCM VM and restarted Migration task
- Failed again with same error
- Tried to capture dump file to serial, but was returning 0 size file.

Attempt 3: Day 2
- DNS resolution wasn't done. Fixed that. (A & PTR records working now).
- Deleted CUCM VM and recreated it
- Executed Migration again
- Failed again - same error

Attempt 4: Day 3
- Tried with new image: 10.5(2)SU1: Bootable_UCSInstall_UCOS_10.5.2.11900-3.sgn.iso
- Also tried something different this time. Tried to edit the <VM>.vmx file, find 'ethernet0.virtualDev' (should be on line 42) and move it to the end of the file.. follow steps below or just open it in a text editor and cut (remove blank line) and paste at the bottom of the file..

Before

After


   - 
    cucm.ova.README.txt

- Deleted CUCM VM and recreated it
- Executed Migration again
- Failed again - same error

Attempt 5: Day 4
- Apparently there is a certain procedure to follow to dump the log files (see below). Followed these steps and got a 25Mb file.

- The log file has no extension. After you download it to your computer, rename it to .tar
- Open this using WinRAR and you will see a multiple folders and files..
- Navigate to...
\var\log\active\cm\trace\dbl\sdi\installdb_ru.log.err

- Found the below...

- Based on this.. went to the current CUCM and found this...note the hex value: 4ee12cca-....2110 in the URL - got that from the log file. See above

- The orphaned Phone Button Template for Android phones (which actually did have any dependencies) was causing the issue.
- Deleted the template as no device was attached to it

- Recreated CUCM VM (left network adapter setting as is, didn't modify <VM>.vmx file
- Recreated new migration task 
- Executed Migration task
- Voila!!! that worked.. 
- PCD has moved to the next Task: IM&P Install & Restore