Why Do We Need an Upgrade?
The wise Greek philosopher, Heraclitus, once said, “Change is the only constant in life”. This law of life could not be truer in the case of ever-changing software lifecycles. Software programs perennially evolve to serve businesses in the best capacity, thus making a case for enterprises to upgrade and adapt to the latest and greatest version of the software available, helping survive business exigencies. Software applications age faster than your car. After all, age catches up.
In this data-driven world, as the volume of data every business deals with keeps increasing, software programs catering to associated domains like Data Integration, Data Analytics, and Master Data Management have gone through major revamps and enhancements. When was the last time a business upgraded their software stack just because they had a few million dollars to spend? Every upgrade happens for a reason, and many times, more than one. The reasons may vary across businesses, however, some of the common triggers are:
- Applications out of lifecycle support
- Moving from on-premise to cloud
- Latest is always the greatest (in terms of features, performance, etc.)
- Performance degradation
- Infrastructure alignment
Mastech InfoTrellis – a pioneer in the Master Data Management solutions space – enables our esteemed customers around the world and across industries – Banking & Financial Services, Insurance, Manufacturing, Life Sciences, and Education – to reap the benefits of Master Data Management software upgrades by maneuvering some of the complex upgrades the industry has ever endeavored.
In this blog, we discuss the key areas of focus for upgrading IBM MDM Advance Edition (AE).
Key Focus Areas
Assessment and Discovery Phase
The upgrade journey kicks off with an Assessment and Discovery phase that sets the tone for the long haul. Remember, nothing is bigger than the business goal. First and foremost, the assessment phase should give a sense of confirmation that the business goal will be achieved by the upgrade. Key things to assess and gauge during this phase are business drivers, systems landscape, current pain points, integration patterns, security models, software stacks, key dependencies, and potential risks. And, remember to glean as much information as possible and document it all.
Artifacts are Assets
The following artifacts need to be consolidated in accordance with the existing IBM MDM AE system:
- Design Documents – For systems involving a large number of services with heavy customizations, having the design documents in place makes the job of understanding MDM customizations in the system easier
- Architecture – High-level and low-level
- Infrastructure Design
- Security Patterns
- Integration Design – All connected systems and integration patterns (real-time, near real-time, or batch), volume, frequency
- DB Scripts – If the custom DB scripts are not up to date, it becomes a tedious task to analyze the database and come up with all the custom DB scripts for the upgrade development
- Test Suites – This will form the regression suite to verify the correctness of the upgraded system
- Issue Log – During the development phase of the upgrade, we often find some questionable behavior in the current system, which could potentially be unidentified bugs. It is important to have a list of known issues from the existing system, and during the initial assessment, a prioritization and resolution for the known issues need to be taken.
Before moving to a point of no return, performing an upgrade ensures that everything is backed up to roll back to an earlier state if things don’t go as planned. After all, even the best plan going awry is not something unheard of in software upgrades.
- Database, DB Configurations
- MDM Server Application Configurations
- Source Code
- Custom Jars, EBAs
- Application Server Profiles, Configurations
In our MDM journey, infrastructure often has brought up the most interesting challenges of MDM software upgrades. Remember to consider the following key areas while planning and setting up infrastructure for the newest IBM MDM AE version:
- The volume of Master Data to be ingested to the upgraded MDM system
- Latency and throughput expected from the business-critical systems consuming Master Data
- Performance improvements over the existing system
- Compatibility of existing system hardware with the prerequisite system requirements for latest MDM. Some shall be repurposed while obsolete ones must be replaced.
- Design an infrastructure architecture that doesn’t just handle planned volumes and provides SLA-mandated throughput and latency, but also accommodates planned growth over time
- It’s always a better practice to design HA-enabled, load-balanced, low-latency application servers with failover support across multiple zones. For DB servers, again, go with an HA- and DR-enabled architecture.
Infrastructure Upgrade Steps
Step 1: MDM DB upgrade
The first step for the MDM DB upgrade is to opt between the following two traditional styles:
- In-place Upgrade
- Side-by-side Upgrade
Organizations short of resources that need to host multiple database environments usually opt for this style, where an existing DB instance is shut down, upgraded, and brought back to life.
Side-by-side is a relatively safer route, as the existing DB instances are left untouched till the latest version is ready to serve business-critical systems. The figure below outlines scenarios when a particular approach is preferred over the other.
While the developer machine can be a fresh DB installation of the latest version, where we only have to port the custom scripts, the lower and higher environments are recommended to use the existing MDM DB and upgrade it to the required version, helping save the time for data migration.
MDM DB Upgrade Approaches
In case of upgrading the MDM DB, there are two approaches to be debated and decided – a manual versus an automatic upgrade.
The manual upgrade involves running a series of version-specific scripts for upgrading the MDM database. It is done incrementally, starting with the earliest version and ending with the latest version. The automatic upgrade uses the upgrade_mdm madconfig script, which simplifies the database upgrade process through automation. Although a manual upgrade takes more time, it is preferable as it is easy to identify errors at each stage, if any. Below are the pros and cons of each approach:
Some other items to be taken note of include:
- Check the compatibility of the character encoding of the older DB version with the required DB version. For example, for MDM 11.5 expect the database encoding to be Al32UTF8.
- If the older DB has custom indexes for performance reasons, make sure to port them to the new DB.
- If there any non-English characters to be inserted in code tables, make sure to find the appropriate NLS_LANG for the language.
Step 2: Application Server Installation
Once the DB is upgraded, the next step is to install WAS. While the recommendation for a DB upgrade is to migrate the existing DB, for the application server and MDM, we do fresh installations. After installation:
- Migrate all the security-related settings
- Create data sources for the upgraded DB and verify connectivity
- Configure additional items like queues, topics, etc.
Step 3: MDM Installation
If steps 1 and 2 are carried out successfully, then this step should be a straightforward task.
Application Migration Recommendations
Application migration is a major activity that includes the following tasks.
- Migrating the code base to the latest MDM workbench
- Resolving compiler errors
- Running the custom DB scripts on the upgraded DB
- Moving property JAR customizations
- Deploying the new CBA on the server
Application Migration Recommendations
The following are some recommendations we would make for a smooth passage in this phase.
- Never club upgrades with enhancements
- OTS is turned on by default in the latest MDM. To get the extension fields and “admincontequiv” fields in the inquiry response, either turn off the OTS or update the inquiry level queries.
- Never package functional changes along with the MDM upgrade like moving from side-table extension to inline extension or moving from DME to PME.
- Prefer the OOTB capabilities in newest MDM over the custom-built ones from earlier MDM offering similar capability.
- For integrations of the MDM system with other connected systems in the organization, prefer JAX-WS or JAX-RS over deprecated RMI or JAX-RPC
Code Generation Decisions:
- Although the MDM workbench provides a workspace migration feature, it is not a good idea to import the existing project into the new workspace. The ideal approach should be to regenerate the data model (Additions/Extensions/Code Tables) in the latest workbench and generate the code.
- Extensions to entities like privacy preference, the LOB relationship has to be handled manually as the latest MDM workbench doesn’t support the same
- Move any logical extensions from older implementation to the new XSD
- Register the new blueprint with all the business proxies, behavior extensions, external rules, logical extensions, and standardizers from the older implementation
- Ensure that the newly generated Component IDs, Error Reason IDs, and Error Message IDs are synced with the older ones.
- Compare and fix discrepancies observed with DB scripts generated from the latest workbench against custom scripts from the older system
- Execute the custom DB scripts in a fresh DB installation to understand if there any conflicts. Once conflicts are resolved, execute the same in the upgraded DB.
- After the custom DB scripts are executed on the upgraded DB, do a complete export of table data of the older and upgraded DB for comparison. This is important as there could be a discrepancy in the OOTB values of some tables.
- For e.g., the Name column of CDCOUNTRYTP differs between v11.0 and v11.5, the Severity column in ERRREASON differs between v9.0 and v11.0.
- Identify deprecated MDM classes and methods and substitute those with the latest API. For e.g., TCRMCodeTableHelper in v9.0 is deprecated in v11.0 and is replaced with CodeTypeComponentHelper.
- Remove cyclic dependencies. Always keep the workbench generated code and custom code in separate projects.
- Ensure that the Manifest file imports and exports are in place.
- Resolve any access restriction issues with jars
- Recompile all third-party jars to be compatible with latest JDK version supported by MDM
The final leg of the upgrade process is to test the upgraded application. The test suite consolidated from the older implementation is first executed in the developer system that has a fresh DB. This will help to isolate code level problems from DB migration or application deployment problems in the upgraded environments. The test suit is then executed in both the older and the upgraded environments. The results from both the environments have to be compared to ensure that the functionalities are intact.
And finally, run your performance test case suites to verify if the optimal performance is accomplished by the upgraded MDM application. Get into further profiling and employing optimizing techniques to get the desired throughput.
Mastech InfoTrellis, from our rich experience with master data management services and software programs, is aware that even the best-laid plans often go awry. No matter how meticulously planned and executed an upgrade is, it’s when the rubber meets the road we realize how successful it is. From our rich experience of complex upgrades, we always advocate a soft go-live before the brand-new MDM is operational and starts serving a business in its fullest capacity.
A preliminary round of health, performance, and integration checks has to be carried out to verify if the data from MDM can be consumed by the integrated systems and vice-versa. If there are concerns with the format of inflow or outflow data, then in the Integration Layer, necessary transformations need to be carried out.
Sr. Technical Consultant, MDM – Mastech InfoTrellis
Sr. Technical Consultant, MDM – Mastech InfoTrellis