I have a strong belief that Information Technology Operations is one of the most critical departments of an organization. In a large and multifarious organization like the Canadian Tire Corporation, Information and the systems that this information flows through act as a nervous system of a company. These heterogeneous systems comprise a complex umbrella of various technologies and products. It is the job of an I.T operations team to ensure that these systems work seamlessly, and are always operational. It is no exaggeration to say that the operations team guarantees customers receive continuous value from the services provided by our firm.
Within this vast and intricate world of I.T Operations, I belong to a team that is responsible for making sure that the TIBCO infrastructure runs smoothly and providing quick resolution of production issues that may impact our real-time services. Resolution of these issues require a deep understanding of the underlying systems that constitute our TIBCO environment. It also requires a holistic understanding of various other technology products and areas that comprise an integrated I.T ecosystem.
Drawing from over a decade of experience working in TIBCO operations at various organizations, I have come to form a conclusive opinion that the field of operations is tightly integrated with the architectural design on top of which the operations run. To support and troubleshoot the operational problems, it is imperative to learn and be thorough with how the underlying system is structured. In this regard, I would like to present my case as to how Co-Pilot has played a transformative role in bridging the gap of my understanding between TIBCO Operations and TIBCO Architecture. It has fundamentally changed the way I approach the complex incidents that I encounter in managing and maintaining this Middleware infrastructure of which TIBCO is a component.
Within the broader net of TIBCO operations and TIBCO Architecture, I would like to put forward two interconnected examples that demonstrate a consolidated view of how Microsoft CoPilot acted as an enabler in transforming my comprehension and implementation of day to day work here at DiT team within Canadian Tire.
Case Study 1 – Understanding the new TIBCO EMS and TIBCO FTL design: Enterprise Messaging Service (EMS) is the messaging backbone of our environment which enables applications to communicate with each other via XML-based messages. These XML based messages contain of the actual payload or business data that the applications process. This workflow is directly linked to the customer-facing services provided across various Canadian Tire Corporation (CTC) channels. The EMS server utilizes destinations like queues and topics to manage messages generated by producer applications and retrieved by consumer applications (primarily BusinessWorks or BW applications in our environment). In our legacy systems, we have been using a standalone EMS system in our production environment running on version 8.4. This legacy configuration does not employ an Active-Standby architectural design, which would otherwise ensure that if the primary EMS crashes, a standby server takes over with minimal to no disruption.
The design/architecture of EMS / FTL drastically revamped when our entire TIBCO based ecosystem was migrated to a newer version. This was a major shift in building a new set up with both TIBCO BW migrating from the older 6.5 version to the newer 6.10 version and EMS migrating from the older 8.4 version to the newer 10.3 version. The major shift in the set up was the use of TIBCO FTL , acronym of Faster than light, as the foundational backbone of the new EMS. The new setup is called EMS on FTL stores, where FTL provides a synchronization and communication mechanism between the three EMS services that form a Quorum in the new design. Since the architecture was setup by the external HCL team, it seemed an uphill (but an imperative) journey to understand this complicated system design that acts as a bedrock of our operations. This is where CoPilot came to the rescue.
I regard AI in general, and our CTC based CoPilot as a modern alternative of a book club or discussion forum serving as a space to brainstorm ideas and refine conceptual understanding. It is like a laboratory where one can “stress-test” his ideas. To master the new EMS and FTL setup, I adopted a conversational learning approach, which is central to how Copilot builds expertise. I got to know how different layers of FTL , consisting of FTL cluster service, FTL persistence service and FTL realm service work together as a cohesive system to harmonize the workings of the EMS cluster. This intensive, two-way dialogue was highly iterative: Copilot corrected gaps in my understanding of the new design, while I occasionally corrected its contradictory statements. This collaborative process ultimately allowed me to build a crystal-clear insight into our new EMS on FTL architecture.
The grasp of conceptual understanding did not occur instantly. It required several back and forth arguments. As mentioned above, building on knowledge with the use of CoPilot is an iterative process. This back-and-forth approach helped me connect the dots, moving from an exhaustive grasp of how the FTL cluster server operates as the topmost layer to understanding how the FTL persistence service (used to persist data in the EMS cluster data stores) and the FTL realm service (used to synchronize and replicate configurations across the EMS servers) work together beneath that layered structure.
The first test done with CoPilot was to check how data replication happens across the EMS cluster hosted on three Linux based servers. To kick off the process, I provided CoPilot with the configuration of our EMS cluster system by attaching the Business-EMS.yaml file which consisted of two major pieces of information. The first was the path of the data store where all the persistent messages are stored and the second was the path of rs.db file which includes the configuration files in a .JSON format. In order to check if we have real time data replication, CoPilot recommended that I verify if the data store path is on nfs (Network file system) or XFS (which would indicate a standalone data store). I ran the df -Th /path-to-store-File and the output showed that the file system is XFS rather than a networked file system.
I was not convinced thus far since the store.db file in the store directory was almost the same size across all the three EMS servers. Upon putting forward this point, CoPilot prompted me to do a hands on test by suggesting that I create a test queue and push a message to this queue on the Active EMS server, kill the Active server so that Standy EMS Server takes on the Active role and verify if the queue and message appeared on the new Active server. I executed this test and observed that the queue indeed did got reflected on the new Active EMS server but the message did not got replicated. The concern was still not addressed. If the data replication in the data store was not real- time, how did the sizes of the files in the data store across the three EMS servers hosting the cluster matched?
Prompting this concern to CoPilot, and engaging in more intense conversations, I got to know that only persistent messages (these are the messages that get written to the disk so that they are never lost , unlike the non-persistent messages which get stored in EMS memory) get replicated real-time. So, based on our “discourse”, I performed another test of similar nature and pushed a persistent message on the test queue and I was glad to know that it was reflected on the new Active server. Knowing this aspect of the new EMS/FTL architectural design was the first big step towards building a solid foundation.
The next vital step was to understand how the EMS server interacts with FTL during boots-up. From my interactions with generative AI, I have experienced that the best way to extract the most from an AI agent like CoPilot is not to just ask a straight question like “how does something works?”. Instead, a better approach is to prompt it with your own initial conceptual understanding and build upon that foundation based on Copilot’s responses. This makes the interaction far more engaging and helps sharpen your knowledge within a specific context. I believe this collaborative feedback loop is exactly what makes prompt engineering so powerful.
I followed the same strategy and after some oscillating conversations, I was able to refine my understanding about the EMS start up/boot up process. Over a period of time, I have developed a habit of presenting a final summary to Co-Pilot based on our long conversations to chisel out the rough edges and close the loop. With the problem at hand, I did the same and proffered my final summary with the process that looked like the following – “When EMS server starts, it brings up its main process called tibemsd which calls the FTL realm service, which in turn loads the configurations from rs.db to the memory of tibemsd process. When we create a queue, tibemsd process stores it in its memory and also connects with FTL realm service which in turn stores it in the rs.db file across all the three EMS servers.”
The third step was to know how the FTL manages the Fault-Tolerance of the EMS. To know about it, it was imperative to first be acquainted with the difference between FTL cluster service, FTL persistence service and the FTL realm service. CoPilot did a wonderful job by presenting a diagram of how these three FTL processes are layered and function in conjunction with each other. This was the perfect example of the quote that “A picture is worth a thousand words”. Through the diagram, I got to know the complex idea in a very efficient manner.
Case Study 2 – Troubleshooting EMS / FTL issue: Knowledge brings confidence but like any area of research, this confidence is tested when you encounter the problem in real-time. In my case, this “opportunity” came when our team got alerted that our Active EMS in production suddenly and unexpectedly crashed and the Standby resumed the role of Active. Drawing on my understanding, I was able to analyze the logs of the crashed EMS server in a more effective manner. The logs indicated a “temporary network disconnection”. So we assumed that was a one-off incident and might not occur again. However, the issue repeated itself after a couple of days. I knew it was something beyond just a minor network disruption. I contacted the CTC Network team to know if there was any change made by them on our EMS servers during the timeframe when the issue occurred. Network team responded that there was no activity carried out on their end.
CoPilot once again became my “trusted colleague”. After I described the nature of the “roadblock” we were facing, It suggested that we need to upgrade the EMS to version 10.4.1 from 10.4.0 and FTL to version 7.1 from 7.0. By scanning the vast ocean of TIBCO documentation, CoPilot highlighted an inherent and a known bug in the current versions of the software that we were using. With management’s approval, I initiated the process to implement an in-place upgrade of the EMS and FTL instances. An in-place upgrade allows you to deploy the new software using the existing configurations, meaning we did not need new servers to execute this overhaul. Not only CoPilot described the entire process of upgradation, but also facilitated in deepening the comprehension of various features of the new versions. It laid out the exact plan of action including any prerequisites and migration stages required for this upgradation.
With the enriching assistance of CoPilot, I was able to upgrade EMS and FTL to the newer versions in our development environment, and we are now ready to execute the process in the higher environments.
To sum up the above two inter-related cases studies revolving around TIBCO EMS and FTL, I would like to mention that the value that you get from CoPilot is directly dependent on the input that you provide. It ultimately boils down to “prompt engineering” which includes scraping the concepts through long and concentrated conversations till every dot connects. This iterative approach is exactly how Copilot transforms your outlook and method for handling daily technical work.
