8CIOReview | | OCTOBER 2023IN MY OPINIONIt is common to associate digital transformation with UX, agile methodologies, apps, microservices, and cloud computing. When we think about startups with no legacies, this combination seems perfect. However, for large corporations, the situation becomes more challenging as legacy integration remains a key component behind the scenes to support the new digital mindset. Unfortunately, legacy integration is often underestimated or neglected due to poor documentation and reliance on deprecated technologies. However, the challenge of handling real-time data to feed digital apps will eventually arise.Traditional approaches typically rely on batch processing and data replicas, but the volume of data and infrastructure restrictions can quickly lead to frustration. Batch processing takes time, and even incremental replication can be difficult when dealing with outdated legacy systems that have incompatible data schemas. As the situation progresses, it becomes increasingly challenging when additional new digital apps require different types of data, yet they are still dependent on the same legacy systems. Eventually, the legacy systems may reach a breaking point where they are unable to handle the high volume of requests, leading to crashes and creating significant problems for both the legacy systems and the digital apps. The pressure mounts as the demand exceeds the capabilities of the legacy infrastructure.Undoubtedly, this challenge is also present in the education sector. For instance, when it comes to attracting students, but not limited to that, we face the task of making course offers readily available across multiple digital channels in real-time, all supported by our existing legacy systems. In response to this need, our company has initiated a project to address and handle these sorts of requirements.Idea revolves around harnessing the power of an event-driven architecture facilitated by change data capture (CDC) at the database level, all while ensuring transparency to the legacy application. By utilizing CDC, we can capture data in real-time and effortlessly stream it to multiple digital app consumers. Our approach involves adding a new computing layer for CDC, which enhances the process of data integration. Despite this addition, our approach still offers the advantage of seamless integration with legacy applications, without the need for any additional computing resources or modifications to the existing codebase. Moreover, the data streaming process can be easily configured to capture a wide range of data events, including inserts, updates (considering the before and after state), and deletion indications. This flexibility enables comprehensive data synchronization, ensuring that all relevant data changes are efficiently captured and streamed in real-time. With our approach, we can read data once and make it available for reuse multiple times in real-time without causing any crashes or disruptions to the legacy systems.On the other hand, our approach allows flexibility and respects the diverse paces and methodologies of different squads within the organization. Some squads solely consume data from our data stream, while others may create their own replicas for local persistence, such as our CRM solution. However, in all cases, data is available for consumption in real-time, eliminating the need for batch processing BRIDGING THE GAP: EFFECTIVELY MANAGING DATA IN THE ERA OF DIGITAL APPS AND LEGACIESBy Alexandre Gonzalez, Chief Technology Officer, YDUQSAlexandre Gonzalez
< Page 7 | Page 9 >