|
Convergence of ETL and Application Integration
|
Spreadsheets have been around since the late 1970s. They were an instant sensation. Over the years, spreadsheets have evolved and matured, but the basic form and substance of spreadsheets has hardly changed. While new features and capabilities continue to be added to spreadsheets, for the most part spreadsheet technology has reached a plateau. This is typical of a highly successful product. However, it is becoming clear that new approaches and paradigms should and are beginning to emerge.
|
|
Creating interfaces between applications to exchange data is almost as old as computing itself. As systems evolved from mainframes to open systems, from monolithic ERPs to independent applications, the need for these systems and applications to exchange data became more and more important.
More recent is the emergence of data warehousing and business intelligence, which require collecting vast amounts of data from multiple systems.
In the past few years, both needs data exchange between application and loading of Data Warehouses have been systemized with the introduction of EAI (Enterprise Application Integration) suites and complex ETL (Extraction Transformation and Loading) solutions.
Even though these two types of solutions are very distinct and fulfill different needs, there are many common points that make their convergence inevitable. This article examines three major areas where convergence is today a reality.
Data Latency
An Application Integration suite is by essence a real-time or near-real-time solution. Latency is determined by the need, and not all data flows have the same latency requirement, but whether it is a few seconds or several hours, the data needs to be propagated in a timely fashion. Moreover, the data transfers occur while the applications are running and serving users the disruption to the transactional capabilities of the systems should thus be minimized.
Conversely, a traditional ETL process runs in batch mode daily, weekly or even monthly and usually while production systems are either taken offline, or at least show minimal activity (middle of the night, weekends, etc.)
The recent years have shown an increase of interest for real-time Data Warehousing and Business Activity Monitoring (BAM), where data needs to be analyzed in a timeframe sufficient to be able to make proactive (as opposed to reactive) decisions. At this stage, the Data Warehouse can be viewed as one of many systems that must be updated in a timely fashion with new data, making it very much a part of the Application Integration scheme. However Data Warehouse loading still encompasses several requirements such as complex transformation and data dependencies that must be taken into account.
|
|
|
Oracle #1 in Business Analytics According to IDC Research
|
The Business Intelligence Search Engine has all the answers.
|
Find all you need on The Business Intelligence Search Engine.
|
|
|
|
|
|
|
|
Other
Articles by this Author
|
|
|
