Question
xiao jikang · Sep 17, 2018

Is there a function similar to Oracle's CDC in the cache database?

After enabling archiving log functions, Oracle can get real-time change data through CDC.

Can Cache database realize such functions?

My requirement is to take incremental data from the cache database regularly every day and synchronize it to other relational databases.

Incremental data can be text files.

30
6 0 16 783
Log in or sign up to continue

Hi Xiao.

Many people here know Caché. Not so many know Oracle.

If you would explain what is Oracle CDC. What it does, and what features do you need, then we would be able to answer how you can do this in Caché.

Thank you,
Alexander.

Sorry, I didn't expect that.

My requirement is to take incremental data from the cache database regularly every day and synchronize it to other relational databases.

Incremental data can be text files.

I will add it. For example, the amount of data in my cache database has reached 10Tb, I have imported this 10Tb data into hadoop hdfs, and then analyze it through hbase. However, business data continues to be generated. The cache does not support direct docking of hdfs to store data. Therefore, I need the same cdc record table as oracle to store all dml and ddl operations. I only need to extract the data from the cdc record table. Do the corresponding operation in hdfs. The journal records the information of the global. If it is out of the cache, the global information is useless. So many times we need data at the table level.

I would like to generate text files from the daily increments of the cache database to facilitate synchronization with other relational databases, such as oracle, db2, mysql...

Do not consider real-time, synchronize once a day.

Because the amount of data is too large, we can not use the full amount of synchronization.

You say shadow and mirror are operations between cache and cache.

I would like to generate text files from the daily increments of the cache database to facilitate synchronization with other relational databases, such as oracle, db2, mysql...

Do not consider real-time, synchronize once a day.

Because the amount of data is too large, we can not use the full amount of synchronization.

Can DeepSee solve my problem?

Journal is definitely a valuable source.
BUT: It documents Global changes. It doesn't know about objects nor rows in tables.

From what I've gathered from Oracle, CDC is just a way to replicate data onto another system. If that's what you're looking for, Caché mirroring is the answer you're looking for. Take a look at:

https://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY...

That's the general mirroring information for how it works. Contained in that chapter is information about reporting asyncs, which is what it sounds like Oracle's CDC is for:

https://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY...

Though, it looks like Oracle replicates data to a different database on the same system, which mirroring will not allow. If you want to do replication to a different database on the same instance (though I can't think of a use case for that), you could use shadowing instead:

https://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY...

with ^OBJ.DSTIME you get the registered changes.
It is up to you to process it and to generate whatever format or style you prefer when and how often you like

So it's nothing ready and YOU have to write some code and manage it. 

DeepSee does nothing for you.

I would recommend the DSTIME approach as well. We use this for syncing up information from our cache ERP to our BI tool. DeepSea is actually using DSTIME to show changes as well. 

With using DSTIME you are allowed to write code and determine what DSTime batches you want to synchronize with, allowing you to fully control how often you want to sync up data. 

How to get the complete journal?