JDBC

Syndicate content 12 

IRIS and Ensemble are designed to act as an ESB/EAI. This mean they are build to process lots of small messages.

But some times, in real life we have to use them as ETL. The down side is not that they can't do so, but it can take a long time to process millions of row at once.

To improve performance, I have created a new SQLOutboundAdaptor who only works with JDBC.

BatchSqlOutboundAdapter

Extend EnsLib.SQL.OutboundAdapter to add batch batch and fetch support on JDBC connection.

0   0 1
0

comments

123

views

0

rating

Astronomers’ tools

5 years ago, on December 19, 2013, the ESA launched an orbital telescope called Gaia. Learn more about the Gaia mission on the official website of the European Space Agency or in the article by Vitaly Egorov (Billion pixels for a billion stars).

However, few people know what technology the agency chose for storing and processing the data collected by Gaia. Two years before the launch, in 2011, the developers were considering a number of candidates (see “Astrostatistics and Data Mining” by Luis Manuel Sarro, Laurent Eyer, William O’Mullane, Joris De Ridder, pp. 111-112):

Comparing the technologies side-by-side produced the following results (source):

TechnologyTime
DB213min55s
PostgreSQL 814min50s
PostgreSQL 96min50s
Hadoop3min37s
Cassandra3min37s
Caché2min25s

The first four will probably sound familiar even to schoolchildren. But what is Caché XEP?

Last comment 9 January 2019
+ 8   1 3
511

views

+ 8

rating

As we all know, Caché is a great database that accomplishes lots of tasks within itself. However, what do you do when you need to access an external database? One way is to use the Caché SQL Gateway via JDBC. In this article, my goal is to answer the following questions to help you familiarize yourself with the technology and debug some common problems.

Last comment 28 December 2018
+ 8   2 4
1933

views

+ 8

rating

This is a sample Ensemble/Health Connect production which demonstrates how to receive an HL7 order (ORM) inbound from a file, extract fields (in this case, basic demographic information), and insert those into a table in an external SQL database via ODBC.  

Included in the zip file:

  • Exported code
  • Sample ORM message
  • 'How to configure' doc 
Last comment 19 April 2018
+ 1   0 3
943

views

+ 1

rating

Apache Spark has rapidly become one of the most exciting technologies for big data analytics and machine learning. Spark is a general data processing engine created for use in clustered computing environments. Its heart is the Resilient Distributed Dataset (RDD) which represents a distributed, fault tolerant, collection of data that can be operated on in parallel across the nodes of a cluster. Spark is implemented using a combination of Java and Scala and so comes as a library that can run on any JVM.

+ 11   0 4
1731

views

+ 11

rating

I've asked a lot of questions leading up to this, so I wanted to share some of my progress.

The blue line represents the number of messages processed.  The background color represents the average response time.  You can see ticks for each hour (and bigger ticks for each day).   Hovering over any point in the graph will show you the numbers for that period in time.

This is super useful for "at a glance" performance monitoring as well as establishing patterns in our utilization.

Last comment 4 November 2016
+ 5   0 0
281

views

+ 5

rating

The Caché System Management Portal includes a robust web-based SQL query tool, but for some applications it’s more convenient to use a dedicated SQL client installed on a user’s PC.

SQuirreL SQL is a well known open source SQL client built in Java, which uses JDBC to connect to a DBMS. As such, we can configure SQuirreL to connect to Caché using the Caché JDBC driver.

Last comment 13 July 2016
+ 7   0 9
2950

views

+ 7

rating