JDBC

Syndicate content 11 

Hi,
We recover a large amount of data from an external database (SQLServer, about 1 million rows in JDBC).
However, we have treatment time issue.
This process takes more than 30 minutes whereas on a "classic" SQL Server Management Studio type request takes less than a minute.

While searching on the internet, I came across this article: http://makejavafaster.blogspot.com/2015/06/jdbc-fetch-size-performance.html
It explains how in Java, we can tweak the FetchSize parameter of the JDBC driver to optimize this kind of process.

Is there a way to access this JDBC parameter through the EnsLib.SQL.OutboundAdapter adapter?

If not, how do you deal with this kind of situation of large volumetry

Last answer 20 September 2018 Last comment 7 days ago
0   1 5
164

views

0

rating

IRIS and Ensemble are designed to act as an ESB/EAI. This mean they are build to process lots of small messages.

But some times, in real life we have to use them as ETL. The down side is not that they can't do so, but it can take a long time to process millions of row at once.

To improve performance, I have created a new SQLOutboundAdaptor who only works with JDBC.

BatchSqlOutboundAdapter

Extend EnsLib.SQL.OutboundAdapter to add batch batch and fetch support on JDBC connection.

Benchmark

Benchmarks released on Postgres 11.2 with 1 000 000 rows fetched and 100 000 rows inserted on 2 columns.

alt text

Prerequisites

Can be used on IRIS or Ensemble 2017.2+.

Installing

Clone this repositor

0   0 1
0

comments

53

views

0

rating

I am under the impression that inside HealthShare you can Services, Processes, and Operations. The service takes incoming data, the processes can process that data, and then the operations can send that data out.

 

I am curious if anyone has any experience, guides, or advice on how I can send data to a service using a Java application. I intend for the data to be in XML format. I am also curious if I succeed at picking up the data in the service, how I send the data back to the Java application using an Operation.

 

As of now I am attempting to get this to work on a local environment.

 

Any input is appreciated,

Thank you,

Jess Farmer

0   0 2
0

comments

52

views

0

rating

Hi, I'm looking for a way to implement the JDBC pattern "executeBatch" in ObjectScript: https://www.tutorialspoint.com/jdbc/jdbc-batch-processing.htm

Since Ensemble 2017.2.2 there is a new method called executeParametersBatch : https://docs.intersystems.com/ens20172/csp/docbook/relnotes/index.html#D...

Even in the Ensemble 2018.1 change log, mention a patch on this method: https://docs.intersystems.com/ens20181/csp/docbook/relnotes/index.html#J...

Do you have any examples or tracks to use the executeParametersBatch method of %Net.Remote.Java.JDBCGateway?
Otherwise have you found other workaround for batch insert in JDBC through an EnsLib.SQL.OutboundAdapter

Last answer 22 January 2019 Last comment 23 January 2019
0   0 3
124

views

0

rating

Astronomers’ tools

5 years ago, on December 19, 2013, the ESA launched an orbital telescope called Gaia. Learn more about the Gaia mission on the official website of the European Space Agency or in the article by Vitaly Egorov (Billion pixels for a billion stars).

However, few people know what technology the agency chose for storing and processing the data collected by Gaia. Two years before the launch, in 2011, the developers were considering a number of candidates (see “Astrostatistics and Data Mining” by Luis Manuel Sarro, Laurent Eyer, William O’Mullane, Joris De Ridder, pp. 111-112):

Comparing the technologies side-by-side produced the following results (source):

TechnologyTime
DB213min55s
PostgreSQL 814min50s
PostgreSQL 96min50s
Hadoop3min37s
Cassandra3min37s
Caché2min25s

The first four will probably sound familiar even to schoolchildren. But what is Caché XEP?

Last comment 9 January 2019
+ 8   1 3
474

views

+ 8

rating

As we all know, Caché is a great database that accomplishes lots of tasks within itself. However, what do you do when you need to access an external database? One way is to use the Caché SQL Gateway via JDBC. In this article, my goal is to answer the following questions to help you familiarize yourself with the technology and debug some common problems

Last comment 28 December 2018
+ 8   2 4
1792

views

+ 8

rating

We are trying to script a High Availability Shutdown/Start script in case we need to fail over to one of our other servers we can be back up within mins. Is there a way to configure the startup procedure to Automatically Stop/Start the JDBC server when shutting down or starting up cache? is there an auto setting we can change?

Thanks

Scott Roth

The Ohio State University Wexner Medical Center

Last answer 15 November 2018 Last comment 18 November 2018
0   0 4
152

views

0

rating

I'm new to Iris, trying to follow this guide ( https://community.intersystems.com/post/k-means-clustering-iris-dataset ) about using clustering. I've followed all instructions until showing the data. At that point, `dataFrame.show()` fails with `java.sql.SQLException: Access Denied`.

I'm using IRIS 2018.12.609.0 on Windows 10 Pro 64 bits. Python is 3.6.6, PySpark is 2.3.1, installed from Anaconda (Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32)

Last answer 12 September 2018 Last comment 4 October 2018
0   0 1
202

views

0

rating

We have a new requirement being push down by our Data Security to no longer use Local SQL Accounts to access our Databases. So they asked me to create a Service Account that is on the Domain for our connections to each database.

I tried just changing my JDBC connection to using this Service Account and Password but I am not having any luck trying to connect to the database.

" Connection failed.
Login failed for user 'osumc\CPD.Intr.Service'. ClientConnectionId:ade97239-c1c8-4ed1-8230-d274edb2e731 "

In reading some of the material about using a Domain Service Account it mentions having kerberos installed. Is this needed for Ensemble JDBC connection to a Microsoft SQL Database to work using a Domain Service Account?

Does anyone use JDBC connections to connect to non cache databases using a Domain Service Account? If so how was this accomplished?

Thanks

Scott

Last answer 16 August 2018 Last comment 16 August 2018
0   0 3
359

views

0

rating

I have several stored procedures that when I execute them they will only return a single snapshot. In my BP I have been setting this to a Snapshot variable then looping using a WHILE through the snapshot variable just to get that single value.

Since it is only a single row, is there an easier way where I don't have to do a WHILE loop to pull the values out of that row? Can I call First Row or something like that to get me just the row into the Snapshot variable?

Last answer 14 June 2018
0   0 2
0

comments

149

views

0

rating

Without installing Kerberos has anyone Authenticated a SQL JDBC connection? Currently we are using local SQL Accounts to sign onto External SQL Databases, but we are being told that we need to switch to Service accounts that live on a Active Directory Domain. 

I wrote with a little help a ZAUTHENICATE to do the Authentication for Ensemble, can I use something like that to connect to an External SQL Database using a Service Account on a Active Directory Domain?

Thanks

Scott

Last comment 7 June 2018
0   0 3
0

answers

158

views

0

rating

Hello glabal masters,

I am trying to develop WAS using Cache and Spring Framework.

The ON DUPLICATE KEY UPDATE is not working.

Source : 

INSERT INTO 
            ocsGUMCHECDINFO (CODE,DODATE,NAME,DAY,PURPOSE,REQUIREMENT,BEFORENOTE,AFTERNOTE,CNT,REPORT,CONSENT,CONTACT)
        VALUES
            (#{code},#{doDate},#{name},#{day},#{purpose},#{requirement},#{beforeNote},#{afterNote},#{cnt},#{report},#{consent},#{contact})
        ON DUPLICATE KEY UPDATE DAY=#{name},PURPOSE=#{day},REQUIREMENT=#{requirement},BEFORENOTE=#{beforeNote},AFTERNOTE=#{afterNote},CNT=#{cnt},REPORT=#{report},CONSENT=#{consent},CONTACT=#{contact}

Is "ON DUPLICATE KEY UPDATE" not working in Cache SQL? Is there any other way if this isn't working?

Thank you

Last answer 25 May 2018 Last comment 25 May 2018
0   0 2
133

views

0

rating

Hello Global masters!

I developed Web with Spring framework (JAVA, Jdbc Cache) and I wrote SQL when I made it.

And Window program with VB6, Cache Object and VisM.

Is there a significant performance gap? SQL and Cache Object?

same function, same computer performance just different Java and VB6.

Thank you.

Minsu.

Last answer 11 May 2018 Last comment 11 May 2018
0   0 2
223

views

0

rating

LS,

I'm executing a query using JDBC on a PostgreSQL db:

SET statement_timeout TO 600000000; COMMIT SELECT * FROM bi_hour

The query is aborted with the following message:

FOUT #5023: Fout in Java Gateway: JDBC Gateway getClob(0,2) errorBad value for type long : active

The column 'blocked_status' contains the value 'active' is of type 'text'. I figure somewhere the SQL Gateway tries to convert the text value into a long but I can't find where, any suggestions?

Last answer 27 April 2018 Last comment 27 April 2018
0   0 2
118

views

0

rating

This is a sample Ensemble/Health Connect production which demonstrates how to receive an HL7 order (ORM) inbound from a file, extract fields (in this case, basic demographic information), and insert those into a table in an external SQL database via ODBC.  

Included in the zip file:

  • Exported code
  • Sample ORM message
  • 'How to configure' doc 
Last comment 19 April 2018
+ 1   0 3
871

views

+ 1

rating

Apache Spark has rapidly become one of the most exciting technologies for big data analytics and machine learning. Spark is a general data processing engine created for use in clustered computing environments. Its heart is the Resilient Distributed Dataset (RDD) which represents a distributed, fault tolerant, collection of data that can be operated on in parallel across the nodes of a cluster. Spark is implemented using a combination of Java and Scala and so comes as a library that can run on any JVM. Spark also supports Python (PySpark) and R (SparkR) and includes libraries for SQL (SparkSQL), machine learning (MLlib), graph processing (GraphX), and stream processing (Spark Streaming).

+ 11   0 4
1645

views

+ 11

rating

Has anyone use any other setting then -1 on the Stay Connected setting on a SQL.OutboundAdapter? I am having issues with a particular database and wonder if it would be beneficial since I only connect once a day to read the data from the Adapter to create an HL7 message.

Thanks

Scott Roth

Last answer 6 April 2018 Last comment 9 April 2018
0   0 3
187

views

0

rating

Hi

I am fairly new to Ensemble development.

I was wondering what is the best process of returning the resultset from a SQL outbound adapter to the BPL that had invoked it.

Thank you for your help.

Kind Regards,

Salma

Last answer 2 February 2018 Last comment 5 February 2018
0   0 2
235

views

0

rating

I'm trying to create an sal connections from one cache system to another. I've created the connection part  and I think that is working. When trying to link a table through the link table wizard, I'm getting an error "<MAXSTRING>zCreateOneLinkTable+52^%CSP.UI.SQL.LinkWizardResultPage1.

Is this due to my global block size in the new system not being large enough? It is currently set at 8192.

Thank you

Mike

Last answer 16 January 2018 Last comment 16 January 2018
0   0 2
163

views

0

rating

What happens if you don't declare a Persistent value when you call ExecuteQuery()? What does Ensemble set as the key value for your query? I have a query that I've executed on SQL Server, and I get 15 rows, but because this is my second time querying the data Ensemble thinks it exists. 

The Method in question is SelectProviderClarityAudit. If I call this query multiple times it is not returning the same number of results each time in Ensemble.

Last answer 12 January 2018 Last comment 12 January 2018
0   0 2
155

views

0

rating

I'm referring to the document Using Cache with JDBC. Using this example, it sets the URL of the object ds of class CacheDataSource.

ds.setURL("jdbc:Cache://127.0.0.1:1972/SAMPLES");

My question is would you just copy and paste the URL from the login page to the string? I tried doing that and it returns an error saying "Invalid log file xyz"

 

Any help will be greatly appreciated! Thank you

Last answer 30 December 2017 Last comment 3 January 2018
0   0 1
550

views

0

rating

I ran the below query in three different modes.  Coordinated Universal Time is 5 hours ahead of Eastern Time so there is difference in value from column 1 and column 2 in ODBC, Display mode, but not in Logical Mode. By default the query executes in ODBC mode when we query the data from outside world(via ODBC connection). 

I don't know  why query 2 and query 3 outputs different from query 1.

Query 1. Ran in Logical mode, DATEPART() took in memory stored timestamp data(stored in UTC)

'Hour ' returned are same for both the columns

Query 2. Ran in ODBC mode, DATEPART() took in memory stored timestamp data(stored in UTC) ,but column1 converts data from logical mode to display mode while column 2 is still in logical mode .

Last answer 23 November 2017 Last comment 24 November 2017
0   0 1
139

views

0

rating

Hi,

I'm developing an integration between Caché servers by ODBC conection, and I have the following problem.

In this call:

call COSClass_Methode('222169^^98^155^64530^06:30^021542987897458855441112877855^1^0^281992^GC')

the ODBC driver is truncating the string to 50 characters.


If I run this same command with $system.SQL.Shell(), this doesn't occur.
I did a test creating several parameters for COSClass_Methode, and they all have a 50 character limitation.

Does anyone know why the ODBC Cache Driver limits the number of characters per parameter of this type of call?
Or, how can I adjust this?

Example of implementation http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=BGOD_gateway#BGOD_C127276

Last answer 6 September 2017 Last comment 7 September 2017
0   1 1
462

views

0

rating