Discussion
Dan Pasco · Apr 23, 2021

External Language Server in 2021.1

Now that IRIS 2021.1 is available as a preview version, I would like to demonstrate a "new" feature. The Java Gateway has been around for a while now but in 2021.1 it has new skills. External Language Servers are available for Java, DotNet, and Python. Here is a quick - very quick - demo of using the External Java Server. Please don't focus solely on what this demo is doing but rather on what is happening in this demo. First, I acquire a gateway connection oref. This gateway connection is connected to the External Java Server - one of the External Language Servers.

set java = $system.external.getJavaGateway()

Next, I need to add my jar file to the class path:

do java.addToPath("~/projects/external/java/iris-external/target/iris-external-1.0-SNAPSHOT.jar")

I have a method in my external.test.Horizons class that queries the Sample.Person table in some IRIS instance.

    public static ExternalResult getPersons(int rowcount) throws SQLException {
        return new ExternalResult(getPersonsResults(rowcount));
    }

    private static ResultSet getPersonsResults(int rowcount) throws SQLException {
        Properties properties = new Properties();
        properties.put("user", "_SYSTEM");
        properties.put("password", "SYS");
        Connection connection = DriverManager.getConnection(IRIS_URL, properties);
        Statement statement = connection.createStatement();
        ResultSet result = statement.executeQuery("select top " + rowcount + " name, dob, ssn, home_street, home_city, home_state, home_zip from sample.person order by name");
        return result;
    }

Simply invoke this method using the ELS invoke method:

set persons = java.invoke("external.test.Horizons","getPersons",10)

In my example, persons is holding a reference to an instance of my external.test.ExternalResult Java class. This class is a wrapper around a JDBC Result Set. I've implemented a renderTable() method that displays the contents of the ResultSet using an AsciiTable library:

do persons.renderTable()

And the output:

┌──────────────────┬──────────┬───────────┬──────────────────────┬──────────┬──────────┬────────┐
│Name              │DOB       │SSN        │Home_Street           │Home_City │Home_State│Home_Zip│
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Adam,Barb X.      │1990-08-15│568-33-3285│4907 Clinton Court    │Boston    │ID        │67314   │
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Adam,Milhouse K.  │1989-07-05│797-35-8746│9623 Franklin Blvd    │Youngstown│AR        │31403   │
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Adams,Mo Z.       │1962-02-15│263-25-6173│5449 Washington Street│Washington│WA        │32342   │
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Ahmed,Al S.       │1985-01-22│623-94-2440│8103 Main Court       │Oak Creek │TN        │96256   │
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Ahmed,Barb R.     │1956-12-13│373-71-8424│9509 Clinton Avenue   │Newton    │CT        │56929   │
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Ahmed,Charlotte P.│1947-05-09│944-70-3491│1610 Second Place     │Oak Creek │VT        │54476   │
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Ahmed,Debby W.    │2005-05-20│611-75-3175│150 First Court       │Ukiah     │HI        │93961   │
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Ahmed,Yan V.      │1949-07-22│552-24-3419│3684 Madison Drive    │Denver    │OH        │12175   │
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Allen,Geoffrey D. │1963-09-21│956-84-9269│6628 Madison Place    │Vail      │MT        │86032   │
├──────────────────┼──────────┼───────────┼──────────────────────┼──────────┼──────────┼────────┤
│Allen,Howard B.   │1943-01-09│834-27-9607│6899 Clinton Court    │Vail      │GA        │69486   │
└──────────────────┴──────────┴───────────┴──────────────────────┴──────────┴──────────┴────────┘

DotNet and Python External Language Servers work similarly.

40
3 0 9 161
Log in or sign up to continue

Replies

In the 2021.1 preview, you can execute $system.external.Help() to get a description of the External Language Server interface.

Thanks, but a terminal command (requiring you to install the preview release) is not what I call documentation. But I did install it. To give an example of what the "Help()" gives me:

Help(method)
     Write out a list of the methods of this object to the console.
addToPath(path)
     <p>
createServer(serverDef)
     Create a new server.  This function requires the "%Admin_Manage" resource.

Interestingly, on the local installed preview, the class documentation does show the %SYSTEM.external class (with ever so slightly more information, btw.). I really hope that this lack of documentation is fixed before release, because as it is now, it is unusable.

Regard,
Gertjan.

You are right, of course. I was puzzled by the lack of documentation myself. I did some research and I've been told that there will be documentation available for the External Language Servers and $system.external.

The $system.external.Help() feature is something that has been part of the product for many years. It is not a substitute for proper documentation but serves to aid the command line user when questions regarding function names and arguments need a quick answer.

The ELS features are based on what is known as "Dynamic Object Gateways". There is extensive documentation available for them. That documentation is a bit dated in that it doesn't incorporate the new $system.external API's but still mostly accurate. Obtaining a gateway connection to an External Language Server by simply invoking $system.external.getGateway(<gatewayname>) is just one way we've simplified that interface.

There are default ELS's defined for Java, Python, DotNet, and R. Each is supported not only by the getGateway() that accepts the name of the ELS but also by get<Language>Gateway() functions for Java, DotNet, Python and R that return gateway connections to the default language servers.

Could you explain the term you used, Language Server?

Language Server now used for editors. And I see no reactions to it in your article. And InterSystems already implemented real language server for VSCode, and I have implemented one more for any other editor.

I haven’t used any of the Gateways before, nor have I played with any of the External Language Servers. It looks like the new skill demoed here is that an IRIS object (persons in this example) is able to hold a reference to a Java class outside of IRIS, and you can call methods from that Java class as if they are methods inside IRIS, including making use of Java libraries (AsciiTable in this example). So Java System.out redirects to the IRIS Terminal window? If so, that all seems amazing! Do you have some examples you can share about how we expect developers to use this?

Exactly - you haven't used the Gateways before. Why? IMO, the Gateways (mostly Java and DotNet) were a bit cumbersome to deal with. With ELS, we have default servers that can be easily managed and discovered, they require little or no configuration to get started (some ELS's do require some configuration if the language platform support is not discoverable by IRIS), and the interface is simple and direct.

One quick example, using Python, is to use the os module to get the current working directory:

USER>set python = $system.external.getPythonGateway()

USER>write python.invoke("os","getcwd")
/opt/intersystems/iris/xdbc/mgr

All that is required is a gateway connection to an external server, your code needs to be visible to that external server, either by direct placement into the default path for that language platform or by explicitly adding it by calling addToPath(), and public interfaces in your external code. By "external", I mean code that isn't written inside of the IRIS Server - ObjectScript.

When your external code writes to the "system output" device, that output is redirected to the IRIS current device. In my above example, the renderTable() function simply constructs a formatted string using the AsciiTable library (got it from GitHub) and writes it using System.out.println(formattedstring). I simply copied the output that was displayed in my IRIS session terminal window and pasted it in the original post. No extra work involved.

If your external code returns an object then you can indeed make use of that object as if it were a local IRIS Object - because it is. It is actually a network proxy object that communicates with the original external language object. That communication can actually be full duplex - your external code can talk to IRIS and IRIS can talk to your external code.

One really simple example that I think is quite profound is using JDBC in IRIS. This isn't JDBC Gateway - that still exists and it is what it is. I am talking about using a Java JAR file that contains a JDBC driver and you want to use it. I'll try a simple demo here. I have MariaDB installed on my system so I'll just use it to query a table I have defined there.

First - using mariaDB from the command line interface:

 ~ % mysql -u myusername -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 10.5.8-MariaDB Homebrew

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases
    -> ;
+--------------------+
| Database           |
+--------------------+
| SAMPLES            |
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
5 rows in set (0.010 sec)

MariaDB [(none)]> use SAMPLES
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [SAMPLES]> show tables;
+-------------------+
| Tables_in_SAMPLES |
+-------------------+
| person            |
+-------------------+
1 row in set (0.000 sec)

MariaDB [SAMPLES]> select * from person limit 5
    -> ;
+-----------------------+-------------+------------+-----------------------+------------+------------+----------+
| name                  | ssn         | dob        | home_street           | home_city  | home_state | home_zip |
+-----------------------+-------------+------------+-----------------------+------------+------------+----------+
| Basile,Molly M.       | 452-57-8033 | 1994-06-02 | 1153 First Street     | Xavier     | SD         | 98033    |
| Cooke,Howard F.       | 131-62-3894 | 2017-12-09 | 5172 Washington Place | Zanesville | NE         | 44980    |
| Donaldson,Phil R.     | 480-79-5019 | 1990-01-23 | 4429 Elm Street       | Miami      | WA         | 67638    |
| Eisenstien,Michael D. | 655-11-6334 | 1948-08-14 | 4676 Elm Avenue       | Reston     | KY         | 52729    |
| Faust,Mo E.           | 772-42-3921 | 2018-01-10 | 826 Maple Avenue      | Youngstown | OH         | 37180    |
+-----------------------+-------------+------------+-----------------------+------------+------------+----------+
5 rows in set (0.016 sec)

MariaDB [SAMPLES]>

Next, I'll follow up with an example of doing this from within IRIS.

    set java = $system.external.getJavaGateway()
    do java.addToPath("/home/myhome/mariadb-java-client-2.7.2.jar")
    set mariads = java.new("org.mariadb.jdbc.MariaDbDataSource","jdbc:mariadb://myhost:3306/SAMPLES")
    set mariaconn = mariads.getConnection("myuser","secret")
    set pstmt = mariaconn.prepareStatement("select * from person limit 5")
    set jdbcresult = pstmt.executeQuery()
    set jdbcmetadata = jdbcresult.getMetaData()
    for i=1:1:jdbcmetadata.getColumnCount() {
        write jdbcmetadata.getColumnName(i),$char(9)
    }
    while jdbcresult.next() {
        write !,$increment(rowcnt),$char(9)
        for i=1:1:jdbcmetadata.getColumnCount() {
            write jdbcresult.getString(i),$c(9)
        }
    }
    write !!!,"OR I COULD JUST USE MY ALREADY IMPLEMENTED RESULT SET RENDERER FROM ANOTHER JAVA PROJECT",!!!
    do java.addToPath(..#EXTERNALLIBPATH)
    set ers = java.new("external.test.ExternalResult",pstmt.executeQuery())
    do ers.renderTable()
    do mariaconn.close()
    do java.disconnect()

And the results:

USER>do ##class(external.test.MariaJDBC).communityDemo()
name    ssn    dob    home_street    home_city    home_state    home_zip    
1    Basile,Molly M.    452-57-8033    1994-06-02    1153 First Street    Xavier    SD    98033    
2    Cooke,Howard F.    131-62-3894    2017-12-09    5172 Washington Place    Zanesville    NE    44980    
3    Donaldson,Phil R.    480-79-5019    1990-01-23    4429 Elm Street    Miami    WA    67638    
4    Eisenstien,Michael D.    655-11-6334    1948-08-14    4676 Elm Avenue    Reston    KY    52729    
5    Faust,Mo E.    772-42-3921    2018-01-10    826 Maple Avenue    Youngstown    OH    37180    

OR I COULD JUST USE MY ALREADY IMPLEMENTED RESULT SET RENDERER FROM ANOTHER JAVA PROJECT

┌─────────────────────┬───────────┬──────────┬─────────────────────┬──────────┬──────────┬────────┐
│name                 │ssn        │dob       │home_street          │home_city │home_state│home_zip│
├─────────────────────┼───────────┼──────────┼─────────────────────┼──────────┼──────────┼────────┤
│Basile,Molly M.      │452-57-8033│1994-06-02│1153 First Street    │Xavier    │SD        │98033   │
├─────────────────────┼───────────┼──────────┼─────────────────────┼──────────┼──────────┼────────┤
│Cooke,Howard F.      │131-62-3894│2017-12-09│5172 Washington Place│Zanesville│NE        │44980   │
├─────────────────────┼───────────┼──────────┼─────────────────────┼──────────┼──────────┼────────┤
│Donaldson,Phil R.    │480-79-5019│1990-01-23│4429 Elm Street      │Miami     │WA        │67638   │
├─────────────────────┼───────────┼──────────┼─────────────────────┼──────────┼──────────┼────────┤
│Eisenstien,Michael D.│655-11-6334│1948-08-14│4676 Elm Avenue      │Reston    │KY        │52729   │
├─────────────────────┼───────────┼──────────┼─────────────────────┼──────────┼──────────┼────────┤
│Faust,Mo E.          │772-42-3921│2018-01-10│826 Maple Avenue     │Youngstown│OH        │37180   │
└─────────────────────┴───────────┴──────────┴─────────────────────┴──────────┴──────────┴────────┘

I'll be posting the first draft of "Using External Language Servers" on Monday. This draft will only cover the basics, but it should be considerably better than nothing. I'll be adding a lot more before the final release.