%External.Messaging and Kafka

Question

Question

Michael Davidovich · Mar 30, 2023

I am trying to write a message to a Kafta cluster programmatically (i.e. not in Ensemble or using a Production) using %External.Messaging. I am consistently getting a 'topic not found error' even though I have created the topic on the cluster. I dug into the client I created an when accessing the property KaftaClient.jclient.gateway I get the following output:

<THROW>%DispatchGetProperty+25^%Net.Remote.Object.1 *%Net.Remote.Exception <GATEWAY> com.intersystems.gateway.GatewayException com.intersystems.gateway.JavaGateway.dynamicExecuteGet(JavaGateway.java:3761) Field not found: com.intersystems.messaging.KafkaClient.gateway

I did check and reset my JAVA_HOME env variable thinking this way the problem, and after restarting the server I still get this error. I can't seem to edit the External Language Servers in the management portal (I run a local instance of IRIS and have all roles), however I may be confusing a gateway with an external language server? My JAVA_HOME is a is a JRE (C:\Program Files\Java\jre1.8.0_361) not a JDK if that matters?

I'm aware that there may be a multitude of reasons why this isn't working but this stood out to me as something that could be a reason things aren't working. I read on a Stack Exchange article that people have the same issue using Spring Boot were missing some kind of Java-JSON dependency, so I thought to look for something Java related to solve my problem.

I would sure appreciate some thoughts if anyone has some to spare!

Product version: IRIS 2022.1

$ZV: IRIS for Windows (x86-64) 2022.1.2 (Build 574U) Fri Jan 13 2023 15:00:26 EST

Discussion (4)1

Log in or sign up to continue

Michael Davidovich · Apr 24, 2023

I would just recommend to anyone who comes across this to use a Python library like kafka-python. The documentation is clearer on how to configure the client or producer clearly. You can pass in a configuration to %External.Messaging but it's not clear how the parameters need to be formatted.

1 0

score 0 · Answer 1 · 2023-04-01T03:50:18-04:00

Calling @Bob Kuszewski.

My JAVA_HOME is a is a JRE (C:\Program Files\Java\jre1.8.0_361) not a JDK if that matters?

No, it should not.

I can't seem to edit the External Language Servers in the management portal

Please stop it before editing. You also should be able to create a new External Language Server.

score 0 · Answer 2 · 2023-04-03T13:18:23-04:00

@Eduard Lebedyuk thanks for your response here!

I changed directions and used embedded Python and a library called kafka-python which appears to be well-used and supported. Using this, I've had no issues getting connected to my server, plus now I have access to more commands for Kafka.

Before I edit or setup a new External Language Server 1) do you think the %Net.Remote.Exception is related to the Java Home on the external language server not being setup and 2) if I had multiple external language servers setup, how would I tell %External.Messaging (and specifically the Kafka client) to use one or the other?

It would be ideal to stick with COS libraries and features, but I think the Python library is the best solution as it offers more in terms of an API, but I'd still like to get %External.Messaging working for other purposes.

score 0 · Answer 3 · 2023-04-03T13:35:19-04:00

Setting the Java Home setting on the External Language server still gives me the same error in KaftaClient.jclient.gateway.