Net.HttpRequest - blank response? | InterSystems Developer Community

Web Services, Caché

Hi,

I am trying to create a script to connect to a website and extract all the content of the page. And in future to connect to an API.

Set httprequest=##class(%Net.HttpRequest).%New()
DO httprequest.Get("http://www.intersystems.com",2)
Write ">"_httprequest.HttpResponse

But the output is a wonderful blank HttpResponse

Any ideas?

Thanks

Answers

Answer by Robert Cemper 1 August 2018

There are some mistakes.

#1 the links should be "HTTPS://www.intersystems.com" and you didn't set a ssl/tls config.

if you use

Set sc=httprequest.Get("http://www.intersystems.com",2) Do $system.OBJ.DisplayError(sc)

you get ERROR #6159: ===> SSL missing

#2 HttpResponse is an ObjectReferce not a Property

set res=httprequest.HttpResponse ZW res res=<OBJECT REFERENCE>[3@%Net.HttpResponse] +----------------- general information --------------- | oref value: 3 | class name: %Net.HttpResponse | reference count: 3 +----------------- attribute values ------------------ | ContentBoundary = "" | ContentInfo = "" | ContentLength = 178 | ContentType = "text/html" | Data = "4@%Stream.GlobalCharacter" ;;; here is your reply |Headers("CONNECTION") = "keep-alive" |Headers("CONTENT-LENGTH") = 178 |Headers("CONTENT-TYPE") = "text/html" | Headers("DATE") = "Wed, 01 Aug 2018 15:25:05 GMT" |Headers("LOCATION") = "https://www.intersystems.com/" | Headers("SERVER") = "nginx" | Headers("X-TYPE") = "default" | HttpVersion = "HTTP/1.1" | ReasonPhrase = "Moved Permanently" | StatusCode = 301 | StatusLine = "HTTP/1.1 301 Moved Permanently" +-----------------------------------------------------

The content is in a Stream!!
So Write is totally inappropriate to show it. Instead:

do res.OutputToDevice() ;;;or similar HTTP/1.1 301 Moved Permanently CONNECTION: keep-alive CONTENT-LENGTH: 178 CONTENT-TYPE: text/html
DATE: Wed, 01 Aug 2018 15:25:05 GMT LOCATION: https://www.intersystems.com/ SERVER:nginx
X-TYPE: default <html> <head><title>301 Moved Permanently</title></head> <body bgcolor="white"> <center><h1>301 Moved Permanently</h1></center> <hr><center>nginx</center> </body> </html>

+ 1

Comment by Antonio Garcia ... 2 August 2018

Yo are absolutely right although I am still confused.

I copied httprequest.Get("http://www.intersystems.com",2) from one example found in the documentation so I was expected to work. Also, to get data from a website don't think you have to use https or SSL right? I tried with different APIs in other applications and just passing the right URL I was able to retreive the data.

About httprequest.HttpResponse you are right again... I was using httprequest.HttpResponse.Data and outputtodevice but I was having an error as the object did not exist...so I tried to check if the object was really created or not...

What would be the right script to connect then? I am trying to get data from this API https://api.postcodes.io/postcodes/

Thanks

Comment by Antonio Garcia ... 2 August 2018

I have tried to created an SSL/TLS configuration but not sure why is not working:

It does not matter what I try to test it...always returns the same error:

ERROR #989: SSL connection failed, make sure server address and port (not url) is specified

I tried "intersystems.com" 443, "postcodes.io" 443...nothing

Those sites I am trying to connect to....don't really need anything special to connect...just url

Thanks

Comment by Robert Cemper 2 August 2018

You have to test on server = "www.intersystem.com" not just domain.
443 is ok

USER>s req=##class(%Net.HttpRequest).%New() USER>s req.SSLConfiguration="SSL" USER>set sc=req.Get("https://www.intersystems.com") USER>zw sc sc=1 USER>set data=req.HttpResponse.Data USER>do data.OutpuToDevice()

>>>> lot of content <<<<

Answer by Iain Bray 2 August 2018

This will work. The second parameter of '2' indicates test mode and will write out the response to the current device

set httprequest = ##class(%Net.HttpRequest).%New()
set httprequest.Server = "www.intersystems.com"
set status = httprequest.Get("/",2)

You should see something like this:

HTTP/1.1 301 Moved Permanently
CONNECTION: keep-alive
CONTENT-LENGTH: 178
CONTENT-TYPE: text/html
DATE: Thu, 02 Aug 2018 10:21:56 GMT
LOCATION: https://www.intersystems.com/
SERVER: nginx
X-TYPE: default

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

Alternatively you can do this:

Setup an SSL/TLS Configuration. This is very simple and doesn't have any values apart from a name. Mine is called 'test'

Then use the following code:

set httprequest = ##class(%Net.HttpRequest).%New()
set httprequest.Server = "www.intersystems.com"
set httprequest.SSLConfiguration = "test"
set httprequest.Https = 1
set status = httprequest.Get("/")
if status {
do httprequest.HttpResponse.Data.Rewind()
do {
write httprequest.HttpResponse.Data.Read()
} while 'httprequest.HttpResponse.Data.AtEnd
}

Comment by Antonio Garcia ... 3 August 2018

Thanks for your help... I think the problem I am having is not with my code but with ensemble or IT. Apparently somehow the connection to external content is blocked. I can access to the websites with no issues from web browsers but when trying to access from studio or test SSL from ensemble...error!!

Is there any security configuration to block that kind of access in ensemble?

Thanks

Comment by Iain Bray 3 August 2018

Ok, so a couple of things to check:

1. When you say that you can connect from a browser but not from Ensemble, is the browser that you're using on the same computer as Ensemble?

2. Instead of a 'do httprequest.Get(...)' use 'set status=httprequest.Get(...)' and then inspect the value of 'status'. You can do 'do $system.OBJ.DisplayError(status)' in order to get a readable output. If the result is that you cannot open a TCP/IP socket then Ensemble cannot see the other computer. Otherwise the status should give you more information.

Are you addressing the other computer by its IP address or its DNS name? If you're using the DNS name then try using the IP address. If that works then edit your 'hosts' file to put the name and IP address in there and go back to using the DNS name to get to the other computer.

Let me know how you get on.

Iain

Comment by Iain Bray 3 August 2018

Apologies regarding the DNS name. I've just remembered that you're trying to get a response from 'www.intersystems.com' so the DNS wouldn't have an effect. What you should check is that the Ensemble instance has access to the internet.

There isn't anything in Ensemble that will prevent access, it might be something at the O/S or network level.

Iain

Comment by Antonio Garcia ... 3 August 2018

Thanks for the help:

1. I have tried from the server where ensemble is running. In IE, I can access www.intersystems.com but when trying from studio...nothing blank response...all from the same matchine!

2. I tried that and yes...ERROR #6059: Unable to open TCP/IP socket to server www.intersystems.com:80 so I think you are right but I cannot believe the problem is no visibility when in IE I can access the website with no issues.

Comment by Antonio Garcia ... 3 August 2018

I have tried by IP, apparently 38.111.0.52

so in IE: http://38.111.0.52:80 => ok, https://38.111.0.52:443 => ok

but using:

Set httprequest=##class(%Net.HttpRequest).%New()
Set httprequest.Server = "38.111.0.52"
Set httprequest.Port=80
set status = httprequest.Get("/",2)
DO $system.OBJ.DisplayError(status)

ERROR #6059: Unable to open TCP/IP socket to server 38.111.0.52:80

I have no idea what's going on

Comment by Iain Bray 3 August 2018

Hi,

So I checked that the web site will receive a ping, some don't for security purposes, and it seems ok:

ping www.intersystems.com

PING e9933.g.akamaiedge.net (104.103.202.60): 56 data bytes

64 bytes from 104.103.202.60: icmp_seq=0 ttl=56 time=80.280 ms

64 bytes from 104.103.202.60: icmp_seq=1 ttl=56 time=76.565 ms

64 bytes from 104.103.202.60: icmp_seq=2 ttl=56 time=53.358 ms

ping 104.103.202.60

PING 104.103.202.60 (104.103.202.60): 56 data bytes

64 bytes from 104.103.202.60: icmp_seq=0 ttl=56 time=17.981 ms

64 bytes from 104.103.202.60: icmp_seq=1 ttl=56 time=17.399 ms

64 bytes from 104.103.202.60: icmp_seq=2 ttl=56 time=86.598 ms

64 bytes from 104.103.202.60: icmp_seq=3 ttl=56 time=53.091 ms

64 bytes from 104.103.202.60: icmp_seq=4 ttl=56 time=54.445 ms

Try opening up a terminal on the Ensemble box and PING www.intersystems.com. If that works then we know that the network is ok.

Good Luck

Iain

Comment by Antonio Garcia ... 3 August 2018

not working but ping could be blocked in a firewall...and when I tried from my local PC, ping did not work either...

I could be wrong but I am pretty sure is something blocking that connection but maybe based on the protocol...not sure...what I cannot understand is why browse to the web site works but httprequest not.

Thanks

Comment by Jeffrey Drumm 3 August 2018

It's possible that a proxy is in use for web access. Browsers can generally auto-discover and configure for those, but other web clients can't ... at least most of the time.