Question
Antonio Garcia ... · Aug 1, 2018

Net.HttpRequest - blank response?

#SOAP #Caché

Hi,

I am trying to create a script to connect to a website and extract all the content of the page. And in future to connect to an API.

Set httprequest=##class(%Net.HttpRequest).%New()
DO httprequest.Get("http://www.intersystems.com",2)
Write ">"_httprequest.HttpResponse 

But the output is a wonderful blank HttpResponse

Any ideas?

Thanks

0
0 1,492
Discussion (13)3
Log in or sign up to continue
Robert Cemper · Aug 1, 2018

There are some mistakes.

#1 the links should be "HTTPS://www.intersystems.com" and you didn't set a ssl/tls config.

if you use 

Set sc=httprequest.Get("http://www.intersystems.com",2)
Do $system.OBJ.DisplayError(sc)

you get ERROR #6159: ===> SSL missing

#2 HttpResponse is an ObjectReferce not a Property

set res=httprequest.HttpResponse
ZW res 
res=<OBJECT REFERENCE>[3@%Net.HttpResponse]
+----------------- general information ---------------
|      oref value: 3
|      class name: %Net.HttpResponse
| reference count: 3
+----------------- attribute values ------------------
|    ContentBoundary = ""
|        ContentInfo = ""
|      ContentLength = 178
|        ContentType = "text/html"
|               Data = "4@%Stream.GlobalCharacter"   ;;; here is your reply
|Headers("CONNECTION") = "keep-alive"
|Headers("CONTENT-LENGTH") = 178
|Headers("CONTENT-TYPE") = "text/html"
|    Headers("DATE") = "Wed, 01 Aug 2018 15:25:05 GMT"
|Headers("LOCATION") = "https://www.intersystems.com/"
|  Headers("SERVER") = "nginx"
|  Headers("X-TYPE") = "default"
|        HttpVersion = "HTTP/1.1"
|       ReasonPhrase = "Moved Permanently"
|         StatusCode = 301
|         StatusLine = "HTTP/1.1 301 Moved Permanently"
+-----------------------------------------------------

The content is in a Stream!! 
So Write is totally inappropriate to show it. Instead:

do res.OutputToDevice()   ;;;or similar

HTTP/1.1 301 Moved Permanently
CONNECTION: keep-alive
CONTENT-LENGTH: 178
CONTENT-TYPE: text/html
DATE: Wed, 01 Aug 2018 15:25:05 GMT
LOCATION: https://www.intersystems.com/
SERVER:nginx
X-TYPE: default
 
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
1 0
Antonio Garcia ... · Aug 2, 2018

Yo are absolutely right although I am still confused.

I copied httprequest.Get("http://www.intersystems.com",2) from one example found in the documentation so I was expected to work. Also, to get data from a website don't think you have to use https or SSL right? I tried with different APIs in other applications and just passing the right URL I was able to retreive the data.

About httprequest.HttpResponse  you are right again... I was using httprequest.HttpResponse.Data and outputtodevice but I was having an error as the object did not exist...so I tried to check if the object was really created or not...

What would be the right script to connect then? I am trying to get data from this API https://api.postcodes.io/postcodes/

Thanks

0 0
Antonio Garcia ... · Aug 3, 2018

Thanks for your help... I think the problem I am having is not with my code but with ensemble or IT. Apparently somehow the connection to external content is blocked. I can access to the websites with no issues from web browsers but when trying to access from studio or test SSL from ensemble...error!!

Is there any security configuration to block that kind of access in ensemble?

Thanks

0 0
Antonio Garcia ... · Aug 2, 2018

I have tried to created an SSL/TLS configuration but not sure why is not working:

It does not matter what I try to test it...always returns the same error:

ERROR #989: SSL connection failed, make sure server address and port (not url) is specified

I tried "intersystems.com" 443, "postcodes.io" 443...nothing

Those sites I am trying to connect to....don't really need anything special to connect...just url

Thanks

0 0
Robert Cemper · Aug 2, 2018

You have to test on server  = "www.intersystem.com"  not just domain.
443 is ok

 
USER>s req=##class(%Net.HttpRequest).%New()
USER>s req.SSLConfiguration="SSL"
USER>set sc=req.Get("https://www.intersystems.com")
USER>zw sc
sc=1
USER>set data=req.HttpResponse.Data
USER>do data.OutpuToDevice()
>>>>    lot of content  <<<<
0 0
Antonio Garcia ... · Aug 3, 2018

Thanks for the help:

1. I have tried from the server where ensemble is running. In IE, I can access www.intersystems.com but when trying from studio...nothing blank response...all from the same matchine!

2. I tried that and yes...ERROR #6059: Unable to open TCP/IP socket to server www.intersystems.com:80 so I think you are right but I cannot believe the problem is no visibility when in IE I can access the website with no issues.

0 0
Antonio Garcia ... · Aug 3, 2018

I have tried by IP, apparently 38.111.0.52

so  in IE:  http://38.111.0.52:80 => ok, https://38.111.0.52:443 => ok

but using:

Set httprequest=##class(%Net.HttpRequest).%New()
Set httprequest.Server = "38.111.0.52"
Set httprequest.Port=80
set status = httprequest.Get("/",2)
DO $system.OBJ.DisplayError(status)

ERROR #6059: Unable to open TCP/IP socket to server 38.111.0.52:80

I have no idea what's going on

0 0
Iain Bray · Aug 3, 2018

Hi,

So I checked that the web site will receive a ping, some don't for security purposes, and it seems ok:

ping www.intersystems.com

PING e9933.g.akamaiedge.net (104.103.202.60): 56 data bytes

64 bytes from 104.103.202.60: icmp_seq=0 ttl=56 time=80.280 ms

64 bytes from 104.103.202.60: icmp_seq=1 ttl=56 time=76.565 ms

64 bytes from 104.103.202.60: icmp_seq=2 ttl=56 time=53.358 ms

ping 104.103.202.60

PING 104.103.202.60 (104.103.202.60): 56 data bytes

64 bytes from 104.103.202.60: icmp_seq=0 ttl=56 time=17.981 ms

64 bytes from 104.103.202.60: icmp_seq=1 ttl=56 time=17.399 ms

64 bytes from 104.103.202.60: icmp_seq=2 ttl=56 time=86.598 ms

64 bytes from 104.103.202.60: icmp_seq=3 ttl=56 time=53.091 ms

64 bytes from 104.103.202.60: icmp_seq=4 ttl=56 time=54.445 ms

Try opening up a terminal on the Ensemble box and PING www.intersystems.com. If that works then we know that the network is ok.

Good Luck

Iain

0 0
Antonio Garcia ... · Aug 3, 2018

not working but ping could be blocked in a firewall...and when I tried from my local PC, ping did not work either...

I could be wrong but I am pretty sure is something blocking that connection but maybe based on the protocol...not sure...what I cannot understand is why browse to the web site works but httprequest not.

Thanks

0 0
Jeffrey Drumm · Aug 3, 2018

It's possible that a proxy is in use for web access. Browsers can generally auto-discover and configure for those, but other web clients can't ... at least most of the time.

0 0
Iain Bray · Aug 3, 2018

Apologies regarding the DNS name. I've just remembered that you're trying to get a response from 'www.intersystems.com' so the DNS wouldn't have an effect. What you should check is that the Ensemble instance has access to the internet.

There isn't anything in Ensemble that will prevent access, it might be something at the O/S or network level.

Iain

0 0
Iain Bray · Aug 3, 2018

Ok, so a couple of things to check:

1. When you say that you can connect from a browser but not from Ensemble, is the browser that you're using on the same computer as Ensemble?

2. Instead of a 'do httprequest.Get(...)' use 'set status=httprequest.Get(...)' and then inspect the value of 'status'. You can do 'do $system.OBJ.DisplayError(status)' in order to get a readable output. If the result is that you cannot open a TCP/IP socket then Ensemble cannot see the other computer. Otherwise the status should give you more information.

Are you addressing the other computer by its IP address or its DNS name? If you're using the DNS name then try using the IP address. If that works then edit your 'hosts' file to put the name and IP address in there and go back to using the DNS name to get to the other computer.

Let me know how you get on.

Iain

0 0
Iain Bray · Aug 2, 2018

This will work. The second parameter of '2' indicates test mode and will write out the response to the current device

set httprequest = ##class(%Net.HttpRequest).%New()
set httprequest.Server = "www.intersystems.com"
set status = httprequest.Get("/",2)

You should see something like this:

HTTP/1.1 301 Moved Permanently
CONNECTION: keep-alive
CONTENT-LENGTH: 178
CONTENT-TYPE: text/html
DATE: Thu, 02 Aug 2018 10:21:56 GMT
LOCATION: https://www.intersystems.com/
SERVER: nginx
X-TYPE: default
 
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

Alternatively you can do this:

Setup an SSL/TLS Configuration. This is very simple and doesn't have any values apart from a name. Mine is called 'test'

Then use the following code:

set httprequest = ##class(%Net.HttpRequest).%New()
set httprequest.Server = "www.intersystems.com"
set httprequest.SSLConfiguration = "test"
set httprequest.Https = 1
set status = httprequest.Get("/")
if status {
     do httprequest.HttpResponse.Data.Rewind()
     do {
          write httprequest.HttpResponse.Data.Read()
     } while 'httprequest.HttpResponse.Data.AtEnd
}

0 0