Eduard Lebedyuk · Nov 24, 2020 go to post

I encounter this issue fairly often, but I need not a complete documentaiton but rather Interoperability production documentation.

As all the class info is also available as a %Dictionary package I just query it and generate XLSX.

Here are some queries to get started (but they usually need to be adjusted on per-project basis). Also queries should be rewritten to use SubclassOf proc instead of the current matching. Also I'm not sure why I don't pass filestream directly. That also  needs to be fixed.

 

Queries

/// Generate docs
Class Utils.Doc
{

/// Generate Docs Tables
/// w $system.Status.DisplayError(##class(Utils.Doc).generate(,"defaultSettings"))
/// w $system.Status.DisplayError(##class(Utils.Doc).generate(,"productionClasses"))
/// w $system.Status.DisplayError(##class(Utils.Doc).generate(,"productionSystemClasses"))
/// w $system.Status.DisplayError(##class(Utils.Doc).generate(,"productionSystemClassesSettings"))
/// w $system.Status.DisplayError(##class(Utils.Doc).generate(,"productionClassesSettings"))
/// w $system.Status.DisplayError(##class(Utils.Doc).generate(,"productionItemsSettingsCall"))
/// w $system.Status.DisplayError(##class(Utils.Doc).generate(,"productionItems"))
/// w $system.Status.DisplayError(##class(Utils.Doc).generate(,"modelClasses"))
/// w $system.Status.DisplayError(##class(Utils.Doc).generate(,"modelPropeties"))
ClassMethod generate(filename As %String = {##class(%File).TempFilename("xlsx")}, query As %String, args...) As %Status
{
    #dim sc As %Status = $$$OK
    #dim stream As %Stream.TmpCharacter = ##class(%Stream.TmpCharacter).%New()
    
    set sql = ..getSQL(query)
    set sc = ##class(util.XLSX).generateStreamFromSQL(.stream, sql, args...)
    quit:$$$ISERR(sc) sc
    
    set file = ##class(%FileCharacterStream).%New()
    set:$p(filename,".",*)'="xlsx" filename = filename _".xlsx"
    set file.Filename = filename
    
    set sc = file.CopyFrom(.stream)
    quit:$$$ISERR(sc)
    
    set sc = file.%Save()
    quit sc
}

ClassMethod getSQL(name) As %String
{
    #dim sc As %Status = $$$OK
    
    set class = $case($l(name, ":"), 2:$p(name, ":"), :$classname())
    set queryName = $p(name, ":", *)
    
    if ##class(%Dictionary.QueryDefinition).IDKEYExists(class, queryName) {
        set query = ##class(%Dictionary.QueryDefinition).IDKEYOpen(class, queryName,,.sc)
        throw:$$$ISERR(sc) ##class(%Exception.StatusException).CreateFromStatus(sc)
        set sql = query.SqlQuery
    } elseif ##class(%Dictionary.XDataDefinition).IDKEYExists(class, queryName) {
        #dim stream As %Stream.Object = ##class(isc.util.XmlUtils).getClassXData(class, queryName)
        set sql = stream.Read($$$MaxLocalLength)
    } else {
        throw ##class(%Exception.StatusException).CreateFromStatus($$$ERROR($$$GeneralError, $$$FormatText("Class %1 does not have Query or XData with name %2", class, queryName)))
    }
    
    set:(removeNL = $$$YES) sql = $replace(sql, $$$NL, " ")
    return sql
}

/// Generic settings for all Interoperability BH
Query defaultSettings() As %Query
{
SELECT
  prop.name Setting,
  TRIM('"' FROM MAX(prop.InitialExpression)) "Default",
  LIST(prop.parent) Classes,
  MAX(prop.Description) Descrition
FROM %Dictionary.PropertyDefinition prop
  JOIN %Dictionary.CompiledParameter par ON par.parent = prop.parent AND par.Name = 'SETTINGS'
  JOIN %Dictionary.CompiledClass cls ON prop.parent = cls.Name
WHERE (cls.Super LIKE '%Ens.Host%' OR cls.Name = 'Ens.Host' OR cls.Name = 'Ens.BusinessProcessBPL') AND
      par."_Default" LIKE '%' || prop.Name || '%'
GROUP BY prop.Name
ORDER BY 1
}

/// Production BHs
Query productionClasses() As %Query
{
SELECT
  Name Class,
  Description Descrition
FROM %Dictionary.CompiledClass cls
WHERE Name LIKE 'production.%'
      AND Super In ('Ens.BusinessProcessBPL', 'Ens.BusinessService', 'Ens.BusinessOperation', 'Ens.BusinessProcess')
ORDER BY 1
}

/// System BHs
Query productionSystemClasses() As %Query
{
SELECT
  Name Class,
  Description Descrition
FROM %Dictionary.CompiledClass cls
WHERE Name In ('util.SQLInboundAdapter', 'EnsLib.SQL.InboundAdapter', 'EnsLib.JavaGateway.Service', 'EnsLib.Workflow.Operation', 'Ens.InboundAdapter')
ORDER BY 1
}

/// System settings for BHs
Query productionSystemClassesSettings() As %Query
{
SELECT
  prop.parent Class,
  prop.name Setting,
  TRIM('"' FROM prop.InitialExpression) "Default",
  prop.Description Descrition
FROM %Dictionary.PropertyDefinition prop
  JOIN %Dictionary.CompiledParameter par ON par.parent = prop.parent AND par.Name = 'SETTINGS'
  JOIN %Dictionary.CompiledClass cls ON prop.parent = cls.Name
WHERE cls.Name IN ('util.SQLInboundAdapter', 'EnsLib.SQL.InboundAdapter', 'EnsLib.JavaGateway.Service', 'EnsLib.Workflow.Operation', 'Ens.InboundAdapter')  AND
      par."_Default" LIKE '%' || prop.Name || '%'
ORDER BY 1,2
}

/// BHs Settings
Query productionClassesSettings(production) As %Query
{
SELECT
  prop.parent Class,
  prop.name Setting,
  TRIM('"' FROM prop.InitialExpression) "Default",
  prop.Description Descrition
FROM %Dictionary.PropertyDefinition prop
  JOIN %Dictionary.CompiledParameter par ON par.parent = prop.parent AND par.Name = 'SETTINGS'
  JOIN %Dictionary.CompiledClass cls ON prop.parent = cls.Name
WHERE cls.Name LIKE :production || '.%' and
      par."_Default" LIKE '%' || prop.Name || '%'
ORDER BY 1,2
}

/// Production BH
Query productionItems(production) As %Query
{
SELECT
  Name Элемент,
  ClassName Class
  /*Comment Comment*/
FROM Ens_Config.Item
WHERE Production = :production
}

/// Get settings for production
Query productionItemsSettingsCall(production) As %Query
{
SELECT *
FROM util.productionItemsSettings(:production)
}

/// Get settings for production
Query productionItemsSettings(production As %String) As %Query(CONTAINID = 0, ROWSPEC = "Элемент:%String,Setting:%String,Цель:%String,Значение:%String") [ SqlName = productionItemsSettings, SqlProc ]
{
}

ClassMethod productionItemsSettingsExecute(ByRef qHandle As %Binary, production As %String) As %Status
{
    set obj = ##class(Ens.Config.Production).%OpenId(production,,.sc)
    quit:$$$ISERR(sc) sc
    do ..clearProductionItems(.obj)
    set qHandle("production") = obj
    set qHandle("item") = 1
    set qHandle("itemCount") = qHandle("production").Items.Count()
    
    set qHandle("setting") = 1
    set qHandle("settingCount") = qHandle("production").Items.GetAt(qHandle("item")).Settings.Count()
    quit sc
}

ClassMethod productionItemsSettingsFetch(ByRef qHandle As %Binary, ByRef Row As %List, ByRef AtEnd As %Integer = 0) As %Status [ PlaceAfter = productionItemsSettingsExecute ]
{
    #dim sc As %Status = $$$OK
    #dim item As Ens.Config.Item = qHandle("production").Items.GetAt(qHandle("item"))
    #dim setting As Ens.Config.Setting = item.Settings.GetAt(qHandle("setting"))

    set AtEnd = 0
    set Row = $lb(item.Name, setting.Name, setting.Target, setting.Value)
    
    if qHandle("setting")<qHandle("settingCount") {
        set qHandle("setting") = qHandle("setting") + 1
    } else {
        if qHandle("item")<qHandle("itemCount") {
            set qHandle("item") = qHandle("item") + 1
            set qHandle("setting") = 1
            set qHandle("settingCount") = qHandle("production").Items.GetAt(qHandle("item")).Settings.Count()
        } else {
            set AtEnd = 1
        }
    }
    
    quit sc
}

/// Remove setting-less hosts
ClassMethod clearProductionItems(ByRef production As Ens.Config.Production)
{
    #dim item As Ens.Config.Item
    for i = production.Items.Count():-1:1 {
        set item = production.Items.GetAt(i)
        do:item.Settings.Count()=0 production.Items.RemoveAt(i)
    }
}

ClassMethod productionItemsSettingsClose(ByRef qHandle As %Binary) As %Status [ PlaceAfter = productionItemsSettingsFetch ]
{
    kill qHandle
    quit $$$OK
}

/// Data model
Query modelClasses() As %Query
{
SELECT
  Name Class,
  Description Descrition
FROM %Dictionary.ClassDefinition
WHERE Name LIKE 'model.%' AND GeneratedBy IS NULL -- AND Name NOT LIKE 'model.ref.%'
}

/// Data model properties
Query modelPropeties() As %Query
{
SELECT
  cls.Name Class,
  prop.Name Свойство,
  prop.Type Тип,
  prop.Description Descrition
FROM %Dictionary.ClassDefinition cls
  JOIN %Dictionary.PropertyDefinition prop ON cls.Name = prop.parent
WHERE cls.Name LIKE 'model.%' AND cls.Name NOT LIKE 'model.ref.%'
}

}
Eduard Lebedyuk · Nov 24, 2020 go to post

$zf functions (3-6 to be precise) cannot call arbitrary libraries but only InterSystems IRIS callout libraries.

You need to write a C library which is an InterSystems IRIS callout library and which calls the dll you need. Here's documentation on that. And here's a sample callout library.

Another approach would be using the CNA community project to call your library. CNA provides an interface for using native C-compatible shared libraries.

Eduard Lebedyuk · Nov 15, 2020 go to post

Looks like a user permissions issue.

question: which user does the Cache-terminal login as, is this different to the win10-services cache.exe settings.

Yes, the terminal works under your OS user, Cache (and InterSystems IRIS) background jobs work under system account user (you can check services - Cache to see the user).

You need to give permissions to access the share to system account user.

Eduard Lebedyuk · Nov 13, 2020 go to post

As I said use object access instead.

I have written a code sample which can be used with Native API for Python (although the first part with directory creation should probably just be called from PYTHON as is if you're on a same machine).

Locals are not supported by Native API for Python.

Eduard Lebedyuk · Nov 13, 2020 go to post

You can either

1. You can use object access from Python so this code can be invoked via Native API for Python:

Set name = "ABC"
Set dir = ##class(%File).SubDirectoryName(##class(%File).ManagerDirectory(), name, 1)
Set sc = ##class(%File).CreateDirectory(dir)
Write $SYSTEM.Status.GetErrorText(sc)

Set db=##Class(SYS.Database).%New()
Set db.Directory = dir
Set sc = db.%Save()
Write $SYSTEM.Status.GetErrorText(sc)

Set dbConfig=##Class(Config.Databases).%New()
Set dbConfig.Directory = dir
Set dbConfig.Name = name
Set sc = dbConfig.%Save()
Write $SYSTEM.Status.GetErrorText(sc)

2. Use SQL via xDBC connection (CREATE DATABASE).

Eduard Lebedyuk · Nov 13, 2020 go to post

1. Use Visual Trace to see message processing times. You can also query this information via SQL (Ens.MessageHeader table).

2. Add $$$TRACE events. The best time counter is $zh. They also can be queried via SQL (Ens_Util.Log table).

Eduard Lebedyuk · Nov 13, 2020 go to post

There's a lot of different searches available in InterSystems IRIS.

Can you show an example search you want explained?

Eduard Lebedyuk · Nov 11, 2020 go to post

Which Service are you referring to?

Haven't seen this setting in default InterSystems IRIS adapters.

Eduard Lebedyuk · Nov 11, 2020 go to post

When you send request from BS to BO use SendRequestSync (and not SendRequestAsync). This way you can get a response back.

Eduard Lebedyuk · Nov 10, 2020 go to post

When I describe InterSystems IRIS, I always start with how it is a multimodel DBMS at its core.

In my opinion that is its main advantage (on the DBMS side). And the data is stored only once. You just choose the access API you want to use.

  • You want some sort of summary for your data? Use SQL!
  • Do you want to work extensively with one record? Use objects!
  • Want to access or set one value and you know the key? Use globals!

More about multi-model approach of InterSystems IRIS.

But InterSystems IRIS is not only a DBMS, there are a lot of features, including:

  • Application engine
  • Integration engine
  • BI and NLP analytics, reports
  • AI and ML
Eduard Lebedyuk · Nov 9, 2020 go to post

I'd mimic %DispatchSetMultidimProperty signature. Shame local cannot be casted into a multi-argument call. Wrap in $g if values can be skipped.

/// do ##class().Test()
ClassMethod Test()
{
    kill ^data
    do ..setValue("val", "key1")
    do ..setValue("val", "key1", "key2", "key3", "key4")
    zw ^data
}

ClassMethod setValue(val, args...)
{
    if args=1 {
        set ^data(args(1)) = val
    } elseif args=2 {
        set ^data(args(1), args(2)) = val
    } elseif args=3 {
        set ^data(args(1), args(2), args(3)) = val
    } elseif args=4 {
        set ^data(args(1), args(2), args(3), args(4)) = val
    }
}
Eduard Lebedyuk · Nov 9, 2020 go to post

Terminal works under your OS user, BPL works under InterSystems user (typically irisusr).

This type of error is often caused by:

  1. IRIS OS user does not have permissions to access the input and/or output file
  2. IRIS OS user does not have required executables in PATH
  3. IRIS OS user does not have required shared libraries in LD_LIBRARY_PATH

To solve issue (1) create input and output files in IRIS Temp directory - a path returned by:

write ##class(%File).NormalizeDirectory(##class(%SYS.System).TempDirectory())

To test for issues (2) and (3) add a $zf call which inits all your dependencies and returns version or some other trivia.

To adjust PATH and LD_LIBRARY_PATH use System Management Portal. Here's some info.

I recommend checking environment variables from inside your process with:

write $SYSTEM.Util.GetEnviron(Name)

For quick testing you can adjust environment variables for the current process with this community utiliy.

Eduard Lebedyuk · Nov 8, 2020 go to post

Strange.

Try excluding IRIS directory from antivirus scanning and reinstall the instance.

Eduard Lebedyuk · Nov 6, 2020 go to post

Here's the code to get capitalized piece of string by number:

/// Get Capitalized piece.
///   str - string to search
///   piece - 1-based piece to return
/// write ##class(test.CustomQuery).GetCapitalizedPiece()
ClassMethod GetCapitalizedPiece(str As %String = "lowerHelloWorldAgainHelloWorldAgainHelloWorldAgainHelloWorldAgain", piece As %Integer = 1) As %String
{
    
    // Only uppercase letters
    set upper = $tr(str, $zcvt(str, "l"))
    
    // Number of uppercase letteres
    set capCount = $l(upper)
    
    // Quit if requested piece can not exist
    quit:piece>capCount ""
    quit:piece<1 ""
    
    // First letter in our capitalized piece
    set startLetter = $e(upper, piece)
    
    // All previous capitalized letters
    set prevLetters = $e(upper, 1, piece - 1)

    // Check if current capitalized letter is a first capitalized letter
    // If not - skip previous capitalized letters
    if '$find(prevLetters, startLetter) {
        set start = $find(str, startLetter) - 1
    } else {
        set count = $l(prevLetters, startLetter) + 1
        set start = 0
        while $i(count,-1) {
            set start = $find(str, startLetter, start)
        }
        set start = start - 1
    }
    
    // Is this capitalized piece the last one?
    if piece=capCount {
        set end = $l(str)
    } else {
        set end = $find(str, $e(upper, piece + 1), start) - 2
    }

    quit $e(str, start, end)
}

Advantages over regexp:

  • ~50 times faster
  • Get any piece, not only the second one (although generating required regexp by piece number seems easy to do)
Eduard Lebedyuk · Nov 6, 2020 go to post

OS user must have Windows registry write access to edit server connections.

Try adding a server as a local Administrator user.

Eduard Lebedyuk · Nov 4, 2020 go to post

4 years later I can safely say that Poolsize can exceed the number of CPU Cores.

The main exception is when a job consumes the entire CPU core. Usually a process mostly waits for network, disk io or some other io or whatever. In that case it's perfectly fine for Poolsize to exceed available CPU Cores - let them wait together. On the other hand if a process consumes a CPU core entirely - in that case Poolsize exceeding CPU cores count won't help.

In general increasing Poolsize overly much can lead to overconsumption of CPU or disk or other resources, which depending on resource concurrency model may (and often does) negatively impact overall performance.

To @alex chang I can recommend monitoring queues count, sql queries, system metrics and cpu/ram/hdd consumption - clearly there's a bottleneck somewhere.

Additionally while Visual Trace is a great tool it does not show everything a process does - only messages sent and received. Is there anything a process does between receiving a request and calling another process?

Finally Monlbl can be used to check which part of the generated BPL code takes a lot of time.

Eduard Lebedyuk · Nov 4, 2020 go to post

IIRC you're doing Tesseract OCR - in that case I would recommend the following architecture:

  • %CSP.REST handler to get requests
  • empty BS to proxy requests
  • BO (PEX or otherwise) to interact with Tesseract  (in goes binary stream or a file name out goes text stream)