#InterSystems Natural Language Processing (NLP, iKnow)
7
InterSystems IRIS Natural Language Processing (NLP), formerly known as iKnow, allows you to perform text analysis on unstructured data sources in a variety of natural languages without any prior knowledge of their content. It does this by applying language-specific rules that identify semantic entities. Because these rules are specific to the language, not the content, NLP can provide insight into the contents of texts without the use of a dictionary or ontology. Learn more.
Last week at the InterSystems BeNeLux Symposium, we announced the publication of the InterSystems iKnow Natural Language Processing technology to Open Source. This enables developers and researchers to take advantage of this unique technology at any level of their application stack, workflow or experiment.
To achieve optimized AI performance, robust explainability, adaptability, and efficiency in healthcare solutions, InterSystems IRIS serves as the core foundation for a project within the x-rAI multi-agentic framework. This article provides an in-depth look at how InterSystems IRIS empowers the development of a real-time health data analytics platform, enabling advanced analytics and actionable insights. The solution leverages the strengths of InterSystems IRIS, including dynamic SQL, native vector search capabilities, distributed caching (ECP), and FHIR interoperability. This innovative approach directly aligns with the contest themes of "Using Dynamic SQL & Embedded SQL," "GenAI, Vector Search," and "FHIR, EHR," showcasing a practical application of InterSystems IRIS in a critical healthcare context.
InterSystems has decided to stop further development of the InterSystems IRIS Natural Language Processing, formerly known as iKnow, technology and label it as deprecated as of the 2023.3 release of InterSystems IRIS. InterSystems will continue to support existing customers using the technology, but does not recommend starting new development projects outside of the core text exploration use cases it was originally designed for.
As said in the previous article about the iris-fhir-generative-ai experiment, the project logs all events for analysis. Here we are going to discuss two types of analysis covered by analytics embedded in the project:
This is the second article in a series on iKnow demo applications, showcasing how the concepts and context provided through iKnow's unique bottom-up approach can be used to implement relevant use cases and help users be more productive in their daily tasks. Last week's article discussed the Knowledge Portal, a straightforward tool to browse iKnow indexing results.
This week, we'll look into the Set Analysis demo, a slightly more advanced application where you'll be using the concepts identified by iKnow to organize your content into sets of documents. The original version of this demo was developed by Danny Wijnschenk & Alain Houf for an academy session at GS2015, but the app has evolved significantly since then.
https://www.youtube.com/embed/n-TL_Wh7Tv4 [This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]
I'm working in an application that uses %SIMILARITY to find matches among a set of documents that vary greatly in length. It's generally good but I've noticed issues with ranking short partially-matching documents over longer documents that match the search string entirely.
I'm in a process of acquiring a corpus of documents on educational courses.
For example there is an educational course called "OOP" and it can have documents from 2008, 2009, ... 2016 etc. And there are a lot of these courses, each one with programs from different years (hopefully)
So 1 document is 1 programm of one course for one year.
I want to calculate how much does a course changes per year.
I have a SQL Query using %iFind.Highlight which returns text highlighting certain words and phrases. %iFind.Highlight seems to remove cr/lf from the returned text.
Here's my query
ClassMethod Search(pSessionId As %String, pSearchString As %String) As %String { set tTags="<span style='background-color:yellow;'>" &sql( SELECT %iFind.Highlight(Text , :pSearchString , , :tTags) into :results FROM SSA_OCR.TempSearchable where sessionId = :pSessionId) quit results }
I've been working with SQL using an iFind index to search text. Using the %iFind.Highlight function in my SELECT statement I can get text back that highlights the found words using <b> and </b>
I am aware that using ##class(%iFind.Utils).Highlight, I can pass a parameter to override the <b> tag and use instead a <span> tag with style to change the background color of the found words.
Is there a way to override the <b> tag from a SQL statement?
In Cache'/Ensemble, by specifying the objectsPackage parameter, dictionaries (and other objects) get projected to tables that can be accessed by SQL queries.
But in IRIS (IRIS for UNIX (Ubuntu Server LTS for x86-64 Containers) 2019.1 (Build 507U) Mon Feb 25 2019 13:47:16 EST), when I created a dictionary with ##class(%iKnow.Matching.DictionaryAPI).CreateDictionary(), it does not get projected to a table.
The class APIs correctly retrieve information about this dictionary.
Am I missing something with IRIS, or is there any issues about this?
The InterSystems DBMS has a built-in technology for working with non-structured data called iKnow and a full-text search technology called iFind. We decided to take a dive into both and make something useful. As the result, we have DocSearch — a web application for searching in InterSystems documentation using iKnow and iFind.
https://www.youtube.com/embed/HDTij5GS_qQ [This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]
I have iKnow domain of forum posts, their full text is an iKnow data, and each post also has a number of views as a metadata field.
I want to get a sum of views by concept. Let's say I have a concept called "TESTEST" and there are 10 sources that have this concept. Each source has some views. I want to get views total - impact of this concept so to say.
What's the best iKnow architecture for this use case?
I am trying to create an iKnow domain programmatically like:
Set dom = ##class(%iKnow.Domain).%New("TestDom")
Do dom.SetParameter("DefaultConfig", "MyConfiguration")
Set sc = dom.%Save()
...
Although "MyConfiguration" sets the language to "ja", i.e. Japanese, it doesn't seem to be respected, and what I see on the top right pane in Knowledge Portal is related concepts, instead of proximity profiles, which I expect to see in Japanese language mode.
Also resulting segmentation of sentences looks to be it is in English mode.
https://www.youtube.com/embed/7Y5mdY3M3vo [This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]
By update
Open Exchange app
