New post
AllAccepted answers
Benjamin De Boe · Jan 5, 2017

Hi Benjamin,

The (patented) magic of iKnow is the way how it identifies concepts in sentences and happens in a library shipped as a binary, which we refer to as the iKnow engine and is used by both the iKnow APIs and iFind indices. Most of what happens with that engine's output is not nearly as much rocket science and as Eduard indicated, its COS source code can usually be consulted for clues on how it works if you're adventurous. 

The two options of the GetSimilar() query both work by looking at the top concepts of the reference source and look for other sources that have them as well, using frequency and dominance for weighting in-source relevance for the two options respectively. So not much rocket science and only support for full matches at this point. 

This said, iKnow offers you the building blocks to build much more advanced things, quite possibly inspired by your academical research, leveraging the concept level that is unique to iKnow in identifying what a text is really about. For example, you can build vectors containing entity frequency or dominance and look for cosine similarity in this vector space, or you can leverage topic modelling, but many of these will require quite a bit of computation and actual result quality may depend a bit on the nature of the texts you're dealing with, which is why we chose to stick to very simple things in the kit for now.

However, you can find two (slightly) more advanced options in demos we have published online:

  • The iKnow Investigator demo is part of your SAMPLES namespace and offers, next to the SourceAPI:GetSimilar() option, also an implementation that builds a dictionary from your reference source, matches it against your target sources and looks for the ones with the highest aggregated match score, which accounts for partial matches as well.
  • In the iFind Search Portal demo, the bottom of the record viewer popup displays a list of similar records as well. This one is populated based on a bit of iFind juggling implemented in the Demo.SearchPortal.Utils class, leveraging full entity matches only, but easy to extend to weigh in words as well.

In both cases, there's a myriad of options to refine these algorithms, but all at a certain compute cost, given the high dimensionality introduced by the iKnow entity (and actually even word) level. If you have further ideas or, better yet, sample code to achieve better similar document lists, we'd be thrilled to read about it here on the community ;o)

Thanks,
benjamin

Benjamin De Boe · Dec 22, 2016

Hi Edward,

the thing that comes closest here would be the %iKnow.Queries.SourceAPI:GetSimilar() query, which for a certain seed document, looks for the most similar ones in a domain, optionally constrained by a filter object. The results of that query include a figure like the one you're looking for, expressing how many entities were new in the seed document vs the corpus it's comparing against. Although that that particular calculation isn't available as an atomic function, a simple way to get to what you want would be to use the %iKnow.Filters.SourceIdFilter and just compare against an individual document.

If you prefer to write more code :o), you can just look up the entities in the one document and compare them against those in the others through the %iKnow.Objects.EntityInSourceDetails SQL projection.

Regards,

benjamin

Benjamin De Boe · Nov 3, 2016

I sometimes like to call myself Kyle on Thursdays that feel like Mondays ;o) 

 

Like any operator whose name starts with a %, it's an InterSystems-specific one, so both %CONTAINS and %FIND were added to the standard for specific purposes. %CONTAINS has been part of Caché for a long while and indeed offers a simple means to directly refer to your %Text field. So, you don't need to know the name of your index, but you do need to make sure your text field is of type %Text, so there's still a minimum of table metadata you need to be aware of.

In the case of %FIND, we're actually leveraging the %SQL.AbstractFind infrastructure, which is a much more recent addition that allows leveraging code-based filters in SQL. In this case, the code-based filter implementation is provided by the %iFind infrastructure. Because this AbstractFind interface is, what's in a name, abstract, it needs a bit of direction in the form of the index name to wire it to the right implementation, which in our case comes as the index name. As the AbstractFind interface is expected to return a bitmap (for fast filtering), it also needs to be applied to an integer field, typically the IDKey of your class. So, while it offers a lot of programmatic flexibility to implement optimized filtering code (cf this article on spatial index), it does require this possibly odd-looking construct on the SQL side. But that's easily hidden behind a user interface, of course.

Benjamin De Boe · Nov 3, 2016

A bit of a long read, but very nice illustration of how iKnow's bottom up approach allows you to work with the full concepts as coined by the author rather than a top-down approach relying on predefined lists of terms. If this is only their bachelor thesis, I'm looking forward to see their master thesis :-)

Thanks for sharing Otto!

Benjamin De Boe · Nov 3, 2016

What is the attribute of your texts that you want to categorize? When building TC models on top of iKnow domains, as Chip indicated, the easiest way is to do this based on a metadata field using the %LoadMetadataCategories method on your builder object. The more manual method is using these %AddCategory methods you're using, but they require a filter specification as the second argument (you passed "") to identify which records correspond to the category you're adding. That's what's making it a training set, a set of texts for which the outcome (category) is known, so the TC infrastructure can try to find corellations to exploit. 

Separately, 11 records is a very small training set. I would not expect to find much statistically relevant information to exploit. Even when you're building a rule-based model manually, it'll be barely enough to validate your model. So probably it's worth trying to get hold of (much) more data to start with.

I've uploaded a tutorial we prepared for a Global Summit academy session on the subject in a separate thread (could not attach it to an existing one). That may give you a better idea of what the infrastructure can be used for and also takes you through the GUI, which may be more practical than the code-based approach you're referring to.

Benjamin De Boe · Aug 10, 2016

It's indeed tempting to just stuff interfaces like this in the kit, but it goes a bit beyond the objectives of pure system management interfaces that we'd typically pack with Caché. Also, in the specific case of this Dictionary Builder demo, it uses the programmatic APIs to create dictionaries (requiring allowCustomUpdates=true) and does not update the domain definition itself. We're actually working on making that a smoother process, so when that gets to a point where it can support the interactions implemented in this GUI (and when AngularJS becomes part of our kit), we can reconsider it.

Benjamin De Boe · May 8, 2016

Hi Terri,

[a little late perhaps (and we spoke briefly in Phoenix as well), but for the sake of completeness, here's your response]

The iKnow Architect page only shows domains based on domain definitions, subclasses of %iKnow.DomainDefinition. The ones created "the old way" (programmatically) through the %iKnow.Domain class do not carry the declarative information about where to load data from, hence the majority of the Architect GUI wouldn't be available on those.

As to the metadata loading question: I'm not sure which part of the script you're referring to was taking care of that metadata loading, but maybe this comes back to your question in a separate thread.

regards,

benjamin

Benjamin De Boe · Apr 22, 2016

Hi Orion,

I haven't heard about index data simply disappearing. There should be no reason for that other than calls to %PurgeIndices() or manually dropping the globals containing the data (including ^ISC.IF* ones containing entities and words shared across the namespace). Another potential issue may arise when importing (at a global level) only either the index or the shared data, resulting in them no longer being in sync. Any chance any of that could have happened?

For the missing class, that might be due to a class import as well, after which not all related classes were recompiled to reflect the changes.

Which version are you working on? Some of those recompiling issues may have been addressed in recent versions.

Perhaps the WRC is a better place to get the appropriate follow-up for specific issues like this one. 

regards,
benjamin

Benjamin De Boe · Mar 29, 2016

Hi Benjamin,

the default algorithm indeed won't return scores for each record, but will only make the calculation for all records that contain at least a decent number of entities that are relevant in the source document. You can indeed simply approximate the other documents' score by taking 0.

For your specific use case, you may want to take a look at the text categorization infrastructure. I've posted a tutorial on the topic here.

regards,
benjamin

Benjamin De Boe · Mar 18, 2016

Hi Julie,

For XEP, the XEP guide in the product documentation is probably the best starting point. For iKnow, you can take a look at this video playlist introducing the technology. 

As you may know, InterSystems is also developing a new platform specifically aimed at big data use cases. Part of this new platform will be support for the UIMA standard, as a broader framework for dealing with unstructured data than iKnow's natural language processing alone, allowing you to combine it with third-party or custom utilities. Please send me an email if you'd like to discuss your big data project in more detail.

 

thanks,
benjamin

Benjamin De Boe · Mar 14, 2016

Hi Jack,

there's no need to normalize your search strings, as it's take care of automatically as part of executing your search when appropriate.

When you use DELETE FROM in SQL, or ##class(Your.Table).%DeleteExtent() in COS, the associated iFind indices' data will be erased as well. To drop just the indices data, use ##class(Your.Table).%PurgeIndices() (cf class ref for refinements). Note that, unless you are using index-local storage (new feature in 2016.1), the words and entities tables will not be wiped as they are shared between all iFind indices in your namespace (somewhat conserving space and indexing efficiency).

iFind can calculate a score representing how well a record satisfies a search string, largely based on TFIDF (although it'll leverage the more refined dominance scores for entities when it can). This is also new in 2016.1. See https://community.intersystems.com/code/ifind-search-portal for an example.

 

regards,
benjamin

Benjamin De Boe · Mar 14, 2016

Hi Evgeny, Jack,

 

Ranking is new in 2016.1, and will indeed allow you to retrieve a score expressing how well a record matches a search string. A packagename.tablename_indexnameRank function gets automatically generated when  you compile your class with an iFind index and can be invoked as follows:

SELECT %ID, 
Title,
FullText,
SomePackage.TheTable_MyIndexRank(%ID, 'cocktail* OR (hammock AND NOT bees)')
FROM SomePackage.TheTable
WHERE %ID %FIND search_index(MyIndex, 'cocktail* OR (hammock AND NOT bees)')
ORDER BY 4 DESC

There are no public demo servers exposing this functionality at this time.

 

regards,
benjamin

Benjamin De Boe · Mar 7, 2016

Hi Benjamin,

If you just want a SQL prompt, you can open one from the COS prompt by calling "do $system.SQL.Shell()", or use the SQL page in the system management portal which you can find under the "system exploration" menu.

The SQL lister and loader functionality is meant to populate your domain (rather than query it), but should no longer be invoked directly. Managing a domain can be taken care of much more easily through domain definitions, which can be configured through the iKnow Architect as from 2016.1. But I see you're already using one, otherwise you wouldn't have seen that error message (which BTW informs you that this domain definition is configured not to allow any build/config operation other than through the domain definition itself, which is the default setting for domain defs).

 

If you want to achieve the same result through COS:

write ##class(%iKnow.Queries.EntityAPI).GetTop(.result, domainID)

zwrite result

 

regards,

benjamin

Benjamin De Boe · Mar 3, 2016

Hi Benjamin (sounds like a conversation amongst just Benjamins now!),

 

The knowledge portal demonstration interface you find in the %iKnow.UI package (which gets a significant visual overhaul in 2016.3) is written using InterSystems' Zen technology, a web development framework that helps you combine client-side JavaScript and server-side Caché ObjectScript to build web applications. If you're good with PHP and/or JavaScript, there's no strict need to dig into Zen to build an iKnow-powered application. You can either use ODBC to connect to Caché and use SQL as in the above examples to query an iKnow domain, or you can build a simple REST service on top of iKnow (in Caché ObjectScript) and query that from your PHP/JavaScript code. We'll be releasing an out-of-the-box REST interface with 2016.3, but it's no rocket science to build one that fits your needs on earlier versions. If you already have an ISC sales engineering contact (none of them called Benjamin, unfortunately ;o) ), we can work together to get you up and running.

 

FYI, this github repo contains a simple iKnow demo application written with AngularJS and a REST interface. It's technically speaking a CSP page (yet another ISC web technology at a lower level than Zen), but could have been a straight HTML page.

 

 

Regards,

Benjamin

Benjamin De Boe · Mar 2, 2016

Hi Benjamin,

 

if you're familiar with Caché ObjectScript, that's the easiest way to work with iKnow. For example, the script below will add two short "sources" (documents) to your domain and then query the top concepts:

set domainID = 1, domainName = $system.iKnow.GetDomainName(domainID)

write $system.iKnow.IndexString(domainName, "123", "This is a first piece of text to be added to your iKnow domain!")

write $system.iKnow.IndexString(domainName, "234", "This is the second piece of text to be added to your iKnow domain! And guess what, it's an even more inspirational one!")

write ##class(%iKnow.Queries.EntityAPI).GetTop(.result, domainID)

zwrite result

 

For a good start with iKnow, take a look at this iKnow video and the next ones in the iKnow playlist.

 

If you prefer to work with SQL and loaded your domain through the iKnow Architect in the management portal, you can invoke those same query APIs through either of the following calls (for domain ID = 1):

CALL %iKnow_Queries.EntityQAPI_GetTop(1)

SELECT * FROM %iKnow_Queries.EntityQAPI_GetTop(1)

Benjamin De Boe · Feb 26, 2016

Hi Benjamin,

in order to enable a web application to use iKnow, you need to check the "iKnow" box in the SMPs Web Application management page (System Administration > Security > Applications > Web Applications). This was mentioned in the release notes of the first version introducing the stricter security policies (or at least the routine behind the checkbox is), but isn't mentioned prominently enough in the iKnow guide. We'll look into that.

This is actually only related to the web interfaces, so Atelier is not involved here. To create iKnow domain definitions through the management portal, look for the "iKnow Architect" in the SMP menu for iKnow.

 

regards,

benjamin