New post
Question
· Jun 13, 2018

Get all sources from domain except whose that contain blacklisted entities

#Caché #InterSystems Natural Language Processing (NLP, iKnow)

I have an iKnow domain and a blacklist called Ads.

I want to get all sources that do not contain entities from Ads list.

Currently I'm doing it like this:

ClassMethod NoAds()
{
    #dim domainid As %Integer = 1
    
    set filterNot = ##class(%iKnow.Filters.GroupFilter).%New(domainid,, $$$YES)
    #dim blackListId As %Integer = ##class(%iKnow.Utils.MaintenanceAPI).GetBlackListId(domainid, "Ads")
    set sc = ##class(%iKnow.Utils.MaintenanceAPI).GetBlackListElements(.blackList,..#DomainId, blackListId)
    set filterAds = ##class(%iKnow.Filters.ContainsEntityFilter).%New(domainid, .blackList)
    set sc = filterNot.AddSubFilter(filterAds)
    
    #dim page As %Integer = 1
    #dim pagesize As %Integer = 0
    set sc = ##class(%iKnow.Queries.SourceAPI).GetByDomain(.result, domainid, page, pagesize, filterNot)
    zw result
}

To test results I set GroupFilter to not Negated so just GroupFilter:%New(domainid), I got this results:

 > zw result
....
result(9993)=$lb(977,"974")
result(9994)=$lb(642,"635")
result(9995)=$lb(520,"521")
result(9996)=$lb(473,"472")
result(9997)=$lb(284,"286")
result(9998)=$lb(218,"218")

But if I execute this SQL:

SELECT
Id, Text
FROM Test.Data
WHERE Id In (218, 284, 286, 472, 473)

I don't get results that contains entities from Ads blacklist.

I also tried to remove GroupFilter altogether and pass ContainsEntityFilter directly but got the same results

Discussion (2)0
Log in or sign up to continue
Benjamin De Boe · Jun 14, 2018

Hi Eduard,

looking at your code, there seem to be a few small things that may each contribute to not seeing the results you were expecting:

  1. the MaintenanceAPI:GetBlackListElements() call returns its results as result(n) = $lb(id, string) with n just an incrementing integer representing the row number. At the other end, the ContainsEntityFilter expects array(string) or a $listbuild(string1, string2, ...). So your filter might be selecting sources containing the strings "1", "2", etc
  2. SourceAPI:GetByDomain() returns result(n) = $lb(sourceID, externalID). That source ID is an internally generated integer ID that has no links to your source table Text.Data. The external ID is typically composed of what you selected as group field and identifier field when loading from a SQL table. So depending on how you set up your domain, that may indeed be the ID field of your Text.Data table. It looks like you have the "simple external IDs" feature switched on, which is why your external IDs only consist of the identifier field, making things indeed easier (but usually only useful/safe when loading from a single table!). Note that this is slightly different for DeepSee-managed domains, where the source ID equals the external ID and corresponds to DeepSee's fact ID, but ignore this confusing comment when not using DeepSee
  3. Finally, and likely irrelevant, you're passing in $$$YES when initializing filterNot. I'm not sure where you're loading that macro from, but that should be a %Boolean with a value of 1 to work as expected, where a string value would translate to a %Boolean with value 0.

Hope this helps,

benjamin

1 0