Thanks for time and many suggestions.
Have also been brainstorming what is already in the platform that I could leverage to assist speed up functionality.
Approach is to loop over extent to find the next "set record" that has the most additional numbers available that are not previously selected.
I came up with using Bit Strings instead of IRIS lists at language level.
This allows efficient bit operations via BitLogic "OR" operator.
Storing the BitStrings in a calculated property on record insert and update, mitigates recalculating a Bit String from source string when iterating over an extent each time, looking for the next best record to use.
FInally wrapped this up in a Class Query.
Class TOOT.Data.Instrument Extends %Persistent
{
Property NoteList As %String(MAXLEN = 8000, TRUNCATE = 1);
Property NoteListEmbedding As %Embedding(MODEL = "toot-v2-config", SOURCE = "NoteList");
Property NoteListBit As %Binary [ SqlComputed,SqlComputeOnChange = NoteList];
/// Calculates and stores a bit string during record insert or update
ClassMethod NoteListBitComputation(cols As %Library.PropertyHelper) As %Binary
{
set bitString=""
set numlist=cols.getfield("NoteList")
set numsLen=$L(numlist,",")
for i=1:1:numsLen {
set val=$P(numlist,",",i)
continue:val<6
continue:$D(found(val))
Set found(val)=""
Set $Bit(bitString,val)=1
}
return bitString
}
/// Callable query for getting best document based on bitflags
Query BestNoteList(Top As %Integer=5, Accumulate As %Boolean=0) As %Query(ROWSPEC = "ID:%String,Count:%Integer") [SqlProc]
{
}
ClassMethod BestNoteListExecute(ByRef qHandle As %Binary, Top As %Integer=5, Accumulate As %Boolean=0) As %Status
{
Set:Top<1 Top=5
Set qHandle=$LB(Top,0,"")
Quit $$$OK
}
ClassMethod BestNoteListFetch(ByRef qHandle As %Binary, ByRef Row As %List, ByRef AtEnd As %Integer = 0) As %Status
[ PlaceAfter = BestNoteListExecute ]
{
Set qHandle=$get(qHandle)
Return:qHandle="" $$$OK
Set Top=$LI(qHandle,1)
Set Counter=$LI(qHandle,2)
Set BitString=$LI(qHandle,3)
Set Counter=Counter+1
If (Counter>Top) {
Set Row=""
Set AtEnd=1
quit $$$OK
}
Set statement=##class(%SQL.Statement).%New()
Set tSC=statement.%PrepareClassQuery("TOOT.Data.Instrument","Extent")
Set tResult=statement.%Execute()
Set MaxCount=$BITCOUNT(BitString,1)
Set MaxBitStr=""
Set MaxId=0
While tResult.%Next() {
Set tmpId=tResult.%Get("ID")
Set tmpBit=##class(TOOT.Data.Instrument).%OpenId(tmpId,0).NoteListBit
Set tmpBit=$BITLOGIC(BitString|tmpBit)
Set tmpCount=$BITCOUNT(tmpBit,1)
If tmpCount>MaxCount {
Set MaxCount=tmpCount
Set MaxBitStr=tmpBit
Set MaxId=tmpId
}
}
Do tResult.%Close()
If (MaxId'=0) {
Set Row=$LB(MaxId,MaxCount)
Set AtEnd=0
Set $LI(qHandle,2)=Counter
Set $LI(qHandle,3)=MaxBitStr
} Else {
Set Row=""
Set $LI(qHandle,2)=Counter
Set AtEnd=1
}
Return $$$OK
}
ClassMethod BestNoteListClose(ByRef qHandle As %Binary) As %Status [ PlaceAfter = BestNoteListFetch ]
{
Set qHandle=""
Quit $$$OK
}
}
Calling from the Management Portal:
Where ID is the Record ID and Count is the increasing coverage of bitflags with each itteration of appending a new record.
Temporarily added logging to the Compute Method to confirm not being called during the query running.
Hi Julius, Thanks for clarification questions.
1. Yes looking to understand how developers may approach finding sepecific sets as your example: sets 1,2,4.
2. No, the AllList is informative about possible numbers avaialble to use in the sets in a specific scenario. Following your question maybe a generic solution would be flexible for any number in search for "minimum number of sets to include maximum distinct elements".







Hi Henry,
The embedding class can be extended.
I give an example in demo application: https://openexchange.intersystems.com/package/toot
My parametrs were:
The config
The new config needs to be already present before you can compile a custom embedding class. The "toot" app above shows this in file "iris.script" ie: how you can set this when building in docker from scratch.
It currently is not available to add via SQL at this point of a dockerfile build, hence object insert.
If you have an already installed instance, you can also use SQL to add and also update an existing embedding config.
Custom embedding class
In "toot" app, Look at source file "/src/XML/TOOT_Data_Embedding2_CLS.xml" this shows how the additional parameters are consumed by the custom embedding class.
Discussion
I started discussion article if you learn some new tricks and could consider also share about new challenges there also.
https://community.intersystems.com/post/vector-embeddings-feedback
Hope this helps