Boosting facial recognition with Vector Search

As you have seen in the latest community publications, InterSystems IRIS has included since version 2024.1 the possibility of including vector data types in its database and based on this type of data vector searches have been implemented. Well, these new features reminded me of the article I published a while ago that was based on facial recognition using Embedded Python.

Introduction

For those of you who don't remember what that article was about, it is linked at the end of this article. The way the application worked was to recognize the faces present in any image and subsequently compare it with the faces that we had already identified in our IRIS instance, indicating who it belonged to.

How could we improve our facial recognition application by taking advantage of these new features? Well, the main problem we found in our project is performance. Having to recalculate the vector for each image in our system and compare it to the image to be identified is a time-consuming process that can be greatly improved with vector search.

Let's see how we could implement it with the Open Exchange project that we have associated.

Example project

The project is going to deploy a container in Docker with the latest released version of InterSystems IRIS (as of the date of publication of this article, 2024.1) in which we will create the table on which we will store the images with which to compare. To do this we will launch the following command from the database:

CREATE TABLE Vectorface_Data.Person (name VARCHAR(50), description VARCHAR(1000), photo VECTOR(DECIMAL, 128))

As you can see, our photo column will be a vector with 128 decimal values. Next we are going to deploy a web application in our instance that we will call from Postman:

This web application will use the Vectorface.WS.Service class, which will be responsible for processing the information received by HTTP POST from Postman.

Let's take a look at said class, first the route map:

XData UrlMap [ XMLNamespace = "https://www.intersystems.com/urlmap" ]
{
<Routes>
	<Route Url="/checkSimilarity" Method="POST" Call="CheckSimilarity" />
	<Route Url="/savePhoto" Method="POST" Call="SavePhoto" />
</Routes>
}

We are going to work with two URLs:

• /savePhoto: responsible for receiving the photo in base64, analyzing it, vectorizing the image and storing it in the database.
• /checkSimilarity: which will vectorize the image for comparison and launch the query to the database looking for the most similar image.

Saving photos in IRIS:

ClassMethod SavePhoto() As %Status
{
    Try {
        Do ##class(%REST.Impl).%SetContentType("application/json")
        If '##class(%REST.Impl).%CheckAccepts("application/json") Do ##class(%REST.Impl).%ReportRESTError(..#HTTP406NOTACCEPTABLE,$$$ERROR($$$RESTBadAccepts)) Quit
        // Reading the body of the http call with the person data
        set dynamicBody = {}.%FromJSON(%request.Content)

        set dynamicStream = dynamicBody.%Get("fileData",,"stream<base64")

        set stream=##class(%Stream.FileBinary).%New()
        set sc=stream.LinkToFile("/shared/durable/"_dynamicBody.fileName)
        set sc=stream.CopyFromAndSave(dynamicStream)

        set imageVector = ..Checker("/shared/durable/"_dynamicBody.fileName)       
        set imageVector = $REPLACE(imageVector, $CHAR(13,10),",")
        set imageVector = $REPLACE(imageVector,"['","")
        set imageVector = $REPLACE(imageVector,"']","")
        set imageVector = $REPLACE(imageVector,"'","")
        set imageVector = $REPLACE(imageVector," ",",")

        &sql(INSERT INTO Vectorface_Data.Person VALUES (:dynamicBody.name, :dynamicBody.description, TO_VECTOR(:imageVector, DECIMAL)))

        Do ##class(%REST.Impl).%SetStatusCode("200")
        Do ##class(%REST.Impl).%WriteResponse(imageVector)
        return {"result": "Picture stored"}
        
    } Catch (ex) {
        Do ##class(%REST.Impl).%SetStatusCode("400")
        Do ##class(%REST.Impl).%WriteResponse(ex.DisplayString())
        return {"errormessage": "Client error"}
    }
    Quit $$$OK
}

This function transforms the base64 into a file that is subsequently sent to the Python Checker function, responsible for identifying the face and vectorizing it. With the recovered vector we will insert it into our database in vector format.

Comparing vector images with the database

ClassMethod CheckSimilarity() As %Status
{
    Try {
        Do ##class(%REST.Impl).%SetContentType("application/json")
        If '##class(%REST.Impl).%CheckAccepts("application/json") Do ##class(%REST.Impl).%ReportRESTError(..#HTTP406NOTACCEPTABLE,$$$ERROR($$$RESTBadAccepts)) Quit
        // Reading the body of the http call with the person data
        set dynamicBody = {}.%FromJSON(%request.Content)

        set dynamicStream = dynamicBody.%Get("fileData",,"stream<base64")

        set stream=##class(%Stream.FileBinary).%New()
        set sc=stream.LinkToFile("/shared/durable/"_dynamicBody.fileName)
        set sc=stream.CopyFromAndSave(dynamicStream)

        set imageVector = ..Checker("/shared/durable/"_dynamicBody.fileName)       
        set imageVector = $REPLACE(imageVector, $CHAR(13,10),",")
        set imageVector = $REPLACE(imageVector,"['","")
        set imageVector = $REPLACE(imageVector,"']","")
        set imageVector = $REPLACE(imageVector,"'","")
        set imageVector = $REPLACE(imageVector," ",",")

        set name = ""
        set similarity = ""
        &sql(SELECT TOP 1 name, similarity INTO :name, :similarity  FROM (SELECT name, VECTOR_DOT_PRODUCT(photo, TO_VECTOR(:imageVector, DECIMAL)) AS similarity FROM Vectorface_Data.Person) ORDER BY similarity DESC)

        set result = {"name": "", "similarity":""}
        set result.name = name
        set result.similarity = similarity
        Do ##class(%REST.Impl).%WriteResponse(result.%ToJSON())

        Do ##class(%REST.Impl).%SetStatusCode("200")	
        
    } Catch (ex) {
        Do ##class(%REST.Impl).%SetStatusCode("400")
        return ex.DisplayString()
    }
    Quit $$$OK
}

The function is very similar, with the exception that we do not record the image that we are going to compare, but rather we use the generated vector to compare it with those registered in the database through an SQL query.

Testing the example

We are going to register a few people with their photos in IRIS, for example... all the General Secretaries of the Communist Party of the Soviet Union because... Who hasn't wanted to know which leader of the Soviet Union they look like?

Here we have an example of our POST call:

As you can see we have the fields that identify Brezhnev with his associated base64 photo. This is how it would look in our table with a vector column:

It's time to launch a test, let's see which Secretary General is most like me:

Creepy...it seems that Stalin is my evil twin...although the similarity is quite low, let's try a different photo of Chernenko, in this case the result should be much higher.

Indeed, here we have a much higher value for Chernenko (0.736), which we already had registered in our database.

Conclusion

The inclusion of the vector type and its associated searches opens up a world of opportunities based on exploiting vector-based AI models, I encourage you all to try it!

PS: My thanks to @Thomas Dyar who provided me with the information necessary to develop this example.