Question
· Feb 13

Stopping IntegratedML Training / Long training time

This may be a very specific problem but might help with some general information that I wasn't able to find anywhere.

I have started to train 2 IntegratedML models using AutoML one month ago.

My first attempt was very optimistic with a class that has  26 Properties and 100.000 datasets. I wasn't expecting good results, just wanted to test it. 
(CREATE MODEL ECLASSPREDICT3 PREDICTING (eClass) FROM CRMSHOPARTIKELML)

My second attempt was more specific using the same class structure but reducing the dataset to 10.000 entries and only using certain properties.
CREATE MODEL ECLASSPREDICT4 PREDICTING (eClass) WITH (NormalisierterLangtext VARCHAR, NormalisierterText VARCHAR, NormalisierterHerstell VARCHAR, Keywoerter VARCHAR, GTIN VARCHAR, EAN VARCHAR, DIN VARCHAR) FROM CRMSHOPARTIKELML

I ran the TRAIN command for both successfully and went on vacation, expecting it to be finished when I came back. Now it has been running almost a month and is still in training. 
I can see it both in SQL Activity, aswell as with SELECT * FROM INFORMATION_SCHEMA.ML_TRAINING_RUNS
I see 2 processes running aswell (%ML.AutoML.Provider.1).

Interestingly in SQL Activity under Execution statistics I can see that the first model has "no data yet" but the second one has the following: 

Times executed 3 no data yet
Average runtime 23.9900 sec no data yet
Runtime standard deviation 27.4330 no data yet
Average rowcount 1 row(s) no data yet
Average commands 329,423 commands no data yet

 

Both are running on a Intel Xeon Silver 4208 8 Core 2,1 GHz Machine (only at 20% usage usually). That isn't the best but for the time spent, I was really sure it would be done.

I then decided to start the first model training as I thought it might interfere with the second one. I was unable to do so. Neither the process nor the SQL command will let me interrupt the training. (Error message below) 

So now my question to which I couldn't find any information:

 

1. Is it really plausible, that both Models aren't finished after this long? 

2. How can I stop a model that is training without restarting the machine? Even if it means losing my model training time. 

 

 

 

 

Product version: IRIS 2023.3
$ZV: IRIS for Windows (x86-64) 2023.3 (Build 254U) Wed Nov 8 2023 13:17:47 EST
Discussion (2)2
Log in or sign up to continue