Stopping IntegratedML Training / Long training time
This may be a very specific problem but might help with some general information that I wasn't able to find anywhere.
I have started to train 2 IntegratedML models using AutoML one month ago.
My first attempt was very optimistic with a class that has 26 Properties and 100.000 datasets. I wasn't expecting good results, just wanted to test it.
(CREATE MODEL ECLASSPREDICT3 PREDICTING (eClass) FROM CRMSHOPARTIKELML)
My second attempt was more specific using the same class structure but reducing the dataset to 10.000 entries and only using certain properties.
CREATE MODEL ECLASSPREDICT4 PREDICTING (eClass) WITH (NormalisierterLangtext VARCHAR, NormalisierterText VARCHAR, NormalisierterHerstell VARCHAR, Keywoerter VARCHAR, GTIN VARCHAR, EAN VARCHAR, DIN VARCHAR) FROM CRMSHOPARTIKELML
I ran the TRAIN command for both successfully and went on vacation, expecting it to be finished when I came back. Now it has been running almost a month and is still in training.
I can see it both in SQL Activity, aswell as with SELECT * FROM INFORMATION_SCHEMA.ML_TRAINING_RUNS
I see 2 processes running aswell (%ML.AutoML.Provider.1).
Interestingly in SQL Activity under Execution statistics I can see that the first model has "no data yet" but the second one has the following:
Times executed | 3 | no data yet |
Average runtime | 23.9900 sec | no data yet |
Runtime standard deviation | 27.4330 | no data yet |
Average rowcount | 1 row(s) | no data yet |
Average commands | 329,423 commands | no data yet |
Both are running on a Intel Xeon Silver 4208 8 Core 2,1 GHz Machine (only at 20% usage usually). That isn't the best but for the time spent, I was really sure it would be done.
I then decided to start the first model training as I thought it might interfere with the second one. I was unable to do so. Neither the process nor the SQL command will let me interrupt the training. (Error message below)
So now my question to which I couldn't find any information:
1. Is it really plausible, that both Models aren't finished after this long?
2. How can I stop a model that is training without restarting the machine? Even if it means losing my model training time.
You can kill the process from the process manager of Windows looking for the id of process that you got from IRIS.
And from my experience with IntegratedML... Is not very fast but not so slow.
Thank you that helped stopping the processes although it took a couple hours for the processes to disappear