At the DTL level, how to Rollback a message so as to retry that message again

I wouldn't use a DTL for process flow control.  I would have a BPL make those decisions and use the DTL strictly for transformation of the message.  The BPL diagram can then just fall through to the end once the REST call is successful.  The operation doing the REST call should then handle the infinite retry, Failure Timeout of -1 would tell it to never stop retrying as you said.

Hi!

     Assuming you meant "BPL" (Business Process Language) instead of "DTL" (Data Transformation Langue):

     If you simply want your Business Operation to try forever until it gets it done:

•  On the BPL, make a synchronous call or make an asynchronous call with a sync activity for it
•  On the BO, set FailureTime=-1. Also, try to understand the "Reply Code Actions" setting of your Business Operation. You don't want to retry for all kinds of errors. You probably want to retry for some errors and failure for others. If you set FailureTime=-1 and your Reply Code  Actions decides to retry for that kind of error, it will retry forever until it gets it done. If your Reply code Actions decides to fail for other types of errors, it will return an error to your Business Process.
•  If you know that, for some errors, the BO will return a failure, protect the call you just did on your BPL with a scope action so you can capture this and take additional actions.

More about "Reply Code Actions" here.

Kind regards,

Amir Samary