There is this Memory and Startup Settings.

Are you trying to limit process or shared memory? 

One issue can be just the number of processes.

I would try d ALL^%SS for CPU. d ALL^%SS is used in the ^pButtons report.

Due to shared memory it is hard to assign memory to much past the instance of Cache or IRIS.

One thing to be aware of is if you use messages.log to track IRIS.DAT growth the shrinking of IRISTEMP either through a delete or reset will throw off those numbers. 

@Timo.Lindenschmid's point is a good one. Setting a max size can be done from SMP or in %SYS with ^DATABASE but that likely means you will have issues sooner.
MaxIRISTempSizeAtStart documentation.