What happens when all the instance of the temporary table are in use?

It behavior can control by AE developer. If the runtime options are set to continue “People Tools will insert rows into the base table using the PROCESS_INSTANCE as a key”. If temp table doesn’t contain PROCESS_INSTANCE as a key field in a Temporary table, you should change the Temp table runtime options to “Abort” in the appropriate Application Engine programs.