Answer by ahulpke for Finding the Automorphism Group of a Finitely Presented Group with Solvable Word Problem

There are a number of algorithms for automorphism group computations of finite groups. I am not aware of a generic algorithm for infinite groups.

I suspect that for infinite polycyclic groups it might be possible (modulo some potentially nasty number theory) to generalize the methods for finite solvable groups, but that is a research-level problem without a turn-key implementation.

When you call AutomorphismGroup in GAP on a finitely presented group, it will try to test finiteness first, this is probably the non-terminating calculation you observe.