Managing other portainer instances from one - do I need the server and agent

I have 3 portainer servers on different VPS's on different providers.
I'd like to centrally manage them using one, but I guess be able to login to each one if there is an issue.
Do I need to install both the server and agent on both the 'remote' machines, or can I just have the server and use that.