One thing to note with X11's design, having a server and client, there was nothing requiring both to be on the same machine. You could run an X11 client on your local machine, ssh into a remote machine and use its X11 server.
Lets say you are home and can ssh into a work server. You could run Firefox on the work machine, using it's network and have the visual parts show up on your home computer.
This was very much a Unix, shared resource style design. Servers and thin clients. Put all your horse power in the big machine and connect using your crappy low power system to it.