It’s a little hard to judge without knowing more about the site, and what nature of “slowdowns” you’re experiencing.
For example, if during page load the site responds fairly quickly but loading page resources (images, etc) takes a long time, then you’re probably dealing with ordinary throughput issues, and no amount of server-side caching is going to make a huge difference. If this is the case, you’re already heading down the right path with optimization and client side caching.
If, on the other hand, you find that the site is particularly responding before page load (crunching tags/templates and spitting out a response), then you definitely want to look at server-side caching, and limiting the amount of query overhead wherever you can.
Start by giving your templates a once-over, and make liberal use of the disable parameter wherever you can. It might not seem like much when looking at a single channel entries tag pair, but when you look at the number of tag pairs that are being parsed on a single page, multiplied by the number of active users, it can easy add up.
Another thing I see people doing fairly often (as a shortcut) is filtering entries using conditionals inside a loop, instead of as part of the loop.
{exp:channel:entries channel="some_channel"}
{if some_conditional}
Do something
{/if}
{/exp:channel:entries}
In the example above, that channel entries tag is going to loop through every entry in the channel, but then only display content that matches the conditional. In terms of query overhead, it might be more efficient to try to exclude as many entries as you can up front. Some of this can be done using standard parameters, but in more obscure cases, it might be better to use the Query module, or AB Entry IDs in conjunction with your channel entries tag to exclude as many entries as possible from the loop.
Another thing to look for is large blocks of content inside native EE conditionals. Because of EE’s parsing order, all of the content inside of the conditionals often gets parsed before the conditional itself, and then dropped afterwards. I common example of this is when developers try to use a single template to render different content based on url segment conditionals.
{if segment_2 == ""}
Feed code here
{if:elseif segment_2 == "category"}
Category feed code here
{if:else}
Single entry code here
{/if}
For big block conditionals like this, I recommend the use of Switchee. The addon description explains it fairly well.
As far as caching goes, EE’s native template caching is decent, though not always the best option depending on the nature of your content. I strongly recommend you check out CE Cache.
Typically I don’t cache an entire template with it, but pick and choose areas that I know will cause a significant amount of overhead. Use the global=“yes” parameter as much as you can for things like sidebar feeds that are common on multiple pages.
Let me know if you have any questions about the above. If you’re willing to share a link to the site, or an example of a template that’s particularly slow, we might be able to narrow it down a little better for you.