I had similar unexplained issues with speed (in the past)
The way I resolved it was to restart with a completely clean install on a new staging server (making sure that there is no residual data in my database), use the theme installation wizard to create a child theme, start without demo content (by now I would know what to build and how), using only the most critical plugins, and then start adding plugins one by one, page speed testing as I went along. This seemed to do the trick.
Also, don’t use caching. The site should run fine on its own and caching should be used to tweak speeds as final tailoring. If the site performs ‘okay’ (ish) while using (super)cache, you will never get the performance on par with expectation.
Consolidating javascripts and stylesheets will help as well/…