Here are the systems and tools that power my sites.
This site is hosted by Synthesis web hosting. It’s fast and super reliable, but only available to sites running the Genesis theme from Studiopress.
A good alternative, which is also cheaper, but obviously does not have all the premium bells and whistles, is Hostgator. They are fast, aggressively priced, and support a 1-click install of WordPress.
This site uses Genesis as my blog theme framework, with a custom child-theme. It is super-fast, very reliable, and totally compliant with WordPress best-practices.
Aweber is the only newsletter and email auto responder system I recommend. It has the best deliverability, meaning your emails actually have more chance of getting through to the recipient, has all the features you need, easy to use copy and paste email forms that look great, and a $1 trial that includes all the functionality (unlike some trial services).