I recommend Splunk a lot. I’ve been using it since about 2007, in varying degrees of complexity.
Updated Nov 2013 to use Ansible to provision.
As a log consolidation and data mining tool Splunk is great. But I’ll go for long periods where I don’t get to use it, so my ‘demo’ VMs get killed off. With my discovery of Vagrant, I decided to take my default CentOS 6 VM and put some Config Management to work to do an install of Splunk.
I can now very rapidly pop up a demo Splunk instance if I want to evangelise it some more to customers. I might even take this as the root to building a multi-machine demo.