Here’s how to get Amazon S3 bucket logs into Splunk, and then pull out the useful fields.

You’ll need to turn on logging of your bucket in S3.

Next, install the S3 app for Splunk. Configure an S3 input (Manager >> Data inputs >> S3). Your bucket logs will now automatically feed into Splunk (the sourcetype will be ‘s3’).

I’ve written a ‘rex’ filter to grab the fields I want to work with - date, IP, action (REST) and file[name].

Stick this into the search field, then you can do all the usual Splunk goodness with the resulting fields…

sourcetype="s3" | rex field=_raw "(?x: \[ (?<date> .*) \] \s (?<ip> (?:\d{1,3}\.){3} \d{1,3}) .+? REST\.(?<action> \w+)\.OBJECT \s (?<file> .+?) \s)"