A Developer with a Pencil

Dragonfly on Heroku - the Difference Between the Request Time and the Current Time Is Too Large

Lately we have been experiencing intermittent exceptions on Heroku when uploading images to S3 using “Dragonfly” on Heroku:

1
Excon::Errors::Forbidden: Expected(200) <=> Actual(403 Forbidden)

It seems that this exceptions doesn’t happen on every upload - so while examining the response body from S3 - we got this:

1
<Excon::Response:0x000000062a09d0 @body="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>RequestTimeTooSkewed</Code><Message>The difference between the request time and the current time is too large.</Message><MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds><RequestId>B9CB09E0E0A7054B</RequestId><HostId>K3liRup7BjJoxBXgkCGpD7NSk/0jIUy6+nBY5Y63akNx4MNNLMvj7zSlEadDn87Q</HostId><RequestTime>Mon, 26 Aug 2013 11:39:58 +0000</RequestTime><ServerTime>2013-08-26T11:55:29Z</ServerTime></Error>", @headers={"x-amz-request-id"=>"xxxx", "x-amz-id-2"=>"xxxx", "Content-Type"=>"application/xml", "Transfer-Encoding"=>"chunked", "Date"=>"Mon, 26 Aug 2013 11:55:28 GMT", "nnCoection"=>"close", "Server"=>"AmazonS3"}, @status=403>

and especially this:

1
<Message>The difference between the request time and the current time is too large.</Message>

After investigating a bit, we found that it means that our requests to S3 are timestamped and compared to the server local time in order to assure authenticity, so it seems that something went wrong with our local clock on Heroku. After realizing that the local time on the heroku machines (using the rails console) was correct, we monkeypatched the Dragonfly::DataStorage::S3DataStore module to use the sync_clock method on every access to the storage getter.

config/initializers/sync_dragonfly.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Dragonfly::DataStorage::S3DataStore.module_eval do

  def storage
    require 'fog'

    @storage ||= Fog::Storage.new(
        :provider => 'AWS',
        :aws_access_key_id => access_key_id,
        :aws_secret_access_key => secret_access_key,
        :region => region
    )
    @storage.sync_clock
    @storage
  end
end

What this patch does is simply to sync the clock before returning the storage object.

Currently, one day after we seem to have no exceptions of this kind anymore.

Comments