Skip to content

ETL can't perform clean up due to S3 bad request #38

Description

@mattfullerton

After finally finding somewhere for the bucket where data can be uploaded, the app doesn't appear to be able to get the data back out. The user has S3 full access. I suppose there is a small chance this is the reason for #35...

Debug:
[2015-12-07 23:11:32,967: WARNING/Worker-2] Failed: Clean up source data
[2015-12-07 23:11:33,693: ERROR/MainProcess] Task spendb.tasks.load_from_source[14d7ea38-b1c1-4540-b9df-9d14fca22263] raised unexpected: S3ResponseError: 400 Bad Request
Traceback (most recent call last):
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(_args, *_kwargs)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
return self.run(_args, *_kwargs)
File "/Volumes/MacintoshHD2/Users/mattbirgit/checkouts/spendb/spendb/tasks.py", line 36, in load_from_source
source = tasks.transform_source(dataset, source_name)
File "/Volumes/MacintoshHD2/Users/mattbirgit/checkouts/spendb/spendb/etl/job.py", line 74, in wrapper
job.failed()
File "/Volumes/MacintoshHD2/Users/mattbirgit/checkouts/spendb/spendb/etl/job.py", line 53, in failed
self.end(Run.STATUS_FAILED)
File "/Volumes/MacintoshHD2/Users/mattbirgit/checkouts/spendb/spendb/etl/job.py", line 39, in end
self.log_handler.archive()
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/loadkit/logger.py", line 27, in archive
logfile = LogFile(self.package, name)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/resource.py", line 21, in init
self.meta = ResourceMetaData(self)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/manifest.py", line 38, in init
self.manifest = resource.package.manifest
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/package.py", line 42, in manifest
self._manifest = Manifest(obj)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/manifest.py", line 14, in init
self.load()
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/manifest.py", line 17, in load
if self.object.exists():
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/store/s3.py", line 98, in exists
self._key = self.store.bucket.get_key(self._key_name)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/store/s3.py", line 34, in bucket
self._bucket = self.conn.get_bucket(self.bucket_name)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/boto/s3/connection.py", line 502, in get_bucket
return self.head_bucket(bucket_name, headers=headers)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/boto/s3/connection.py", line 549, in head_bucket
response.status, response.reason, body)
S3ResponseError: S3ResponseError: 400 Bad Request

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions