Project for Prof. Borja Sotomayor Advanced Distributed Systems class under the direction of Mike Wilde and Kyle Chard
What are these files?
***** MERGER ***** ext_mapper_output.swift - a simple merger application (just cats two files) to demonstrate use of the catalog. mapper.py - maps logical filenames to physical file locations in the catalog postmapper.py - submits output of the application back into the catalog
***** CATALOG MANAGEMENT ***** distributefiles.py - logs each file in a given directory into the catalog. The physical location entered into the catalog is one of three gridftp endpoints. Commented out is the code that actually moves each file into a machine connected to the corresponding endpoint. (When transferring modis .tif files, I found that this took about 30 seconds per file, which for 317 files, was way too long.) deletefiles.py - deletes all files matching a key (field) and value pair from the catalog
***** MODIS ***** demo.mcs is like a Makefile for modis. It is what you would type on the command line, complete with all the arguments, to run modis on the mcs workstations. modismapper.py - this is the functionality I added. It queries the catalog to get a list of input files. You can run it independently as ./modismapper.py to see what the catalog returns. sites.xml - this is the sites file generated by running start-coaster-service tc.mcs - this is tc.data (which indicates where to find all the executables your swiftScript references) for the mcs workstation.
What do I need to run them?
To run ext_mapper_output, you need swift. It can be downloaded here: http://www.ci.uchicago.edu/swift/main/ run as: swift ext_mapper_output.swift -in1=cfile* -in2=dfile* -out=whatevername -owner=whomever
To run modis, you need the modis application. As of 5-31-2012, it was here: https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/modis/ to run:
- configure your coaster service as directed here: http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html#_mcs_workstations
- make sure swift/bin is in your path
- in the modis directory, type start-coaster-service
- then type ./demo.mcs
Now you are running modis on 10 mcs machines, collecting 317 input files from 3 different globus endpoints. The output will be a map containing 317 different squares that represent satellite imagery of a square location on the globe.