Processing big datasets

[smathermather]

I don't know if this needs to be part of the odm-benchmarks project, but I have a particularly large dataset to process, so I am doing some monitoring of individual stages so that I can do a better job predicting processing time over the life of the project. I thought I would document that here in case it is useful to see.

[smathermather]

Each stage has a last written file. We can use that last written file to understand when the stage completed. It's possible to do something similar for each sub-stage of OpenSfM as well, but for now, we will restrict ourselves to the overall stage times. The following is with a ~7400 image dataset on a 20 core machine with 768 RAM:

ls -lhtr odm_dem/dtm.tif mve/mve_dense_point_cloud.ply odm_filterpoints/point_cloud.ply
odm_georeferencing/odm_georeferenced_model.laz odm_georeferencing_25d/odm_georeferencing_log.txt odm_meshing/odm_25dmesh
.ply odm_meshing/odm_mesh.ply odm_orthophoto/odm_orthophoto.tif odm_texturing/odm_textured_model_geo.mtl odm_texturing_2
5d/odm_textured_model_geo.mtl opensfm/geocoords_transformation.txt odm_report/shots.geojson img_list.txt
-rw-r--r-- 1 useruser useruser 220K Jun 25 16:20 img_list.txt
-rw-r--r-- 1 useruser useruser  192 Jun 27 00:18 opensfm/geocoords_transformation.txt
-rw-r--r-- 1 useruser useruser  49G Jul  3 15:40 mve/mve_dense_point_cloud.ply
-rw-r--r-- 1 useruser useruser  49G Jul  4 02:05 odm_filterpoints/point_cloud.ply
-rw-r--r-- 1 useruser useruser 7.3M Jul  4 02:23 odm_meshing/odm_mesh.ply
-rw-r--r-- 1 useruser useruser 7.3M Jul  4 05:50 odm_meshing/odm_25dmesh.ply
-rw-r--r-- 1 useruser useruser  36K Jul  4 08:13 odm_texturing_25d/odm_textured_model_geo.mtl
-rw-r--r-- 1 useruser useruser 2.3K Jul  4 09:45 odm_georeferencing_25d/odm_georeferencing_log.txt
-rw-r--r-- 1 useruser useruser  36K Jul  4 10:56 odm_texturing/odm_textured_model_geo.mtl
-rw-r--r-- 1 useruser useruser 4.9G Jul  4 12:26 odm_georeferencing/odm_georeferenced_model.laz
-rw-r--r-- 1 useruser useruser 1.8G Jul  4 18:43 odm_dem/dtm.tif
-rw-r--r-- 1 useruser useruser  16G Jul  4 21:27 odm_orthophoto/odm_orthophoto.tif
-rw-r--r-- 1 useruser useruser  845 Jul  4 21:29 odm_report/shots.geojson

[smathermather]

If we do a bit of calculations against these dates, we can see the progression:

date	stage	stage length (h)	overall time (d)	stage percentage	stage overall percentage
6/25/2020 16:20	start	0.00	0.00	0.00%	0.00%
6/27/2020 0:18	opensfm	31.97	1.33	14.45%	14.45%
7/3/2020 15:40	mve	159.37	7.97	72.06%	86.52%
7/4/2020 2:05	odm_filterpoints	10.42	8.41	4.71%	91.23%
7/4/2020 2:23	odm_meshing	0.30	8.42	0.14%	91.36%
7/4/2020 5:50	odm_meshing	3.45	8.56	1.56%	92.92%
7/4/2020 8:13	odm_texturing_25d	2.38	8.66	1.08%	94.00%
7/4/2020 9:45	odm_georeferencing_25d	1.53	8.73	0.69%	94.69%
7/4/2020 10:56	odm_texturing	1.18	8.78	0.54%	95.23%
7/4/2020 12:26	odm_georeferencing	1.50	8.84	0.68%	95.91%
7/4/2020 18:43	odm_dem	6.28	9.10	2.84%	98.75%
7/4/2020 21:27	odm_orthophoto	2.73	9.21	1.24%	99.98%
7/4/2020 21:29	odm_report	0.03	9.21	0.02%	100.00%

[smathermather]

Honestly, I expected OpenSfM to be a more expensive piece, and props to @pierotofy for bringing down that odm_dem number so much.

[smathermather]

Now I am testing with the same 7400 image dataset but resized using https://github.com/pierotofy/exifimageresize and resizing the images first to 1280 and then setting orthophoto and elevation model resolutions pretty low. The plan is to do hydrological modeling on these data and orthos have already been otherwise generating, so getting a good low res DEM is a good starting product, but why waste the memory and processing time on full resolution products:

First, log in to the docker instance:

docker run -ti -v /home/useruser/outdir:/datasets --entrypoint bash odmnolign

Then, we run the run.py script directly. This gives us some flexibility in resuming, or tweaking as we go:

./run.py --project-path /datasets --project-path /datasets datasetname --dtm --dsm --smrf-scalar 0 --smrf-slope 0.06 --smrf-threshold 1.05 --smrf-window 49 --dem-resolution 414 --depthmap-resolution 320 --ignore-gsd --orthophoto-resolution 138

I expect the OpenSfM step to take about the same length of time: 30+ hours, but depthmaps to be 4x as fast, which should get this done over the weekend instead of 9 days. 🤞

This time, I am also monitoring using Glances so I have the full curve of resource usage through the running of the project.

[smathermather]

I am going to keep this thread going and expand a bit to include info about how to process monster datasets. This will be messy, and riddled with contradictions, but I will ultimately probably end up as a section in docs.opendronemap.org, at least the how-to bits.

[smathermather]

We should probably add a flag for this, but I have removed the align portion of split merge. For massive datasets, the combination of SfM, error distribution, and OpenDroneMap's use of DOP and other accuracy tags seems to make the alignment step not just unnecessary but a source of error. Here's my branch which removes that step in a brutish way... :
https://github.com/smathermather/ODM/tree/noalign

git clone https://github.com/smathermather/ODM
cd ODM
git checkout noalign
docker build -t odmnolign .

[smathermather]

Also, for massive datsets, perhaps the objective (as it is in my case) is to capture elevation models, not orthophotos. So one trick we can do is to resample the data down quite a bit, and then adjust the output resolutions to save on memory usage and processing time. A good tool for that is UAV4Geos https://github.com/pierotofy/exifimageresize which will resize an entire directory of images but keep the exif the same (aside from the exif on image dimensions).

I resized my dataset to 1280 which should leave enough features for matching and enough for depthmaps as well.

When I ran my 7400 image subset at default depthmap resolution of 640, it took 9 days, 5 hours to complete. Reducing that resolution to 320, we get completion in 3 days, 18 hours.

date	stage	stage length (h)	overall time (d)	stage percentage	stage overall percentage
7/10/2020 3:40	start	0.00	0.00	0.00%	0.00%
7/11/2020 14:55	opensfm	35.25	1.47	39.07%	39.07%
7/13/2020 15:57	mve	49.03	3.51	54.34%	93.41%
7/13/2020 16:47	odm_filterpoints	0.83	3.55	0.92%	94.33%
7/13/2020 16:52	odm_meshing	0.08	3.55	0.09%	94.42%
7/13/2020 17:27	odm_meshing	0.58	3.57	0.65%	95.07%
7/13/2020 19:51	odm_texturing_25d	2.40	3.67	2.66%	97.73%
7/13/2020 20:12	odm_georeferencing_25d	0.35	3.69	0.39%	98.12%
7/13/2020 20:31	odm_texturing	0.32	3.70	0.35%	98.47%
7/13/2020 21:20	odm_georeferencing	0.82	3.74	0.91%	99.37%
7/13/2020 21:43	odm_dem	0.38	3.75	0.42%	99.80%
7/13/2020 21:52	odm_orthophoto	0.15	3.76	0.17%	99.96%
7/13/2020 21:54	odm_report	0.03	3.76	0.04%	100.00%

[smathermather]

Also, docker is cool, but when things fail (as they often do with big datasets) reconnecting to the docker machine to inspect can be difficult, since the ODM docker image is meant to turn off automatically when done (whether a failure or not). So, instead of the typical docker run, we can instead start the instance and run the run.py command from within the container. This will ensure we can reconnect, poke around, and do all the needful things:

docker run -ti -v /home/gisuser/outdir:/datasets --entrypoint bash odmnolign

Now we are logged in and can run our OpenDroneMap command:

./run.py --project-path /datasets --project-path /datasets resize --dtm --dsm --smrf-scalar 0 --smrf-
slope 0.06 --smrf-threshold 1.05 --smrf-window 49 --dem-resolution 413 --depthmap-resolution 320 

If we need to reconnect, we can look for our docker image:

docker ps

And then use the machine ID to connect

docker exec -it <docker hash or nickname> bash

We can also use docker logs to check in on the process:

docker logs <docker hash or nickname> | tail

[smathermather]

For particularly large datasets, like the 80,000+ image dataset I am currently attempting, we need to invoke split-merge by using the --split option and we probably also want to use hybrid bundle adjustment to increase the efficiency of our structure from motion step:

./run.py --project-path /datasets --project-path /datasets resize --dtm --dsm \
--smrf-scalar 0 --smrf-slope 0.06 --smrf-threshold 1.05 --smrf-window 49 \
--dem-resolution 413 --depthmap-resolution 320 --split 7000 --use-hybrid-bundle-adjustment

[smathermather]

A lot of folks don't necessarily understand the relationship between depthmap resolution and ground sampling resolution, so I should explain:

Suppose we have a 6400x4200 image with a ground sampling distance (pixel size) of 10cm. The maximum depthmap we want to use would be half the linear image resolution at 3200x2100, which could be specified with --depthmap-resolution 3200. This means the maximum elevation resolution should be 1/2 the ground sampling resolution or 20cm.

I often aim for 1/4 the resolution as it is a bit more robust to noise, especially on specular reflectors like metal rooftops, and saves a lot of computation time. In this case, I would set --depthmap-resolution 1600 and --dem-resolution 40.

In our case above, I have also resized the input images as well. Since I am not too interested in the orthophotos, this saves on memory usage in the texturing step, and as I want this to run as fast as possible, I have optimized for a depthmap resolution of 320.

Most of the time, OpenDroneMap uses good defaults, good optimization, and chooses these settings for us. But for massive datasets, we probably need to be more thoughtful than using the defaults. At least, until we have good standards for these larger datasets and can embody some of this rationale into the code base.

[smathermather]

A nearly useless useful utility to track feature extraction, written in shell:

# Find out how many images there are
totallines=`wc -l ~/outdir/znz/resize/img_list.txt | awk '{print $1}'`

previouspermillage=0
while true
do
    # Count the number of files in the OpenSfM features directory
    num1=`find ~/outdir/znz/resize/opensfm/features/ -type f | wc -l `

    # Calculate percentage of images that have had features extracted
    percentage=`echo $num1 / $totallines \* 100| bc -l`
	
    # We calculate permillage for the sake of determining whether we are 
    # going to display a new number
    permillage=`echo $num1 / $totallines \* 1000| bc -l`

    # If our rounded permillage number has increased, then display a new number
    if [ `printf "%.0f\n" $permillage` -gt `printf "%.0f\n" $previouspermillage` ]
    then
#        echo
        echo -n `printf "%.1f\n" $percentage`	
    fi

    # Speak less. Smile more.
	echo -n "."
    sleep 5

    # Check if we are done
    if [ `printf "%.0f\n" $percentage` -gt 99 ]
    then
        # Bing!
        echo -ne '\007'
        echo "All done with feature extraction!"
        break              
    fi 

previouspermillage=$permillage

done

[coreysnipes]

Just wanted to pop in and say I'm glad you're including this info here @smathermather .

[smathermather]

Hey thanks! I was hoping I wasn't creating unneeded noise. But, I need to track this, and I figure it can become a platform for further use and understanding.

[smathermather]

I have stopped using the above approach of logging into the docker instance and running the script. It seems like enough to just to remove the -rm flag when using docker run, and then it's possible to login to the machine if needed. I have also modified my branch a little more to default to BOW matching in the hopes that I can get faster matching

docker run -ti -v /home/gisuser/outdir/znz:/datasets odmnolign --project-path /datasets resize --dtm --dsm --smrf-scalar 0 --smrf-slope 0.06 --smrf-threshold 1.05 --smrf-window 100 --dem-resolution 413 --depthmap-resolution 320 --split 7000 --use-hybrid-bundle-adjustment

[smathermather]

One of the big challenges with massive datasets is catching issues before they are issues. In my case, that is finding small pockets of bad data. We probably need more robust error checking in the chain. So, for example, I had 4 files with some issue (
they probably didn't fully come back from glacier storage before being transferred). That seems a pretty reasonable error rate, but the challenge is that each restart after finding bad data is a setback of a few days of processing:

2020-07-23 18:04:15,105 DEBUG: Matching deduped2016-10-11_09.23.08.JPG and deduped2016-10-11_09.21.04.JPG.  Matcher: WORDS (symmetric) T-desc: 0.165 T-robust: 0.057 T-total: 0.223 Matches: 447 Robust: 430 Success: True
2020-07-23 18:04:15,120 DEBUG: No segmentation for geoscleanup2016-10-11_09.29.52.JPG, no features masked.
2020-07-23 18:04:15,388 DEBUG: Matching deduped2016-10-11_09.23.08.JPG and geoscleanup2016-10-11_09.29.52.JPG.  Matcher: WORDS (symmetric) T-desc: 0.163 T-robust: 0.105 T-total: 0.269 Matches: 543 Robust: 531 Success: True
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/match_features.py", line 29, in run
    pairs_matches, preport = matching.match_images(data, images, images)
  File "/code/SuperBuild/src/opensfm/opensfm/matching.py", line 43, in match_images
    return match_images_with_pairs(data, exifs, ref_images, pairs), preport
  File "/code/SuperBuild/src/opensfm/opensfm/matching.py", line 67, in match_images_with_pairs
    matches = context.parallel_map(match_unwrap_args, args, processes, jobs_per_process)
  File "/code/SuperBuild/src/opensfm/opensfm/context.py", line 41, in parallel_map
    return Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args)
  File "/usr/local/lib/python2.7/dist-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/usr/local/lib/python2.7/dist-packages/joblib/parallel.py", line 862, in retrieve
    raise exception.unwrap(this_report)
joblib.my_exceptions.JoblibIOError: JoblibIOError
___________________________________________________________________________
...........................................................................
/code/SuperBuild/src/opensfm/bin/opensfm in <module>()
     29 args = parser.parse_args()
     30 
     31 # Run the selected subcommand
     32 for command in subcommands:
     33     if args.command == command.name:
---> 34         command.run(args)
...........................................................................
/code/SuperBuild/src/opensfm/opensfm/commands/match_features.py in run(self=<opensfm.commands.match_features.Command instance>, args=Namespace(command='match_features', dataset='/datasets/resize/opensfm'))
     24     def run(self, args):
     25         data = dataset.DataSet(args.dataset)
     26         images = data.images()
     27 
     28         start = timer()
---> 29         pairs_matches, preport = matching.match_images(data, images, images)
        pairs_matches = undefined
        preport = undefined
        data = <opensfm.dataset.DataSet object>
        images = [u'DSC00034_2978.jpg', u'DSC00179_4743.jpg', u'DSC00180_4703.jpg', u'DSC00183_4739.jpg', u'DSC00184_4720.jpg', u'DSC00190_4051.jpg', u'DSC00209_4338.jpg', u'DSC00211_4349.jpg', u'DSC00213_4333.jpg', u'DSC00216_4301.jpg', u'DSC00221_3845.jpg', u'DSC00225_3831.jpg', u'DSC00229_4355.jpg', u'DSC00230_3827.jpg', u'DSC00230_4335.jpg', u'DSC00235_4346.jpg', u'DSC00240_4160.jpg', u'DSC00241_3752.jpg', u'DSC00241_4176.jpg', u'DSC00244_4182.jpg', ...]
     30         matching.save_matches(data, images, pairs_matches)
     31         end = timer()
     32 
     33         with open(data.profile_log(), 'a') as fout:
...........................................................................
/code/SuperBuild/src/opensfm/opensfm/matching.py in match_images(data=<opensfm.dataset.DataSet object>, ref_images=[u'DSC00034_2978.jpg', u'DSC00179_4743.jpg', u'DSC00180_4703.jpg', u'DSC00183_4739.jpg', u'DSC00184_4720.jpg', u'DSC00190_4051.jpg', u'DSC00209_4338.jpg', u'DSC00211_4349.jpg', u'DSC00213_4333.jpg', u'DSC00216_4301.jpg', u'DSC00221_3845.jpg', u'DSC00225_3831.jpg', u'DSC00229_4355.jpg', u'DSC00230_3827.jpg', u'DSC00230_4335.jpg', u'DSC00235_4346.jpg', u'DSC00240_4160.jpg', u'DSC00241_3752.jpg', u'DSC00241_4176.jpg', u'DSC00244_4182.jpg', ...], cand_images=[u'DSC00034_2978.jpg', u'DSC00179_4743.jpg', u'DSC00180_4703.jpg', u'DSC00183_4739.jpg', u'DSC00184_4720.jpg', u'DSC00190_4051.jpg', u'DSC00209_4338.jpg', u'DSC00211_4349.jpg', u'DSC00213_4333.jpg', u'DSC00216_4301.jpg', u'DSC00221_3845.jpg', u'DSC00225_3831.jpg', u'DSC00229_4355.jpg', u'DSC00230_3827.jpg', u'DSC00230_4335.jpg', u'DSC00235_4346.jpg', u'DSC00240_4160.jpg', u'DSC00241_3752.jpg', u'DSC00241_4176.jpg', u'DSC00244_4182.jpg', ...])
     38     # Generate pairs for matching
     39     pairs, preport = pairs_selection.match_candidates_from_metadata(
     40         ref_images, cand_images, exifs, data)
     41 
     42     # Match them !
---> 43     return match_images_with_pairs(data, exifs, ref_images, pairs), preport
        data = <opensfm.dataset.DataSet object>
        exifs = {u'DSC00034_2978.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473250145.0, u'focal_ratio': 0, u'gps': {u'altitude': 74.435, u'latitude': -6.050773555555556, u'longitude': 39.21183877777778}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, u'DSC00179_4743.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473251824.0, u'focal_ratio': 0, u'gps': {u'altitude': 88.705, u'latitude': -6.061974111111111, u'longitude': 39.207787083333336}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, u'DSC00180_4703.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473251828.0, u'focal_ratio': 0, u'gps': {u'altitude': 86.979, u'latitude': -6.061924805555555, u'longitude': 39.20743983333334}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, u'DSC00183_4739.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473251839.0, u'focal_ratio': 0, u'gps': {u'altitude': 87.518, u'latitude': -6.06194775, u'longitude': 39.20635772222222}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, u'DSC00184_4720.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473251843.0, u'focal_ratio': 0, u'gps': {u'altitude': 87.861, u'latitude': -6.061936972222222, u'longitude': 39.205986055555556}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, u'DSC00190_4051.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473251820.0, u'focal_ratio': 0, u'gps': {u'altitude': 79.314, u'latitude': -6.059792916666667, u'longitude': 39.21130225}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, u'DSC00209_4338.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473251954.0, u'focal_ratio': 0, u'gps': {u'altitude': 88.492, u'latitude': -6.060617416666666, u'longitude': 39.21422186111111}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, u'DSC00211_4349.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473251961.0, u'focal_ratio': 0, u'gps': {u'altitude': 89.117, u'latitude': -6.060633027777778, u'longitude': 39.21359697222223}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, u'DSC00213_4333.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473251969.0, u'focal_ratio': 0, u'gps': {u'altitude': 87.932, u'latitude': -6.060612166666666, u'longitude': 39.21292638888889}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, u'DSC00216_4301.jpg': {u'band_name': u'RGB', u'camera': u'v2 sony dsc-wx220 1280 960 brown 0 rgb', u'capture_time': 1473251980.0, u'focal_ratio': 0, u'gps': {u'altitude': 86.411, u'latitude': -6.060592916666667, u'longitude': 39.212014583333335}, u'height': 960, u'make': u'SONY', u'model': u'DSC-WX220', u'orientation': 1, u'projection_type': u'brown', ...}, ...}
        ref_images = [u'DSC00034_2978.jpg', u'DSC00179_4743.jpg', u'DSC00180_4703.jpg', u'DSC00183_4739.jpg', u'DSC00184_4720.jpg', u'DSC00190_4051.jpg', u'DSC00209_4338.jpg', u'DSC00211_4349.jpg', u'DSC00213_4333.jpg', u'DSC00216_4301.jpg', u'DSC00221_3845.jpg', u'DSC00225_3831.jpg', u'DSC00229_4355.jpg', u'DSC00230_3827.jpg', u'DSC00230_4335.jpg', u'DSC00235_4346.jpg', u'DSC00240_4160.jpg', u'DSC00241_3752.jpg', u'DSC00241_4176.jpg', u'DSC00244_4182.jpg', ...]
        pairs = [(u'deduped2016-08-02_14.23.18.JPG', u'all2017-05-23_11.29.11.JPG'), (u'deduped2017-02-10_08.52.13.JPG', u'deduped2017-02-10_08.51.39.JPG'), (u'deduped2016-09-08_12.17.18.JPG', u'deduped2016-09-08_14.10.05.JPG'), (u'deduped2000-01-01_00.47.13.JPG', u'geoscleanup2000-01-01_00.47.24.JPG'), (u'deduped2016-09-07_10.42.44.JPG', u'deduped2016-09-07_09.33.17.JPG'), (u'deduped2016-09-27_09.01.55.JPG', u'deduped2016-09-27_09.02.02.JPG'), (u'deduped2016-08-31_09.57.52.JPG', u'deduped2016-08-31_09.59.03.JPG'), (u'geoscleanup2000-01-01_00.23.35.JPG', u'geoscleanup2000-01-01_00.24.35.JPG'), (u'deduped2016-09-21_11.45.01.JPG', u'deduped2016-08-31_09.12.46.JPG'), (u'deduped2017-06-16_08.50.19.JPG', u'deduped2017-06-16_08.50.25.JPG'), (u'deduped2017-06-17_09.04.08.JPG', u'deduped2017-06-17_09.04.03.JPG'), (u'deduped2017-06-24_09.36.32.JPG', u'deduped2017-06-24_09.18.45.JPG'), (u'deduped2016-09-01_09.52.58.JPG', u'deduped2016-09-01_10.10.46.JPG'), (u'deduped2016-09-02_08.38.28.JPG', u'deduped2016-09-02_13.37.31.JPG'), (u'deduped2016-08-03_14.58.03.JPG', u'geoscleanup2016-07-30_14.22.00.JPG'), (u'deduped2016-09-06_11.11.15.jpg', u'deduped2016-09-06_11.11.11.jpg'), (u'deduped2016-09-21_09.43.26.JPG', u'deduped2016-09-21_09.43.31.JPG'), (u'deduped2016-09-01_09.58.39.JPG', u'deduped2016-09-01_09.58.43.JPG'), (u'deduped2016-09-07_10.06.38.jpg', u'deduped2016-09-07_09.32.13.jpg'), (u'deduped2016-09-07_09.3.

I still need to take a look at the 4 bad files and find out why they are failing, but it seems we are having uncaught errors in the feature extraction step, and then these stop the toolchain at the matching stage.