Merge pull request #134 from madratman/PR/final_round

Final round

Author: madratman

README.md

@@ -12,6 +12,16 @@
 
 ### Downloading and running AirSim Binaries
 #### Downloading
+- Final round binaries and environments (v1.1)
+	- tl;dr:
+		- [Linux] Use the [download_final_round_binaries.sh](download_final_round_binaries.sh) script
+	- Long version:
+		- Download the v1.1 [Linux](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/releases/tag/v1.1-linux) or [Windows](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/releases/tag/v1.1-windows) `AirSim.zip`, and unzip it. 
+		- Download your qualifier environments (shipped in pakfiles) - `Final_Tier_1_and_Tier_2.pak`  and ` Final_Tier_3.pak`.
+		- Move the environment pakfiles into `AirSim/AirSimExe/Content/Paks`. 
+		- Download and move the `settings.json` file to `~/Documents/AirSim/settings.json`. 
+		- Use `airsimneurips` >= 1.2.0
+
 - Qualifier binaries and environments (v1.0)
 	- tl;dr:
 		- [Linux] Use the [download_qualification_binaries.sh](download_qualification_binaries.sh) script
@@ -108,7 +118,7 @@ We recommend you used python >= 3.6. Python 2.7 will go [out of support soon](ht
 	- Going through both open and closed issues in this repository might answer some of your questions. The search bar on top left can prove useful.    
 	- [AirSim upstream API](https://microsoft.github.io/AirSim/docs/apis/) and [examples](https://github.com/microsoft/AirSim/tree/master/PythonClient) can also be of use. However, please note that the main AirSim repo's API is not used in the competition (there's some overlap and some differences), however is a good learning resource. 
 
-## Submitting Results and Leaderboard
+## Submitting Results and Leaderboard - Qualification Round
 - For the qualification round, we have one race track for each tier. The relevant binaries (v1.0) are available for [linux](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/releases/tag/v1.0-linux) and [windows](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/releases/tag/v1.0-windows)
 	- Tier 1: This is in the Soccer Field environment.    
 	THe race track is in the `Qual_Tier_1_and_Tier_3.pak` pakfile
@@ -164,7 +174,58 @@ Please read [the race monitoring section](https://github.com/microsoft/AirSim-Ne
 	- We have emailed you a private key, which should be entered in the `Team ID` field. This helps us verify it was your team who indeed made the submission.   
 	- The [leaderboard](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/leaderboard.html) is updated once per day at 2100 PST.   
 	If you do not see your results after 24 hours, please [email us](mailto:[email protected]) with your team name and submitted log files.
-	
+
+## Submitting Results and Leaderboard - Final Round
+- For the final round, we have one race track for each tier. The relevant binaries (v1.1) are available for [linux](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/releases/tag/v1.1-linux) and [windows](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/releases/tag/v1.1-windows)
+	- Tier 1: This is in the Soccer Field environment.    
+	THe race track is in the `Final_Tier_1_and_Tier_2.pak` pakfile
+	- Tier 2: This is in the Soccer Field environment.    
+	The race track is in the `Final_Tier_1_and_Tier_2.pak` pakfile. 
+	- Tier 3: This is again in the ZhangJiaJie environment.    
+	The race track is in the `Final_Tier_3.pak` pakfile. 
+
+- How to generate logfiles for each tier:
+	- Loading level and starting race:
+		- Please update your airsimneurips pythonclient (should be >=1.2.0). 
+		- Calling `simStartRace(race_tier=1, 2, or 3)` generates the appropriate log files. You can only run `tier N` races in `Final_Tier_N` levels. 
+		- Tier 1: 
+			```python
+				airsim_client.simLoadLevel('Final_Tier_1')
+				airsim_client.simStartRace(tier=1)
+			```
+
+		- Tier 2: 
+			```python
+				airsim_client.simLoadLevel('Final_Tier_2')
+				airsim_client.simStartRace(tier=2)
+			```
+
+		- Tier 3: 
+			```python
+				airsim_client.simLoadLevel('Final_Tier_3')
+				airsim_client.simStartRace(tier=3)
+			```
+	- As Tier 2 focuses on perception and Tier 3 focuses on both perception and planning, note that `simGetObjectPose` returns noisy gate poses.
+
+	- As soon as `simStartRace(tier=1)`  or `simStartRace(tier=3)` is called, `drone_2` (MSR opponent racer) will start flying. 
+
+	- See `baseline_racer.py` for sample code. The previous bullet points are being called in wrapper functions in the following snippet in `baseline_racer.py`:
+	```python
+		baseline_racer.load_level(args.level_name)
+		baseline_racer.start_race(args.race_tier)
+	```
+
+- To submit your results to the final leaderboard:
+	- Navigate to the [submission site](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/upload.html), enter your team name in the proper field, and upload any number of [race logs](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/blob/master/docs/competition_guidelines.md#race-monitoring).   
+It's ok to make a submission for as little as a single track and/or a single tier.   
+You can find race logs inside of `AirSimExe/Saved/Logs/RaceLogs` in your downloaded binary folder.   
+Please read [the race monitoring section](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/blob/master/docs/competition_guidelines.md#race-monitoring) in the competition guidelines for more details. 
+	- The leaderboard will publish the results of a drone that is named `drone_1` (call [`generate_settings_file.py`](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/blob/master/baselines/generate_settings_file.py) to generate an AirSim settings file, as done for the `baseline_racer` below. 
+	- Please submit a PDF file in the `report` section to help us verify the honesty of your submission by the Dec 5th, 2359 PST deadline. Please summarize your approach for all tiers you make a submission for, with appropriate citations. The report PDF size should not exceed 10 MB, and should be a maximum of 6 pages in length. We leave the exact format of the report to your descrition, but the [IEEE template](https://ras.papercept.net/conferences/support/tex.php) is a good choice.  
+	- We have emailed you a private key, which should be entered in the `Team ID` field. This helps us verify it was your team who indeed made the submission.   
+	- The [final leaderboard](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/leaderboard_final.html) is updated once per day at 2100 PST.   
+	If you do not see your results after 24 hours, please [email us](mailto:[email protected]) with your team name and submitted log files.
+
 ## Sample code
  - Plan and move on minimum jerk trajectory using gate ground truth poses:
     - Generate an AirSim settings.json file
@@ -203,35 +264,97 @@ Please read [the race monitoring section](https://github.com/microsoft/AirSim-Ne
 		- R. Spica, D. Falanga, E. Cristofalo, E. Montijano, D. Scaramuzza, and M. Schwager, "A Real-Time Game Theoretic Planner for Autonomous Two-Player Drone Racing", in the Proccedings of Robotics: Science and Systems (RSS), 2018. 
 
 ## Quick API overview 
-- Changing unreal environments  
-	There are two ways to swap between environments / "unreal level", either via AirSIm API or by the UI menu.
-	- Python API 
-	Use `simLoadLevel(level_name="MainMenu")` to change Unreal environments on the fly.   
-	 Possible values for `level_name` are : `"Soccer_Field_Easy"`, `"Soccer_Field_Medium"`, `"ZhangJiaJie_Medium"`, `"Building99_Hard"`. 
-	Here's a quick snippet to iterate throught all the training environments.   
-	Before trying this, please ensure you've downloaded the corresponding training (v0.3) / qualifier (v1.0) binaries, [as described above](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing#downloading-airsimexe-and-unreal-environments)
-
-		```python
-		import airsimneurips as airsim
-		client = airsim.MultirotorClient()
-		client.confirmConnection()
-
-		# use this for training environments (v0.3)
-
-		client.simLoadLevel('Soccer_Field_Easy')	
-		client.simLoadLevel('Soccer_Field_Medium')	
-		client.simLoadLevel('ZhangJiaJie_Medium')
-		client.simLoadLevel('Building99_Hard')
-
-		# use this for qualification environments (v1.0)
-		client.simLoadLevel('Qualification_Tier_1')	
-		client.simLoadLevel('Qualification_Tier_2')	
-		client.simLoadLevel('Qualification_Tier_3')	
+We added some new APIs (marked with 💚) to [AirSim](https://github.com/Microsoft/Airsim) for the NeurIPS competition binaries. 
 
-		```
-	- UI Menu
-		- Press `F10` to toggle the level menu
-		- Click your desired level. (Note: the UI lists all the pakfiles in the `AirSim/AirSimExe/Content/Paks` directory. Ensure you downloaded the pakfile, if you are not able to see a particular environment)
+#### Loading Unreal Engine environments  
+- [`simLoadLevel(level_name)`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.level_name) 💚    
+Possible values for `level_name` are: 
+	- `"Soccer_Field_Easy"`, `"Soccer_Field_Medium"`, `"ZhangJiaJie_Medium"`, `"Building99_Hard"` in the training binaries (`v0.3`). 
+	- `"Qualification_Tier_1"`, `"Qualification_Tier_2"`, `"Qualification_Tier_3"` in the qualification binaries (`v1.0`). 
+	- `"Final_Tier_1"`, `"Final_Tier_2"`, `"Final_Tier_3"` in the final round binaries (`v1.1`). 
+Before trying this, please ensure you've downloaded the corresponding training (`v0.3`) / qualifier (`v1.0`) / final round (`v1.0`) binaries, [as described above](https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing#downloading-airsimexe-and-unreal-environments)
+
+- UI Menu
+	- Press `F10` to toggle the level menu
+	- Click your desired level. (Note: the UI lists all the pakfiles in the `AirSim/AirSimExe/Content/Paks` directory. Ensure you downloaded the pakfile, if you are not able to see a particular environment)
+
+#### Race APIs:
+- Start a race:
+	[`simStartRace(tier=1/2/3)`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simStartRace) 💚
+
+- Reset race:
+	[`simResetRace()`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simResetRace) 💚
+
+- Check if racer is disqualified:
+	[`simIsRacerDisqualified()`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simIsRacerDisqualified) 💚
+
+- Get index of last gate passed:
+	[`simGetLastGatePassed()`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simGetLastGatePassed) 💚
+
+- Disable generation of logfiles by race APIs:
+	[`simDisableRaceLog`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simDisableRaceLog) 💚
+
+#### Lower level control APIs:
+- FPV like Angle rate setpoint APIs: 
+	- [`moveByAngleRatesThrottleAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveByAngleRatesThrottleAsync) 💚
+	- [`moveByAngleRatesZAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveByAngleRatesZAsync) 💚 (stabilizes altitude)
+
+- Angle setpoint APIs:  
+	- [`moveByRollPitchYawThrottleAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveByRollPitchYawThrottleAsync) 💚
+	- [`moveByRollPitchYawZAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveByRollPitchYawZAsync) 💚 (stabilizes altitude)
+
+- RollPitchYawrate setpoint APIs: 
+	- [`moveByRollPitchYawrateThrottleAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveByRollPitchYawrateThrottleAsync) 💚
+	- [`moveByRollPitchYawrateZ`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveByRollPitchYawrateZ) 💚 (stabilizes altitude)
+
+#### Medium level control APIs:
+- Velocity setpoints
+	- [`moveByVelocityAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveByVelocityAsync)
+	- [`moveByVelocityZAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveByVelocityZAsync) (stabilizes altitude)
+
+- Position setpoints
+	- [`moveToPosition`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveToPositionAsync)
+	- [`moveOnPath`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveOnPathAsync)
+	- [`moveToZAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveToZAsync)
+
+#### High level control APIs:
+- Minimum jerk trajectory planning (using [ethz-asl/mav_trajectory_generation](https://github.com/ethz-asl/mav_trajectory_generation)), and trajectory tracking (using a pure pursuit like controller minimizing position and velocity errors), with position setpoints. 
+	Optionally use the `*lookahead*` parameters to start new trajectory from a point sampled `n` seconds ahead for trajectory being tracked currently. 
+	- [`moveOnSplineAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveOnSplineAsync) 💚
+
+- Minimum jerk trajectory planning (using [ethz-asl/mav_trajectory_generation](https://github.com/ethz-asl/mav_trajectory_generation)), and trajectory tracking (using a pure pursuit like controller minimizing position and velocity errors), with position setpoints and corresponding velocity constraints. Useful for making a drone go through a gate waypoint, while obeying speed and direction constraints. 
+	Optionally use the `*lookahead*` parameters to start new trajectory from a point sampled `n` seconds ahead for trajectory being tracked currently. 
+	- [`moveOnSplineVelConstraintsAsync`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.moveOnSplineVelConstraintsAsync) 💚
+
+- Clear and stop following current trajectory. 
+	- [`clearTrajectory`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.clearTrajectory) 💚
+
+#### Gain setter APIs:
+- [`setAngleRateControllerGains`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.setAngleRateControllerGains) 💚
+- [`setAngleLevelControllerGains`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.setAngleLevelControllerGains) 💚
+- [`setVelocityControllerGains`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.setVelocityControllerGains) 💚
+- [`setPositionControllerGains`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.setPositionControllerGains) 💚
+- [`setTrajectoryTrackerGains`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.setTrajectoryTrackerGains) 💚 
+
+#### APIs to help generate gate detection datasets:
+- Object pose setter and getter: 
+	- [`simSetObjectPose`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simSetObjectPose)
+	- [`simGetObjectPose`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simGetObjectPose)
+
+- Object scale setter and getter: 
+	- [`simSetObjectScale`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simSetObjectScale) 💚
+	- [`simGetObjectScale`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simGetObjectScale) 💚
+
+- Object segmentation ID setter and getter: 
+	- [`simGetSegmentationObjectID`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simGetSegmentationObjectID)
+	- [`simSetSegmentationObjectID`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simSetSegmentationObjectID)
+
+- Listing all the objects in the scene: 
+	- [`simListSceneObjects`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simListSceneObjects) 💚
+
+- Gate specific APIs: 
+	- [`simGetNominalGateInnerDimensions`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simGetNominalGateInnerDimensions) 💚
+	- [`simGetNominalGateOuterDimensions`](https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/api.html#airsimneurips.client.MultirotorClient.simGetNominalGateOuterDimensions) 💚
 
 ## Questions
 Please open a Github Issue on **this** repository (not [AirSim](https://github.com/microsoft/AirSim)) for any technical questions w.r.t. the Neurips competition.