Final Froject of MUSA620-Data-Wrangling in UPenn
Author: Yichen Lei
December 14, 2019
Airbnb, as a paid housing sharing system, has flourished in recent years. This unique business model provides travelers with an alternative to hotels, and also allows homeowners with vacant homes to rent out on the platform for revenue. Different from hotel rooms, there are many rooms on Airbnb whose properties are totally different Thus, it is difficult to have a unified pricing suitable for diverse rooms. Helping homeowners price their homes is the goal of this study.
This project uses Random Forest to predict the price of Airbnb in San Francisco using the properties of the house and the spatial properties of its location. Finally, an effective algorithm with a score of 92 on the test set was constructed. In addition, based on the prediction results of the algorithm, I built a web app to check the predicted price of Airbnb in San Francisco. It is built using panels, allowing users to filter by price range, review ratings, number of rooms, and house type. A histogram based on neighborhood statistics can change at the same time as the selection conditions change. Finally, there is a map showing the predicted error percentage, which provides users with the reliability of the results of the census tracts.
The Airbnb data are downloaded from Insight Airbnb, which are composed of four datasets. The four Airbnb files in San Francisco are CSV format, which contain data for four times scraping from September to December in 2019. San Francisco census tract data is download from SF Open Data, which is geojson format.
The first is data cleaning. The raw data from Insight Airbnb has many null values and outliers, as well as misplaced values and types. I used for loops to traverse to clean the wantted attribute columns, and then used spatial join to connect the attributes of the census tracts to Airbnb data. This step also removes spatial outliers.
I create a new attribute "amenities_count". According to my observations, the original attribute "amenities" is the homeowners' enumerations of room amenities, and they are separated by commas. So I count the commas in the amenities strings to reflect the number of facilities in the house. The importance of the predictors barplot shows that its importance is in the top five.
The random forest algorithm of this project uses the number of rooms, house type, review score, census tracts, etc. to predict the log_transformed price. In total, there are 18 continuous variables and 5 categorical variables. Because "property_type" contains too many features, I performed feature engineering on it. I kept 'Apartment', 'Condominium', 'Hotel', and 'House' and classified the other types as 'others'.
By using 3-fold cross validation, I found out that "n_estimators = 100, max_depth = 50" is the best hyperparameter. (I removed 3-fold cross validation in final code to reduce runtime). The final model scored 0.92 on the test set.
Finally, I created the map "Error Percentage by Census Tracts" and predictors' importance barplot, which can help users understand the results of the model.
The web app is built using panel. The main heatmap shows the spatial location of Airbnb. It can show the results that meet the requirements: predict price, review ratings, number of rooms, and room type. In the app function, I pass these variables to control functions. Points function controls the points to meet users' demands. Based on the results of the points function subset, the hist function counts the number in each neighborhood, which achieves the linkage with heatmap. Finally, I added the map of prediction error percentage to the web. This is a key result of the model and can tell users which regions have reliable prediction results.
The test score of model is 0.92. "bedrooms" is the most important predictor in the model, whose importance is 0.368. The mean error percentage is 11.84%, the medium value of it is 6.66%. The census with max error percentage is 332.03; the census with min error percentage is 9802.
The screen shot of my Web App is below:
