notebooks/Principal_Component_Analysis.ipynb

{"cells":[{"cell_type":"markdown","source":["# **Principal Component Analysis**"],"metadata":{"id":"6r8rVH_wrVFN","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"1078443d-f492-4808-91a7-144e1e38e76e"}}},{"cell_type":"markdown","source":["## **Import useful Python packages**"],"metadata":{"id":"YOrmY8FiOMwa","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"3d037dbf-d4f7-45e6-8f31-5c09776ce864"}}},{"cell_type":"code","source":["import requests\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\nimport pyspark\nfrom pyspark.sql import *\nfrom pyspark.sql.types import *\nfrom pyspark.sql.functions import *\nfrom pyspark import SparkContext, SparkConf"],"metadata":{"id":"Fh8zPg5APmYv","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"8a81e29c-14b6-4356-b6fc-a4be687a3ede"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["## **Check everything is ok**"],"metadata":{"id":"_eRvyl2xwCV6","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"1eb694cd-0918-489a-a43d-c20f90dd2bff"}}},{"cell_type":"code","source":["spark"],"metadata":{"id":"4fqJ5f0JE3BL","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"3129e018-434e-4664-8f28-13c000c148e7"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["sc._conf.getAll()"],"metadata":{"id":"qhTN342EEOYZ","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"2c4af690-73a2-44d8-b433-7de06ad02787"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["# **Data Acquisition**"],"metadata":{"id":"Mu77QbD2vC_o","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"1dedea6f-01bc-4012-a34d-44ab774337c5"}}},{"cell_type":"markdown","source":["## **MNIST Dataset**\n\n[MNIST](https://www.kaggle.com/c/digit-recognizer/data) (\"Modified National Institute of Standards and Technology\") is the de facto \"Hello World\" dataset of computer vision. Since its release in 1999, this classic dataset of handwritten digit images has served as the basis for benchmarking classification algorithms."],"metadata":{"id":"STPSeQ4zMaPn","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"a51be0cc-3e44-49bd-8d5a-dfe75d3f5d7e"}}},{"cell_type":"markdown","source":["### Download the dataset to the local driver node's ```/tmp``` folder using ```wget```"],"metadata":{"id":"j5fI3wiGvKBl","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"674f64ff-57b4-4829-8385-ff8acf98e0bd"}}},{"cell_type":"code","source":["%sh wget -P /tmp https://github.com/gtolomei/big-data-computing/raw/master/datasets/mnist-train.csv.bz2"],"metadata":{"id":"kxrLDE_4e7KH","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"909dd41f-b2d7-4cd6-bd22-e4f0b0caeef7"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["%fs ls file:/tmp/"],"metadata":{"id":"quiuGbfyv8vT","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"7f9260dc-76ca-414f-9910-52a3081ab7f2"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Move the file from local driver node's file system to DBFS"],"metadata":{"id":"DevlrMcPw1ZI","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"82db59b0-32a7-41f3-936e-3439cf79089e"}}},{"cell_type":"code","source":["dbutils.fs.mv(\"file:/tmp/mnist-train.csv.bz2\", \"dbfs:/bdc-2020-21/datasets/mnist-train.csv.bz2\")"],"metadata":{"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"d07515d1-b62c-49cf-b8eb-f9d1eb414893"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["%fs ls /bdc-2020-21/datasets/"],"metadata":{"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"a0a1de05-d621-46c1-bed3-f0bb7a1a423d"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Read dataset file into a Spark Dataframe"],"metadata":{"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"1ebbbd89-ae34-4b40-bebd-8e99ddf9201e"}}},{"cell_type":"code","source":["mnist_df = spark.read.load(\"dbfs:/bdc-2020-21/datasets/mnist-train.csv.bz2\", \n                         format=\"csv\", \n                         sep=\",\", \n                         inferSchema=\"true\", \n                         header=\"true\"\n                         )"],"metadata":{"id":"qKi5Hd60FFcX","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"b1dc73af-aad3-40a9-a98b-358b2378c53a"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Check the shape of the loaded dataset, i.e., number of rows and columns"],"metadata":{"id":"IPTd8ep9x74H","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"03597707-9e88-40d4-81f5-3d65d425603a"}}},{"cell_type":"code","source":["print(\"The shape of the dataset is {:d} rows by {:d} columns\".format(mnist_df.count(), len(mnist_df.columns)))"],"metadata":{"id":"JyRyYqeXGA4l","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"26e01875-529a-410f-81ba-b0c8885052c7"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Print out the schema of the loaded dataset"],"metadata":{"id":"-WC4RPQgyEsB","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"41495b73-0db0-48aa-9937-b2a751c888c2"}}},{"cell_type":"code","source":["mnist_df.printSchema()"],"metadata":{"id":"Q3KhtSnvGIwG","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"3f41934b-bbb1-463a-9451-de9d50da69fd"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Dataset Shape and Schema\n\nThe dataset has 785 columns. The first column, called `label`, is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated 28-by-28 pixels image, i.e., a 784-dimensional vector.\n\nEach pixel column in the dataset has a name like <code>pixel**k**</code>, where **`k`** is an integer between 0 and 783, inclusive. \nTo locate this pixel on the image, suppose that we have decomposed **`k`** as **`k`** = `i * 28 + j`, where `i` and `j` are integers between 0 and 27, inclusive. Then <code>pixel**k**</code> is located on row `i` and column `j` of a 28 x 28 matrix, (indexing by zero).\n\nFor example, <code>pixel**31**</code> indicates the pixel that is in the fourth column from the left, and the second row from the top."],"metadata":{"id":"dXlcRfzmNNCv","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"7d87e940-5204-4757-8d68-98ef5cbbb6c3"}}},{"cell_type":"markdown","source":["### Display the first 5 rows of the dataset"],"metadata":{"id":"0Pjib1fiylb6","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"ac4e75cf-4c45-451e-b2e5-7b1d9020dc79"}}},{"cell_type":"code","source":["mnist_df.show(5)"],"metadata":{"id":"a4v-Z92rGXoe","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"5b4ad7f8-71ae-4413-b237-12a807e25cbf"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["# **Data Preprocessing**"],"metadata":{"id":"66pFGdch-BU_","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"7db8203f-baf0-4cb1-a80e-24aff7ca5d27"}}},{"cell_type":"markdown","source":["### Assembling Features into a single column\n\nWe make use of the `VectorAssembler` transformer, which combines a given list of columns into a single vector column."],"metadata":{"id":"XxMxZYeOWGVZ","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"8d4430b7-3403-4b1b-b368-237e77ea50c3"}}},{"cell_type":"code","source":["columns = ['pixel{:d}'.format(k) for k in range(784)]"],"metadata":{"id":"uP79D5otXIHk","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"6b8f9a96-1c31-40b0-a654-09e7c8c2fffe"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["from pyspark.ml.feature import VectorAssembler\n\nassembler = VectorAssembler(inputCols=columns, \n                            outputCol=\"features\")"],"metadata":{"id":"hIcWzsjmWYMn","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"252c2a22-0ded-4088-892a-2c4dd154a5bc"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["mnist_df = assembler.transform(mnist_df)\nmnist_df.select(\"features\", \"label\").show(truncate=False)"],"metadata":{"id":"eNrAcjPxXYgp","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"281e3848-cab1-4e32-a748-eaa5f5c4aa60"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Eventually, retain only 2 columns: assembled features and label"],"metadata":{"id":"ykyuMhkrdGKR","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"d8639b75-38fa-4877-9910-cdd69ffee24a"}}},{"cell_type":"code","source":["mnist_df = mnist_df.select(\"features\", \"label\")"],"metadata":{"id":"yZs3jjvNZRN-","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"4871d140-b0cb-4b1d-bf9f-9464ccb48e79"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["# **Principal Component Analysis (PCA)**"],"metadata":{"id":"Kl92tvdj7Kev","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"7a0d33b0-a7fe-42ea-82ab-d7032b8ebed3"}}},{"cell_type":"code","source":["mnist_df.show(5)"],"metadata":{"id":"RtvqyK51rrGb","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"f8612ef0-e763-4043-9cb5-b806deb0d4b0"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["## **Standardize features**\n\nAs we know, PCA is highly sensitive to feature scale. Let's standardize each feature so that each feature value _x_ is transformed into _z_, as follows:\n$$\n  z = \\frac{x-\\mu}{\\sigma}\n$$\nwhere _mu_ is the sample mean of the feature, and _sigma_ is the unbiased sample standard deviation (computed from all the observations).\nIn such a way, each _z_ will have 0-mean and 1-standard-deviation."],"metadata":{"id":"miMmNWkgCtKo","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"3467d0bf-a09d-4e83-963e-6c63f2f48abc"}}},{"cell_type":"code","source":["from pyspark.ml.feature import StandardScaler\n\nscaler = StandardScaler(inputCol=\"features\", \n                        outputCol=\"std_features\",\n                        withStd=True, withMean=True)\n\n# Compute summary statistics by fitting the StandardScaler\nscalerModel = scaler.fit(mnist_df)\n\n# Normalize each feature to have unit standard deviation.\nmnist_df = scalerModel.transform(mnist_df)"],"metadata":{"id":"OTHlg3IPrJV4","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"d651fc3b-65d4-4b0a-b610-4b71ca63643e"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["mnist_df.show(5, truncate=True)"],"metadata":{"id":"pvVTVijJr8k5","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"36a04ee2-62b4-4dfa-8379-f14d809f2f65"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["## **Run PCA**"],"metadata":{"id":"dXuaAlUPTcdW","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"3f6967c6-40c8-4aba-b2ef-bf9fbdd75cfc"}}},{"cell_type":"code","source":["K = 10 # number of principal components to extract"],"metadata":{"id":"Th0qsp2hbdGK","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"9301ceb9-1468-4b8a-81ff-fb144a9901b8"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["from pyspark.ml.feature import PCA\n\npca_model = PCA(k=K, inputCol=\"std_features\", outputCol=\"pca_features\")\npca_features = pca_model.fit(mnist_df)\npca_mnist_df = pca_features.transform(mnist_df).cache()"],"metadata":{"id":"VVF_bx-v7oRz","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"5f24536e-85ab-46f5-b8f1-a402af990420"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["pca_mnist_df.show(5)"],"metadata":{"id":"CTkJMPL2_eIQ","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"5460d8e1-5aac-48f3-a0cb-8a5b53df9b90"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["## **Plotting**"],"metadata":{"id":"q1ZnV_LDjS9Y","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"be0e9c03-7ff9-4bb0-981f-76fd6d71f650"}}},{"cell_type":"markdown","source":["### Show the proportion of the data variance carried by each principal component (eigenvector)"],"metadata":{"id":"Oq6JtU3NaC4G","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"620a2bc4-2e52-4152-976d-c3d155063e94"}}},{"cell_type":"code","source":["fig, ax = plt.subplots(1,1,figsize=(8,6))\n_ = sns.barplot(x=[i for i in range(K)], \n                y=pca_features.explainedVariance.values, # `explainedVariance` returns the distribution of variance across eigenvectors, i.e., lambda_i/sum lambda_i\n                ax=ax)\n_ = ax.set_xlabel(\"Eigenvalues\", labelpad=16, fontsize=16)\n_ = ax.set_ylabel(\"Proportion of Variance\", fontsize=16)\n_ = ax.set_xticklabels([\"lambda_{:d}\".format(i) for i in range(K)], rotation=45)"],"metadata":{"id":"3yMj2ZJKYnxY","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"00e39e0b-a78f-4651-97a5-4ed23838faed"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Transform the PySpark DataFrame into Pandas' DataFrame"],"metadata":{"id":"5BcdlEdOM3Jo","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"8a98f418-f8a4-40c8-9d51-74e54fffe9f1"}}},{"cell_type":"code","source":["pca_mnist_pdf = pca_mnist_df.toPandas()"],"metadata":{"id":"ZZHkMQOfNAe-","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"9a2dd957-b810-4aaa-b00a-04bed22fb2f3"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["pca_mnist_pdf.head()"],"metadata":{"id":"f_ZPInkPjdWq","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"1d6f758d-3047-4ecd-b461-f79e7c783092"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Plot data projected on the 2 principal components"],"metadata":{"id":"jnTeSniEdhK4","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"a386333f-a5ff-47a8-bc62-623889e8e26a"}}},{"cell_type":"code","source":["fig, ax = plt.subplots(1,1,figsize=(12,8))\n_ = plt.scatter(\n    pca_mnist_pdf.pca_features.map(lambda x: x[0]), \n    pca_mnist_pdf.pca_features.map(lambda x: x[1]),\n    c=pca_mnist_pdf.label, \n    edgecolor='none', \n    alpha=0.8,\n    cmap=plt.cm.get_cmap('tab10', 10),\n    axes=ax\n    )\n_ = ax.set_xlabel('PCA 1', labelpad=20, fontsize = 16)\n_ = ax.set_ylabel('PCA 2', fontsize = 16)\nplt.colorbar();"],"metadata":{"id":"oSBx4JXojaM5","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"fef5cc4c-429c-408b-8efc-c60dcb8183a4"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["# **K-means Clustering**"],"metadata":{"id":"ZE2DTA2NdfSA","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"ebd16c41-04ed-4c58-b750-5a92ad3e6dcd"}}},{"cell_type":"markdown","source":["### Function used for running K-means"],"metadata":{"id":"jPdq2yVE9LzC","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"9a55cf7e-9f34-4bff-a86b-14235fd826fa"}}},{"cell_type":"code","source":["RANDOM_SEED = 42\n\ndef k_means(dataset, \n            n_clusters, \n            distance_measure=\"euclidean\", \n            max_iter=20, \n            features_col=\"features\", \n            prediction_col=\"cluster\", \n            random_seed=RANDOM_SEED):\n  \n  from pyspark.ml.clustering import KMeans\n\n  print(\"\"\"Training K-means clustering using the following parameters: \n  - K (n. of clusters) = {:d}\n  - max_iter (max n. of iterations) = {:d}\n  - distance measure = {:s}\n  - random seed = {:d}\n  \"\"\".format(n_clusters, max_iter, distance_measure, random_seed))\n  # Train a K-means model\n  kmeans = KMeans(featuresCol=features_col, \n                   predictionCol=prediction_col, \n                   k=n_clusters, \n                   initMode=\"k-means||\", \n                   initSteps=5, \n                   tol=0.000001, \n                   maxIter=max_iter, \n                   seed=random_seed, \n                   distanceMeasure=distance_measure)\n  model = kmeans.fit(dataset)\n\n  # Make clusters\n  clusters_df = model.transform(dataset).cache()\n\n  return model, clusters_df"],"metadata":{"id":"JP1deqVUurtB","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"1ea81bf4-9844-4580-b316-df52b1e989d3"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Function used to evaluate obtained clusters"],"metadata":{"id":"pi3FUnyx9ZWi","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"63e45b3f-823d-4c08-9379-ff76b7ed5f4e"}}},{"cell_type":"code","source":["def evaluate_k_means(clusters, \n                     metric_name=\"silhouette\", \n                     distance_measure=\"squaredEuclidean\", # cosine\n                     prediction_col=\"cluster\"\n                     ):\n  \n  from pyspark.ml.evaluation import ClusteringEvaluator\n  \n  # Evaluate clustering by computing Silhouette score\n  evaluator = ClusteringEvaluator(metricName=metric_name,\n                                  distanceMeasure=distance_measure, \n                                  predictionCol=prediction_col\n                                  )\n\n  return evaluator.evaluate(clusters)"],"metadata":{"id":"nZTOi7TWvcXJ","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"1db0c12d-3166-4564-a75a-c365135c022e"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Run K-means by calling the function above"],"metadata":{"id":"T1p5kISG9et-","colab_type":"text","application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"e7e86be9-8c30-4e15-8747-c1fefb22b89e"}}},{"cell_type":"code","source":["model, clusters_df = k_means(pca_mnist_df, \n                             10,\n                             distance_measure=\"euclidean\",\n                             max_iter=1000, \n                             features_col=\"pca_features\"\n                             )"],"metadata":{"id":"ZgvwHoeBvmzY","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"2d11f1d9-a957-40b6-a67e-34b92d524279"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["evaluate_k_means(clusters_df)"],"metadata":{"id":"TYu_vheKwTc0","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"6103b8ae-4ec9-41df-a467-c3b7f7f403de"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["clusters_df.show(5)"],"metadata":{"id":"Q_1-52c-HtVh","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"842cd156-1b00-461d-84d5-d8b84fb43bb5"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["clusters_df.groupBy(\"cluster\").count().sort(\"cluster\").show()"],"metadata":{"id":"O2MLijHVJHIi","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"bf14a751-a2ff-4cde-bf4e-7718d5d8a592"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["# Get unique values in the grouping column\nclusters = sorted([x[0] for x in clusters_df.select(\"cluster\").distinct().collect()])\nprint(\"Cluster IDs: [{:s}]\".format(\", \".join([str(c) for c in clusters])))\n\n# Create a filtered DataFrame for each group in a list comprehension\ncluster_list = [clusters_df.where(clusters_df.cluster == x) for x in clusters]\n\n# Show the results\nfor x_id, x in enumerate(cluster_list):\n  print(\"Showing the first 10 records of cluster ID #{:d}\".format(x_id))\n  x.select([\"cluster\", \"label\"]).show(10, truncate=False)"],"metadata":{"id":"Qqft7DRcHUoG","colab_type":"code","colab":{},"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"a6a6f74e-8a80-434f-a30c-302e99d37324"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["## **Compute Cluster Purity**"],"metadata":{"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"424ad3a7-ee5d-4372-a3e2-7538c327c543"}}},{"cell_type":"code","source":["def purity(cluster_i):\n  # get total count of items clustered in cluster_i\n  n_i = cluster_i.count()\n  # group by labels\n  count_labels = cluster_i.groupBy(\"label\").count()\n  # find the maximum count\n  max_i = count_labels.agg(max(\"count\")).first()[0]\n  # find the label associated with the maximum count\n  mode = count_labels.join(\n    count_labels.agg(max(\"count\").alias(\"max_\")), col(\"count\") == col(\"max_\")).limit(1).select(\"label\")\n  \n  # finally, return the most frequent label (of the current cluster) along with its purity\n  return mode.first()[0], max_i/n_i"],"metadata":{"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"4eefb1fd-ac79-49b8-8ad3-fef95b1b5b3d"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"code","source":["purities = []\nfor x_id, x in enumerate(cluster_list):\n  print(\"The most frequent label of cluster id #{:d} is {:d} [purity = {:.3f}]\".format(x_id, purity(x)[0], purity(x)[1]))\n  purities.append(purity(x)[1])\n\nprint(\"The overall average purity is: {:.3f}\".format(np.mean(purities)))"],"metadata":{"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"83d6e22c-9041-40ab-b758-b40b93575de1"}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"type":"ipynbError","data":"","errorSummary":"","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"]}}],"execution_count":0},{"cell_type":"markdown","source":["## **Suggestion: Try Computing Precision, Recall, F1, and Adjusted Rand Index**\n\nYou can try computing additional external metrics as the ones indicated above. A good reference for doing this can be found at the following [link](https://towardsdatascience.com/evaluating-clustering-results-f13552ee7603)."],"metadata":{"application/vnd.databricks.v1+cell":{"title":"","showTitle":false,"inputWidgets":{},"nuid":"cacd669c-d83a-4f60-8a0a-d27ca6f44aaa"}}}],"metadata":{"colab":{"name":"Lecture_07_PCA.ipynb","provenance":[],"collapsed_sections":[],"machine_shape":"hm"},"kernelspec":{"name":"python3","display_name":"Python 3"},"application/vnd.databricks.v1+notebook":{"notebookName":"Principal_Component_Analysis","dashboards":[],"notebookMetadata":{"pythonIndentUnit":2},"language":"python","widgets":{},"notebookOrigID":1514207521556430}},"nbformat":4,"nbformat_minor":0}