-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.xml
More file actions
executable file
·2772 lines (2641 loc) · 280 KB
/
Copy pathindex.xml
File metadata and controls
executable file
·2772 lines (2641 loc) · 280 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>tanja's website</title>
<link>/</link>
<atom:link href="/index.xml" rel="self" type="application/rss+xml" />
<description>tanja's website</description>
<generator>Wowchemy (https://wowchemy.com)</generator><language>en-us</language><lastBuildDate>Tue, 23 Feb 2021 00:00:00 +0100</lastBuildDate>
<image>
<url>/media/TK.jpg</url>
<title>tanja's website</title>
<link>/</link>
</image>
<item>
<title>Learn to access, organise, analyse and argue with data using R</title>
<link>/training/example/example4/</link>
<pubDate>Fri, 20 Nov 2020 00:00:00 +0100</pubDate>
<guid>/training/example/example4/</guid>
<description>
<p>The ever emerging Internet of Things (IoT) produces vast amounts of data that can be analysed to gain useful insights into trends. The R language is widely used by statisticians for data analysis, and the popularity of R programming has increased greatly in the recent years.</p>
<p>In this workshop we teach you to ‘speak data using R’ by taking a step by step approach, starting by explaining core programming principles of the R programming language, to importing data and employing basic wrangling techniques and the grammar of graphics’ philosophy. This will enable you to produce compelling visualisations that can ultimately illustrate useful insights from your data.</p>
<p>Through a series of demonstrations and hands on exercises you will learn some of the fundamental concepts of R and you will get to know how to use its tools in a typical data science project. We will introduce you to graphical and numerical techniques for exploring the information concealed within a dataset. After we develop an understanding of our data we will use R’s reproducible approach in telling our data story by creating R-Markdown documents.</p>
<p>At the end of the workshop, participants will be able to:</p>
<ul>
<li>Import data from different sources into the R environment</li>
<li>Carry out basic data pre-processing &amp; wrangling</li>
<li>Learn to distinguish the most appropriate visualisations to be used for any given data challenge</li>
<li>Build advanced visualisations: graphs and maps</li>
<li>Gain proficiency in data pre-processing, data wrangling and visualisation in R by putting acquired knowledge into application, adopting a reproducible approach through the creation of RMarkdown documents</li>
</ul>
<p><div class="alert alert-note">
<div>
This course is available at 👉 <a href="https://introtor.netlify.app/" class="uri"><a href="https://introtor.netlify.app/">https://introtor.netlify.app/</a></a>
</div>
</div>
</p>
<p>👉 Go to the following GitHub repo to download the material: <a href="https://github.com/TanjaKec/ASDA" class="uri">https://github.com/TanjaKec/ASDA</a></p>
<hr />
<p>This workshop has been delivered for <a href="https://www.asda.com/">ASDA</a> data development training.</p>
<p>The workshop has been delivered in partnership with <a href="https://geolytix.co.uk/">GEOLYTIX</a>.</p>
</description>
</item>
<item>
<title>Build a website with blogdown in R</title>
<link>/training/example/example2/</link>
<pubDate>Sun, 05 Jul 2020 00:00:00 +0100</pubDate>
<guid>/training/example/example2/</guid>
<description>
<p>The workshop provides a practical guide for crateing websites using the <a href="https://cran.r-project.org/web/packages/blogdown/blogdown.pdf"><code>blogdown</code></a> package in R that allows you to create websites from <a href="https://cran.r-project.org/web/packages/rmarkdown/rmarkdown.pdf">R Markdown</a> files using <a href="https://gohugo.io/">Hugo</a>, an open-source static site generator written in <a href="https://golang.org/">Go</a>. In this workshop you learn how to create dynamic R Markdown documents to build static websites allowing you to use R code to render the results of your analysis. The <code>blogdown</code> through the use of R Markdown allows technical writing allowing you to add graphs, tables, <a href="https://www.latex-project.org/">LaTeX</a> equations, theorems, citations, and references. This makes <code>blogdown</code> an perfect tool for designing websites to communicate your R data story telling or just awesome general-purpose websites. After you create your awesome website using the <code>blogdown</code> and Hugo template you can push it onto your <a href="https://github.com/">GitHub</a> and deploy on <a href="https://www.netlify.com/">Netlify</a> all free of charge.</p>
<p>Objectives:</p>
<ul>
<li>Apply <a href="https://themes.gohugo.io/">HUGO theme</a> to create a website using the <code>blogdown</code></li>
<li>Personalise the website</li>
<li>Deploy site from your computer to the Internet</li>
<li>Use GitHub for version control</li>
<li>Use Netlify for continuous deployment</li>
<li>Updating your website: serving site, push to GitHub, deploy</li>
</ul>
<p><div class="alert alert-note">
<div>
This course is available at 👉 <a href="https://websiteinr.netlify.app/" class="uri"><a href="https://websiteinr.netlify.app/">https://websiteinr.netlify.app/</a></a>
</div>
</div>
</p>
<p>👉 Go to the following GitHub repo to download the material: <a href="https://github.com/TanjaKec/eRum2020" class="uri">https://github.com/TanjaKec/eRum2020</a>
and follow the steps given in the Xaringan presentation available from 👉 <a href="https://erum2020.netlify.app/">here</a></p>
<div class="figure"><span id="fig:embed2"></span>
<iframe src="https://erum2020.netlify.app/#1" width="672" height="400px" style="border: none;">
</iframe>
<p class="caption">
Figure 1: Xaringan presentation
</p>
</div>
<hr />
<p>The workshop has been delivered at <a href="https://2020.erum.io/">eRum!2020 - European R Users Meeting</a></p>
</description>
</item>
<item>
<title>Reproducible Research</title>
<link>/training/example/example1/</link>
<pubDate>Sat, 15 Feb 2020 00:00:00 +0100</pubDate>
<guid>/training/example/example1/</guid>
<description>
<p>This course is designed to give you an appreciation of R programming as a tool for reproducible research. It focuses on packages that will help you access your data, do necessary visualisation and communication in a dynamic and reproducible manner.</p>
<div id="if-you-would-like-to-learn-to" class="section level4">
<h4>If you would like to learn to:</h4>
<ul>
<li>Store in one place your research methods, results and interpretation</li>
<li>Update your research methodology and automate the update of the results</li>
<li>Write your research text and code in R Markdown</li>
<li>Generate reproducible reports that display your code and results</li>
<li>Assemble your research reporting as either HTML, a PDF or a Word document using the package rmarkdown</li>
</ul>
</div>
<div id="then-this-course-is-for-you" class="section level4">
<h4>then this course is for you!</h4>
<p><div class="alert alert-note">
<div>
This course is available at 👉 <a href="https://reproducibleresearchinr.rbind.io" class="uri"><a href="https://reproducibleresearchinr.rbind.io">https://reproducibleresearchinr.rbind.io</a></a>
</div>
</div>
</p>
<hr />
<p>The course has been delivered at <a href="http://www.bos.rs/en/">Belgrade Open School (BOS)</a> and <a href="https://ifdt.bg.ac.rs">the Institute of Philosophy and Social Theory (IFDT)</a> at <a href="https://www.bg.ac.rs/en/">the University of Belograde</a>.</p>
</div>
</description>
</item>
<item>
<title>Data challenge with R</title>
<link>/training/example/example3/</link>
<pubDate>Sun, 03 Nov 2019 00:00:00 +0100</pubDate>
<guid>/training/example/example3/</guid>
<description>
<p>This course is designed to give you an appreciation of R programming as a tool for data exploration. It focuses on packages that will help you do exploratory data analysis, visualisation and communication in a dynamic and reproducible manner.</p>
<div id="if-you-would-like-to" class="section level4">
<h4>If you would like to:</h4>
<ul>
<li>Discover how to find and access data and prepare it for exploration and visualistion</li>
<li>Learn to explore, visualize, and analyse data in a dynamic and reproducible manner</li>
<li>Gain experience in data wrangling, exploratory data analysis and data visualisation, and effective communication of results</li>
<li>Work on case studies inspired by real problems and based on open data</li>
</ul>
</div>
<div id="then-this-course-is-for-you" class="section level4">
<h4>then this course is for you! 😀</h4>
<p><div class="alert alert-note">
<div>
This course is available at 👉 <a href="https://datachallengewithr.rbind.io/" class="uri"><a href="https://datachallengewithr.rbind.io/">https://datachallengewithr.rbind.io/</a></a>
</div>
</div>
</p>
<hr />
<p>This workshop is a part of The Autumn Data School program organised as a part of the <a href="https://www.rs.undp.org/content/serbia/en/home/projects/opendata.html">Open Data - Open Opportunities</a> project.</p>
<p>You can download the Autumn Data School program from <a href="https://www.rs.undp.org/content/dam/serbia/undp_rs_Program%20Radionice_Open%20Data-1.pdf">here 👇</a></p>
<p>The workshop has been delivered in partnership with <a href="http://www.rs.undp.org">UNDP Serbia</a></p>
</div>
</description>
</item>
<item>
<title>Subset Variable Selection in R</title>
<link>/post/breg_cv/</link>
<pubDate>Tue, 23 Feb 2021 00:00:00 +0100</pubDate>
<guid>/post/breg_cv/</guid>
<description>
<script src="/rmarkdown-libs/header-attrs/header-attrs.js"></script>
<script src="/rmarkdown-libs/htmlwidgets/htmlwidgets.js"></script>
<script src="/rmarkdown-libs/jquery/jquery.min.js"></script>
<link href="/rmarkdown-libs/datatables-css/datatables-crosstalk.css" rel="stylesheet" />
<script src="/rmarkdown-libs/datatables-binding/datatables.js"></script>
<link href="/rmarkdown-libs/dt-core/css/jquery.dataTables.min.css" rel="stylesheet" />
<link href="/rmarkdown-libs/dt-core/css/jquery.dataTables.extra.css" rel="stylesheet" />
<script src="/rmarkdown-libs/dt-core/js/jquery.dataTables.min.js"></script>
<link href="/rmarkdown-libs/crosstalk/css/crosstalk.css" rel="stylesheet" />
<script src="/rmarkdown-libs/crosstalk/js/crosstalk.min.js"></script>
<hr />
<p>This short tutorial on Subset variable Selection in R comes from pp. 244-251 of <a href="https://www.statlearning.com">“Introduction to Statistical Learning with Applications in R” by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani</a> and chapter “Forward, Backward, and Stepwise Selection”, from pp. 511-519 of <a href="http://appliedpredictivemodeling.com">“Applied Predictive Modeling”</a> by <a href="https://twitter.com/topepos">Max Kuhn</a> and Kjell Johnson.</p>
<p>For the large numbers of explanatory variables, and many interactions and non-linear terms, the process of model simplification can take a very long time. There are many algorithms for automatic variable selection that can help to chose the variables to include in a regression model.</p>
<div id="using-best-subsets-regression-and-stepwise-regression" class="section level2">
<h2>Using Best Subsets Regression and Stepwise Regression</h2>
<p><strong>Stepwise regression</strong> and <strong>Best Subsets regression</strong> (<strong>BREG</strong>) are two of the more common variable selection methods. The <em>stepwise</em> procedure starts from the saturated model (or the maximal model, whichever is appropriate) through a series of simplifications to the minimal adequate model. This progression is made on the basis of deletion tests: F tests, AIC, t-tests or chi-squared tests that assess the significance of the increase in deviance that results when a given term is removed from the current model.</p>
<p>The <em>BREG</em>, also known as “all possible regressions”, as the name of the procedure indicates, fits a separate least squares regression for each possible combination of the <span class="math inline">\(p\)</span> predictors, i.e. explanatory variables. After fitting all of the models, BREG then displays the best fitted models with one explanatory variable, two explanatory variables, three explanatory variables, and so on. Usually, either adjusted R-squared or Mallows Cp is the criterion for picking the best fitting models for this process. The result is a display of the best fitted models of different sizes up to the full/maximal model and the final fitted model can be selected by comparing displayed models based on the criteria of parsimony.</p>
<div id="case-study" class="section level3">
<h3>Case Study</h3>
<p>We want to apply the subset selection approach to the <code>fat</code> data, available in the <code>library(faraway)</code>.
We wish to predict body fat (variable <code>brozek</code>) using all predictors except for <code>siri</code>, <code>density</code> and <code>free</code>.</p>
<p>As instructed, we will start off by creating a subset of all the variables from the <code>fat</code> data set except for <code>siri</code>, <code>density</code> and <code>free</code>. We will save this subset as a new data frame <code>bodyfat</code>. To confirm that the three suggested variables have been removed, we will check for the difference in the dimension of the two data frames and have a glimpse at teh data.</p>
<pre class="r"><code>#If you don&#39;t have the &quot;ISLR&quot; package installed yet, uncomment and run the line below
#install.packages(&quot;ISLR&quot;)
library(ISLR)
library(faraway)
library(tidyr)
suppressPackageStartupMessages(library(dplyr))
bodyfat &lt;- fat %&gt;%
select(-siri, -density, -free)
dim(fat) - dim(bodyfat)</code></pre>
<pre><code>## [1] 0 3</code></pre>
<pre class="r"><code>glimpse(bodyfat)</code></pre>
<pre><code>## Rows: 252
## Columns: 15
## $ brozek &lt;dbl&gt; 12.6, 6.9, 24.6, 10.9, 27.8, 20.6, 19.0, 12.8, 5.1, 12.0, 7.5…
## $ age &lt;int&gt; 23, 22, 22, 26, 24, 24, 26, 25, 25, 23, 26, 27, 32, 30, 35, 3…
## $ weight &lt;dbl&gt; 154.25, 173.25, 154.00, 184.75, 184.25, 210.25, 181.00, 176.0…
## $ height &lt;dbl&gt; 67.75, 72.25, 66.25, 72.25, 71.25, 74.75, 69.75, 72.50, 74.00…
## $ adipos &lt;dbl&gt; 23.7, 23.4, 24.7, 24.9, 25.6, 26.5, 26.2, 23.6, 24.6, 25.8, 2…
## $ neck &lt;dbl&gt; 36.2, 38.5, 34.0, 37.4, 34.4, 39.0, 36.4, 37.8, 38.1, 42.1, 3…
## $ chest &lt;dbl&gt; 93.1, 93.6, 95.8, 101.8, 97.3, 104.5, 105.1, 99.6, 100.9, 99.…
## $ abdom &lt;dbl&gt; 85.2, 83.0, 87.9, 86.4, 100.0, 94.4, 90.7, 88.5, 82.5, 88.6, …
## $ hip &lt;dbl&gt; 94.5, 98.7, 99.2, 101.2, 101.9, 107.8, 100.3, 97.1, 99.9, 104…
## $ thigh &lt;dbl&gt; 59.0, 58.7, 59.6, 60.1, 63.2, 66.0, 58.4, 60.0, 62.9, 63.1, 5…
## $ knee &lt;dbl&gt; 37.3, 37.3, 38.9, 37.3, 42.2, 42.0, 38.3, 39.4, 38.3, 41.7, 3…
## $ ankle &lt;dbl&gt; 21.9, 23.4, 24.0, 22.8, 24.0, 25.6, 22.9, 23.2, 23.8, 25.0, 2…
## $ biceps &lt;dbl&gt; 32.0, 30.5, 28.8, 32.4, 32.2, 35.7, 31.9, 30.5, 35.9, 35.6, 3…
## $ forearm &lt;dbl&gt; 27.4, 28.9, 25.2, 29.4, 27.7, 30.6, 27.8, 29.0, 31.1, 30.0, 2…
## $ wrist &lt;dbl&gt; 17.1, 18.2, 16.6, 18.2, 17.7, 18.8, 17.7, 18.8, 18.2, 19.2, 1…</code></pre>
<p>Using the <code>glimpse()</code> function allows us to notice that all the variables in our new <code>bodyfat</code> data set are the measured type.
#### Get to Know Your Data</p>
<p>Let us take a look at the data using the <code>DT</code> package, which provides an R interface to the JavaScript library <code>DataTables</code>. It will enable us to filter through the displayed data and quickly check for <code>NA</code>s to see if there are any missing values that should be of concern.</p>
<pre class="r"><code>DT::datatable(fat)</code></pre>
<div id="htmlwidget-1" style="width:100%;height:auto;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-1">{"x":{"filter":"none","data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110","111","112","113","114","115","116","117","118","119","120","121","122","123","124","125","126","127","128","129","130","131","132","133","134","135","136","137","138","139","140","141","142","143","144","145","146","147","148","149","150","151","152","153","154","155","156","157","158","159","160","161","162","163","164","165","166","167","168","169","170","171","172","173","174","175","176","177","178","179","180","181","182","183","184","185","186","187","188","189","190","191","192","193","194","195","196","197","198","199","200","201","202","203","204","205","206","207","208","209","210","211","212","213","214","215","216","217","218","219","220","221","222","223","224","225","226","227","228","229","230","231","232","233","234","235","236","237","238","239","240","241","242","243","244","245","246","247","248","249","250","251","252"],[12.6,6.9,24.6,10.9,27.8,20.6,19,12.8,5.1,12,7.5,8.5,20.5,20.8,21.7,20.5,28.1,22.4,16.1,16.5,19,15.3,15.7,17.6,14.2,4.6,8.5,22.4,4.7,9.4,12.3,6.5,13.4,20.9,31.1,38.2,23.6,27.5,33.8,31.3,33.1,31.7,30.4,30.8,8.4,14.1,11.2,6.4,13.4,5,10.7,7.4,8.7,7.1,4.9,22.2,20.1,27.1,30.4,24,25.4,28.8,29.6,25.1,31,28.9,21.1,14,7.1,13.2,23.7,9.4,9.1,13.7,12,18.3,9.2,21.7,21.1,18.6,30.2,26,18.2,26.2,26.1,25.8,15,22.6,8.8,14.3,20.2,18.1,9.2,24.2,9.6,17.3,10.1,11.1,17.7,21.7,20.8,20.1,19.8,21.9,24.7,17.8,19.1,18.2,17.2,21,19.5,27.1,21.6,20.9,25.9,16.7,19.8,14.1,25.1,17.9,27,24.6,14.8,16,14,17.4,26.4,17.4,20.4,15,18,22.2,23.1,25.3,23.8,26.3,21.4,28.4,21.8,20.1,24.3,18.1,22.7,9.9,10.8,14.4,19,28.6,6.1,24.5,9.9,19.1,10.6,16.5,20.5,17.2,30.1,10.5,12.8,22,9.9,14.8,13.3,15.2,26.5,19,21.4,20,34.7,16.5,4.1,1.9,20.2,16.8,24.6,10.4,13.4,28.8,22,16.8,25.8,0,11.9,12.4,17.4,9.2,23,20.1,20.2,23.8,11.8,36.5,16,24,22.3,24.8,21.5,17.6,7.3,22.6,12.5,21.7,27.7,6.8,33.4,16.6,31.7,31.5,10.1,11.3,7.8,26.4,19.3,18.5,19.3,45.1,13.8,8.2,23.9,15.1,12.7,25.3,11.9,6.1,11.3,12.8,14.9,24.5,15,16.9,11.1,16.1,15.5,25.9,25.5,18.4,24,26.4,12.7,28.8,17,33.6,29.3,31.4,28.1,15.3,29.1,11.5,32.3,28.3,25.3,30.7],[12.3,6.1,25.3,10.4,28.7,20.9,19.2,12.4,4.1,11.7,7.1,7.8,20.8,21.2,22.1,20.9,29,22.9,16,16.5,19.1,15.2,15.6,17.7,14,3.7,7.9,22.9,3.7,8.8,11.9,5.7,11.8,21.3,32.3,40.1,24.2,28.4,35.2,32.6,34.5,32.9,31.6,32,7.7,13.9,10.8,5.6,13.6,4,10.2,6.6,8,6.3,3.9,22.6,20.4,28,31.5,24.6,26.1,29.8,30.7,25.8,32.3,30,21.5,13.8,6.3,12.9,24.3,8.8,8.5,13.5,11.8,18.5,8.8,22.2,21.5,18.8,31.4,26.8,18.4,27,27,26.6,14.9,23.1,8.3,14.1,20.5,18.2,8.5,24.9,9,17.4,9.6,11.3,17.8,22.2,21.2,20.4,20.1,22.3,25.4,18,19.3,18.3,17.3,21.4,19.7,28,22.1,21.3,26.7,16.7,20.1,13.9,25.8,18.1,27.9,25.3,14.7,16,13.8,17.5,27.2,17.4,20.8,14.9,18.1,22.7,23.6,26.1,24.4,27.1,21.8,29.4,22.4,20.4,24.9,18.3,23.3,9.4,10.3,14.2,19.2,29.6,5.3,25.2,9.4,19.6,10.1,16.5,21,17.3,31.2,10,12.5,22.5,9.4,14.6,13,15.1,27.3,19.2,21.8,20.3,34.3,16.5,3,0.7,20.5,16.9,25.3,9.9,13.1,29.9,22.5,16.9,26.6,0,11.5,12.1,17.5,8.6,23.6,20.4,20.5,24.4,11.4,38.1,15.9,24.7,22.8,25.5,22,17.7,6.6,23.6,12.2,22.1,28.7,6,34.8,16.6,32.9,32.8,9.6,10.8,7.1,27.2,19.5,18.7,19.5,47.5,13.6,7.5,24.5,15,12.4,26,11.5,5.2,10.9,12.5,14.8,25.2,14.9,17,10.6,16.1,15.4,26.7,25.8,18.6,24.8,27.3,12.4,29.9,17,35,30.4,32.6,29,15.2,30.2,11,33.6,29.3,26,31.9],[1.0708,1.0853,1.0414,1.0751,1.034,1.0502,1.0549,1.0704,1.09,1.0722,1.083,1.0812,1.0513,1.0505,1.0484,1.0512,1.0333,1.0468,1.0622,1.061,1.0551,1.064,1.0631,1.0584,1.0668,1.0911,1.0811,1.0468,1.091,1.079,1.0716,1.0862,1.0719,1.0502,1.0263,1.0101,1.0438,1.0346,1.0202,1.0258,1.0217,1.025,1.0279,1.0269,1.0814,1.067,1.0742,1.0665,1.0678,1.0903,1.0756,1.084,1.0807,1.0848,1.0906,1.0473,1.0524,1.0356,1.028,1.043,1.0396,1.0317,1.0298,1.0403,1.0264,1.0313,1.0499,1.0673,1.0847,1.0693,1.0439,1.0788,1.0796,1.068,1.072,1.0666,1.079,1.0483,1.0498,1.056,1.0283,1.0382,1.0568,1.0377,1.0378,1.0386,1.0648,1.0462,1.08,1.0666,1.052,1.0573,1.0795,1.0424,1.0785,1.0991,1.077,1.073,1.0582,1.0484,1.0506,1.0524,1.053,1.048,1.0412,1.0578,1.0547,1.0569,1.0593,1.05,1.0538,1.0355,1.0486,1.0503,1.0384,1.0607,1.0529,1.0671,1.0404,1.0575,1.0358,1.0414,1.0652,1.0623,1.0674,1.0587,1.0373,1.059,1.0515,1.0648,1.0575,1.0472,1.0452,1.0398,1.0435,1.0374,1.0491,1.0325,1.0481,1.0522,1.0422,1.0571,1.0459,1.0775,1.0754,1.0664,1.055,1.0322,1.0873,1.0416,1.0776,1.0542,1.0758,1.061,1.051,1.0594,1.0287,1.0761,1.0704,1.0477,1.0775,1.0653,1.069,1.0644,1.037,1.0549,1.0492,1.0525,1.018,1.061,1.0926,1.0983,1.0521,1.0603,1.0414,1.0763,1.0689,1.0316,1.0477,1.0603,1.0387,1.1089,1.0725,1.0713,1.0587,1.0794,1.0453,1.0524,1.052,1.0434,1.0728,1.014,1.0624,1.0429,1.047,1.0411,1.0488,1.0583,1.0841,1.0462,1.0709,1.0484,1.034,1.0854,1.0209,1.061,1.025,1.0254,1.0771,1.0742,1.0829,1.0373,1.0543,1.0561,1.0543,0.995,1.0678,1.0819,1.0433,1.0646,1.0706,1.0399,1.0726,1.0874,1.074,1.0703,1.065,1.0418,1.0647,1.0601,1.0745,1.062,1.0636,1.0384,1.0403,1.0563,1.0424,1.0372,1.0705,1.0316,1.0599,1.0207,1.0304,1.0256,1.0334,1.0641,1.0308,1.0736,1.0236,1.0328,1.0399,1.0271],[23,22,22,26,24,24,26,25,25,23,26,27,32,30,35,35,34,32,28,33,28,28,31,32,28,27,34,31,27,29,32,29,27,41,41,49,40,50,46,50,45,44,48,41,39,43,40,39,45,47,47,40,51,49,42,54,58,62,54,61,62,56,54,61,57,55,54,55,54,55,62,55,56,55,61,61,57,69,81,66,67,64,64,70,72,67,72,64,46,48,46,44,47,46,47,53,38,50,46,47,49,48,41,49,43,43,43,52,43,40,43,43,47,42,48,40,48,51,40,44,52,44,40,47,50,46,42,43,40,42,49,40,47,50,41,44,39,43,40,49,40,40,52,23,23,24,24,25,25,26,26,26,27,27,27,28,28,28,30,31,31,33,33,34,34,35,35,35,35,35,35,35,35,36,36,37,37,37,38,39,39,40,40,40,40,40,41,41,41,41,41,42,42,42,42,42,42,42,42,43,43,43,43,44,44,44,44,47,47,47,49,49,49,50,50,51,51,51,52,53,54,54,54,55,55,55,55,55,56,56,57,57,58,58,60,62,62,63,64,65,65,65,66,67,67,68,69,70,72,72,72,74],[154.25,173.25,154,184.75,184.25,210.25,181,176,191,198.25,186.25,216,180.5,205.25,187.75,162.75,195.75,209.25,183.75,211.75,179,200.5,140.25,148.75,151.25,159.25,131.5,148,133.25,160.75,182,160.25,168,218.5,247.25,191.75,202.25,196.75,363.15,203,262.75,205,217,212,125.25,164.25,133.5,148.5,135.75,127.5,158.25,139.25,137.25,152.75,136.25,198,181.5,201.25,202.5,179.75,216,178.75,193.25,178,205.5,183.5,151.5,154.75,155.25,156.75,167.5,146.75,160.75,125,143,148.25,162.5,177.75,161.25,171.25,163.75,150.25,190.25,170.75,168,167,157.75,160,176.75,176,177,179.75,165.25,192.5,184.25,224.5,188.75,162.5,156.5,197,198.5,173.75,172.75,196.75,177,165.5,200.25,203.25,194,168.5,170.75,183.25,178.25,163,175.25,158,177.25,179,191,187.5,206.5,185.25,160.25,151.5,161,167,177.5,152.25,192.25,165.25,171.75,171.25,197,157,168.25,186,166.75,187.75,168.25,212.75,176.75,173.25,167,159.75,188.15,156,208.5,206.5,143.75,223,152.25,241.75,146,156.75,200.25,171.5,205.75,182.5,136.5,177.25,151.25,196,184.25,140,218.75,217,166.25,224.75,228.25,172.75,152.25,125.75,177.25,176.25,226.75,145.25,151,241.25,187.25,234.75,219.25,118.5,145.75,159.25,170.5,167.5,232.75,210.5,202.25,185,153,244.25,193.5,224.75,162.75,180,156.25,168,167.25,170.75,178.25,150,200.5,184,223,208.75,166,195,160.5,159.75,140.5,216.25,168.25,194.75,172.75,219,149.25,154.5,199.25,154.5,153.25,230,161.75,142.25,179.75,126.5,169.5,198.5,174.5,167.75,147.75,182.25,175.5,161.75,157.75,168.75,191.5,219.15,155.25,189.75,127.5,224.5,234.25,227.75,199.5,155.5,215.5,134.25,201,186.75,190.75,207.5],[67.75,72.25,66.25,72.25,71.25,74.75,69.75,72.5,74,73.5,74.5,76,69.5,71.25,69.5,66,71,71,67.75,73.5,68,69.75,68.25,70,67.75,71.5,67.5,67.5,64.75,69,73.75,71.25,71.25,71,73.5,65,70,68.25,72.25,67,68.75,29.5,70,71.5,68,73.25,67.5,71.25,68.5,66.75,72.25,69,67.75,73.5,67.5,72,68,69.5,70.75,65.75,73.25,68.5,70.25,67,70,67.5,70.75,71.5,69.25,71.5,71.5,68.75,73.75,64,65.75,67.5,69.5,68.5,70.25,69.25,67.75,67.25,72.75,70,69.25,67.5,67.25,65.75,72.5,73,70,69.5,70.5,71.75,74.5,77.75,73.25,66.5,68.25,72,73.5,72,71.25,73.75,69.25,68.5,73.5,74.25,75.5,69.25,68.5,70,70,70.25,71.75,69.25,72.75,72,74,72.25,74.5,71.5,68.75,66.75,66.5,67,68.75,67.75,73.25,69.75,71.5,70.5,73.25,66.75,69.5,69.75,70.75,74,71.25,75,71,69.5,67.75,72.25,77.5,70.75,72.75,69.75,72.5,70.25,69,74.5,72.25,67.25,73.5,75.25,69,72.25,68.75,71.5,72.25,73,68.75,70.5,72,73.75,68,72.25,69.5,69.5,67.75,65.5,71,71.5,71.75,69.25,67,71.5,69.25,74.5,74.25,68,67.25,69.75,74.25,71.5,74.25,72,72.5,68.25,69.25,76,70.5,74.75,72.75,68.25,69,71.5,72.75,67.5,70.25,69.25,71.5,74,69.75,73,65.5,72.5,70.25,70.75,68,74.5,71.75,70.75,73,64,69.75,70,71.75,69.25,70.5,72.25,67.5,67.25,68.75,66.75,68.25,74.25,69.5,68.5,65.75,71.75,71.5,67.25,67.5,67.5,72.25,69.5,69.5,65.75,65.75,68.25,72,72.75,68.5,69.25,70.5,67,69.75,66,70.5,70],[23.7,23.4,24.7,24.9,25.6,26.5,26.2,23.6,24.6,25.8,23.6,26.3,26.3,28.5,27.4,26.3,27.3,29.2,28.2,27.6,27.3,29.1,21.2,21.4,23.2,21.9,20.3,22.9,22.4,23.8,23.6,22.2,23.3,30.5,32.2,32,29.1,29.7,48.9,31.8,39.1,29.9,31.2,29.2,19.1,21.3,20.6,20.6,20.4,20.2,21.3,20.6,21.1,19.9,21.1,26.9,27.6,29.3,28.4,29.2,28.2,26.8,27.6,27.9,29.5,28.3,21.3,21.3,22.8,21.6,23.1,21.9,20.8,21.5,23.3,22.9,23.7,26.7,23,25.1,25.1,23.4,25.3,24.5,24.7,26,24.6,26,23.7,23.3,25.4,26.2,23.4,26.3,23.4,26.1,24.8,25.9,23.7,26.7,25.9,23.6,24,25.5,26,24.8,26,26,24,24.7,25.6,26.3,25.6,23.3,24,23.4,23.6,24.3,24.6,25.3,26.2,25.5,23.9,23.9,25.6,26.2,26.4,23.4,25.2,23.9,23.7,24.3,25.8,24.8,24.5,26.8,23.5,24.1,23.3,26.6,24.6,25.3,25.6,21.6,22.1,21.9,27.7,29.8,19.3,31.8,22.5,30.7,19.7,24.4,26.1,21.6,30.4,24.6,20.3,24.4,20.4,25.9,24.4,19.8,29.7,28.1,25.3,30.3,33.3,25.2,23.4,20.6,24.8,24.3,31,21.3,23.7,33.2,27.5,29.8,28,18.1,22.7,23,21.8,23.1,29.7,28.6,27,28,22.5,29.8,27.4,28.3,21.6,27.2,23.1,23.1,22.3,26.4,25.4,22,27.6,23.7,32.3,27.6,27.2,26.1,22.9,22.5,21.4,27.4,23,27.4,22.8,37.6,21.6,22.2,27.2,22.7,24.5,31,25,22.2,26.8,20,25.6,25.3,25.4,25.2,24.1,24.9,24.2,25.2,24.1,26.1,25.8,31.9,22.6,30.9,20.8,33.9,31.8,30.3,29.9,22.8,30.5,21.1,29.1,30.2,27,29.8],[134.9,161.3,116,164.7,133.1,167,146.6,153.6,181.3,174.4,172.3,197.7,143.5,162.5,147,129.3,140.8,162.5,154.3,176.8,145.1,169.8,118.2,122.6,129.8,151.9,120.3,114.9,127,145.7,159.7,149.8,142.5,172.7,170.4,118.4,154.5,142.6,240.5,139.4,175.8,140.1,151.1,146.7,114.7,141.1,118.5,139,117.6,121.2,141.4,129,125.3,142,129.6,154.1,145.1,146.7,141,136.7,161.2,127.4,136.1,133.3,141.7,130.4,119.6,133.1,144.2,136.1,127.8,132.9,146.1,107.9,125.9,121.1,147.5,139.1,127.2,139.5,114.3,111.2,155.6,126,124.1,123.9,134.1,123.8,161.1,150.9,141.3,147.3,150.1,145.9,166.6,185.7,169.6,143.5,128.8,154.2,157.2,138.9,138.6,153.7,133.2,136,162,166.3,160.6,133.1,137.5,133.5,139.7,128.9,129.9,131.7,142.1,153.8,143.1,153.8,150.7,139.6,136.5,127.3,138.5,137.9,130.7,125.8,153,140.5,140.9,133.3,151.2,117.2,128.3,137.1,131,134.4,131.6,169.9,133.8,141.8,129,143.9,168.4,133.6,168.9,147.5,135,168.3,137.2,195.1,130.5,130.9,159.3,142,143.9,163.4,119.1,138.3,136.2,167,159.8,118.8,160.8,175.8,130.7,179.7,149.3,144.2,146.1,123.4,141.7,146.6,170.9,130.2,130.8,171.7,146.1,195.3,162.7,118.5,128.4,139.5,140.8,152.1,179.2,168.3,161.4,141,135,155.2,162.6,170.8,126.5,135.4,122.6,138.4,155.1,132.1,155.9,117.5,144.9,171.4,148.5,174.2,113.5,133.6,144.3,141.8,129.5,159.3,135.9,158.7,139.4,120.2,128.7,141.9,151.7,131.2,151.3,171.9,142.6,133.6,159.5,110.3,144.2,149.9,148.3,139.4,131.4,152.9,148.4,119.9,117.5,137.6,145.2,161.2,135.5,135.1,105.9,149.2,165.6,156.3,143.6,131.8,152.7,118.9,136.1,133.9,142.6,143.7],[36.2,38.5,34,37.4,34.4,39,36.4,37.8,38.1,42.1,38.5,39.4,38.4,39.4,40.5,36.4,38.9,42.1,38,40,39.1,41.3,33.9,35.5,34.5,35.7,36.2,38.8,36.4,36.7,38.7,37.3,38.1,39.8,42.1,38.4,38.5,42.1,51.2,40.2,43.2,36.6,37.3,41.5,31.5,35.7,33.6,34.6,32.8,34,34.9,34.3,36.5,35.1,37.8,39.9,39.1,40.5,40.5,38.4,41.4,35.6,38,37.4,40.1,40.9,35.6,36.9,37.5,36.3,35.5,38.7,36.4,33.2,36.5,36,38.7,38.7,37.8,37.4,38.4,38.1,39.3,38.7,38.5,36.5,37.7,36.5,38,36.7,37.2,39.2,37.5,38,37.3,41.1,37.5,38.7,35.9,40,40.1,37,36.3,40.7,39.6,31.1,38.6,42,38.5,34.2,37.2,37.1,40.2,35.3,38,36.3,36.8,41,38.3,38,40.8,39.5,36.9,36.9,37.7,36.6,38.9,37.5,39.8,38.3,35.5,36.3,37.8,37.8,36.5,37.8,37,37.7,34.3,40.8,37.4,36.5,37.5,35.5,38,35.7,39.2,40.9,35.2,40.6,35.4,41.8,34.1,37.9,38.2,35.6,38.5,37,35.9,36.2,35,38.5,40.7,36,39.5,40.5,38.5,43.9,40.4,37.6,37,34,38.4,38.7,41.5,36,35.3,42.1,38,42.8,40,33.8,35.5,35.3,37.7,39.4,41.9,38.5,40.8,38,36.4,41.8,40.7,38.5,35.4,38.5,35.5,36.5,37.6,37.4,37.8,35.2,37.9,37.9,40.9,41.9,39.1,40.2,36,34.5,35.8,40.2,38.3,39,37.4,41.2,34.8,36.9,39.4,37.6,38.5,42.5,37.4,35.2,41.1,33.4,37.2,38.3,38.1,37.4,35.2,39.4,38,35.1,40.4,38.3,40.6,40.2,37.9,40.8,34.7,38.8,41.4,41.3,40.7,36.3,40.8,34.9,40.9,38.9,38.9,40.8],[93.1,93.6,95.8,101.8,97.3,104.5,105.1,99.6,100.9,99.6,101.5,103.6,102,104.1,101.3,99.1,101.9,107.6,106.8,106.2,103.3,111.4,86,86.7,90.2,89.6,88.6,97.4,93.5,97.4,100.5,93.5,93,111.7,117,118.5,106.5,105.6,136.2,114.8,128.3,106,113.3,106.6,85.1,96.6,88.2,89.8,92.3,83.4,90.2,89.2,89.7,93.3,87.6,107.6,100,111.5,115.4,104.8,112.3,102.9,107.6,105.3,105.3,103,90,95.4,89.3,94.4,97.6,88.5,93.6,87.7,93.4,91.6,91.6,102,96.4,102.7,97.7,97.1,103.1,101.8,101.4,98.9,97.5,104.3,97.3,96.7,99.7,101.9,97.2,106.6,99.6,113.2,99.1,99.4,95.1,107.5,106.5,99.1,96.7,103.5,104,93.1,105.2,110,110.1,97.8,96.3,108,99.7,93.5,100.7,97,96,99.2,95.4,101.8,104.3,99.2,99.3,94,98.9,101,98.7,95.9,103.9,96.2,97.8,94.6,103.6,100.4,98.4,104.6,92.9,97.8,98.3,104.7,98.6,99.5,102.7,92.1,96.6,92.7,102,110.9,92.3,114.1,92.9,108.3,88.5,94,101.1,92.1,105.6,98.5,88.7,101.1,94,103.8,98.9,89.2,111.4,107.5,99.1,108.2,114.9,99.1,92.2,90.8,100.5,98.2,115.3,96.8,92.6,119.2,102.7,109.5,108.5,79.3,95.5,92.3,98.9,89.5,117.5,107.4,109.2,103.4,91.4,115.2,104.9,106.7,92.2,101.6,97.8,92,94,103.7,102.7,91.1,107.2,100.8,121.6,105.6,100.6,102.7,99.8,92.9,91.2,115.6,98.3,103.7,98.7,119.8,92.8,93.3,106.8,93.9,99,119.9,94.2,92.7,106.9,88.8,101.7,105.3,104,98.6,99.6,103.4,100.2,94.9,97.2,104.7,104,117.6,95.8,106.4,93,119.6,119.7,115.8,118.3,97.4,113.7,89.2,108.5,111.1,108.3,112.4],[85.2,83,87.9,86.4,100,94.4,90.7,88.5,82.5,88.6,83.6,90.9,91.6,101.8,96.4,92.8,96.4,97.5,89.6,100.5,95.9,98.8,76.4,80,76.3,79.7,74.6,88.7,73.9,83.5,88.7,84.5,79.1,100.5,115.6,113.1,100.9,98.8,148.1,108.1,126.2,104.3,111.2,104.3,76,81.5,73.7,79.5,83.4,70.4,86.7,77.9,82,79.6,77.6,100,99.8,104.2,105.3,98.3,104.8,94.7,102.4,99.7,105.5,100.3,83.9,86.6,78.4,84.6,91.5,82.8,82.9,76,83.3,81.8,78.8,95,95.4,98.6,95.8,89,97.8,94.9,99.8,89.7,88.1,90.9,86,86.5,95.6,93.2,83.1,97.5,88.8,99.2,91.6,86.7,88.2,94,95,92,89.2,95.5,98.6,87.3,102.8,101.6,88.7,92.3,90.6,105,95,89.6,92.4,86.6,90,90,92.4,87.5,99.2,98.1,83.3,86.1,84.1,89.9,92.1,78,93.5,87,90.1,90.3,99.8,89.4,87.2,101.1,86.1,98.6,88.5,106.6,93.1,93,91,77.1,85.3,81.9,99.1,100.5,76.5,106.8,77.6,102.9,72.8,88.2,100.1,83.5,105,90.8,76.6,92.4,81.2,95.6,92.1,83.4,106,95.1,90.4,100.4,115.9,90.8,81.9,75,90.3,90.3,108.8,79.4,83.2,110.3,92.7,104.5,104.6,69.4,83.6,86.8,90.4,83.7,109.3,98.9,98,101.2,80.6,113.7,94.1,105.7,85.6,96.6,86,89.7,78,89.7,89.2,85.7,103.1,89.1,113.9,96.3,93.9,101.3,83.9,84.4,79.4,104,89.7,97.6,87.6,122.1,81.1,81.5,100,88.7,91.8,110.4,87.6,82.8,95.3,78.2,91.1,96.7,89.4,93,86.4,96.7,88.1,94.9,93.3,95.6,98.2,113.8,82.8,100.5,79.7,118,109,113.4,106.1,84.3,107.6,83.6,105,111.5,101.3,108.5],[94.5,98.7,99.2,101.2,101.9,107.8,100.3,97.1,99.9,104.1,98.2,107.7,103.9,108.6,100.1,99.2,105.2,107,102.4,109,104.9,104.8,94.6,93.4,95.8,96.5,85.3,94.7,88.5,98.7,99.8,100.6,94.5,108.3,116.1,113.8,106.2,104.8,147.7,102.5,125.6,115.5,114.1,106,88.2,97.2,88.5,92.7,90.4,87.2,98.3,91,89.1,91.6,88.6,99.6,102.5,105.8,97,99.6,103.1,100.8,99.4,99.7,108.3,104.2,93.9,91.8,96.1,94.3,98.5,95.5,96.3,88.6,93,94.8,94.3,98.3,99.3,100.2,97.1,96.9,99.6,95,96.2,96.2,96.9,93.8,99.3,98.3,102.2,100.6,95.4,100.6,101.4,107.5,102.4,96.2,92.8,103.7,101.7,98.3,98.3,101.6,99.5,96.6,103.6,100.7,102.1,100.6,99.3,103,98.6,99.8,97.5,92.6,99.7,96.4,104.3,101,104.1,101.4,97.5,95.2,94,100,98.5,93.2,99.5,97.8,95.8,99.1,103.2,92.3,98.4,102.1,95.6,100.6,98.3,107.7,101.6,99.3,98.9,93.9,102.5,95.3,110.1,106.2,92.1,113.9,93.5,114.4,91.1,95.2,105,98.3,106.4,102.5,89.8,99.3,91.5,105.1,103.5,89.6,108.8,104.5,95.6,106.8,111.9,98.1,92.8,89.2,98.7,99.9,114.4,89.2,96.4,113.9,101.9,109.9,109.8,85,91.6,96.1,95.5,98.1,108.8,104.1,101.8,103.1,92.3,112.4,102.7,111.8,96.5,100.6,96.2,101,99,94.2,99.2,96.9,105.5,102.6,107.1,102,100.1,101.7,91.8,94,89,109,99.1,104.2,96.1,112.8,96.3,94.4,105,94.5,96.2,105.5,95.6,91.9,98.2,87.5,97.1,106.6,98.4,97,90.1,100.7,97.8,100.2,94,93.7,101.1,111.8,94.5,100.5,87.6,114.3,109.1,109.8,101.6,94.4,110,88.8,104.5,101.7,97.8,107.1],[59,58.7,59.6,60.1,63.2,66,58.4,60,62.9,63.1,59.7,66.2,63.4,66,69,63.1,64.8,66.9,64.2,65.8,63.5,63.4,57.4,54.9,58.4,55,51.7,57.5,50.1,58.9,57.5,58.5,57.3,67.1,71.2,61.9,63.5,66,87.3,61.3,72.5,70.6,67.7,65,50,58.4,53.3,52.7,52,50.6,52.6,51.4,49.3,52.6,51.9,57.2,62.1,61.8,59.1,60.6,61.6,60.9,61,60.8,65,64.8,55,54.3,56,51.2,56.6,58.9,52.9,50.9,55.5,54.5,56.7,55,53.5,56.5,54.8,54.8,58.9,56,56.3,54.7,57.2,57.8,61,60.4,58.3,58.9,56.9,58.9,57.4,61.7,60.6,62.1,54.7,62.7,59,59.3,60,59.1,59.5,54.7,61.2,55.8,57.5,57.5,61.9,63.7,62.3,61.5,59.3,55.9,58.8,56.8,64.6,58.5,58.5,57.1,60.5,58.1,58.5,60.7,60.7,53.5,61.7,57.4,57,60.3,61.2,56.1,56,58.9,58.8,63.6,58.1,66.5,59.1,60.4,57.1,56.1,59.1,56.4,71.2,68.4,51.9,67.6,56.9,72.9,53.6,56.8,62.1,57.3,68.6,60.8,50.1,59.4,52.5,61.4,64,52.4,63.8,64.8,55.5,63.3,74.4,60.1,54.7,50,57.8,59.2,69.2,50.3,60,69.8,64.7,69.5,68.1,47.2,54.1,58,55.4,57.3,67.7,63.5,62.8,61.5,54.3,68.5,60.6,65.3,60.2,61.1,57.7,62.3,57.5,58.5,60.2,55.5,68.8,60.6,63.5,63.3,58.9,60.7,53,56,51.1,63.7,56.3,60,57.1,62.5,53.8,54.7,63.9,53.7,57.7,64.2,59.7,54.4,57.4,50.8,56.6,64,58.4,55.4,53,59.3,57.1,56.8,54.3,54.4,59.3,63.4,61.2,59.2,50.7,61.3,63.7,65.6,58.2,54.3,63.3,49.6,59.6,60.3,56,59.3],[37.3,37.3,38.9,37.3,42.2,42,38.3,39.4,38.3,41.7,39.7,39.2,38.3,41.5,39,38.7,40.8,40,38.7,40.6,38,40.6,35.3,36.2,35.5,36.7,34.7,36,34.5,35.3,38.7,38.8,36.2,44.2,43.3,38.3,39.9,41.5,49.1,41.1,39.6,42.5,40.9,40.2,34.7,38.2,34.5,37.5,35.8,34.4,37.2,34.9,33.7,37.6,34.9,38,39.6,39.8,38,37.7,40.9,38,39.4,40.1,41.2,40.2,36.1,35.4,37.4,37.4,38.6,37.6,37.5,35.4,35.2,37,39.7,38.3,37.5,39.3,38.2,38,39,36.5,36.6,37.8,37.7,39.5,38.4,39.9,38.2,39.7,38.3,40.5,39.6,42.3,39.4,39.3,37.3,39,39.4,38.4,38.4,39.8,36.1,39,39.3,38.7,40,36.8,38,40,38.1,37.8,38.1,36.3,38.4,38.8,41.1,39.2,39.3,40.5,38.7,36.5,36.6,36,36.8,35.8,39,36.9,38.7,38.5,38.1,35.6,36.9,37.9,36.1,39.2,38.4,42.5,39.6,38.2,36.7,36.1,37.6,36.5,43.5,40.8,35.7,42.7,35.9,43.5,36.8,37.4,40,37.8,40,38.5,34.8,39,36.6,40.6,37.3,35.6,42,41.3,34.2,41.7,40.6,39.1,36.2,34.8,37.3,37.7,42.4,34.8,38.1,42.6,39.5,43.1,42.8,33.5,36.2,39.4,38.9,39.7,41.3,39.8,41.3,40.4,36.3,45,38.6,43.3,38.9,38.4,38.6,38,40,39,39.2,35.7,38.3,39,40.3,39.8,37.6,39.4,36.2,38.2,35,40.3,38.8,40.9,38.1,36.9,36.5,39,39.2,36.2,38.1,42.7,40.2,35.2,37.1,33,38.5,42.6,37.4,38.8,35,38.6,38.9,35.9,35.7,37.1,40.3,41.1,39.1,38.1,33.4,42.1,42.4,46,38.8,37.5,44,34.8,40.8,37.3,41.6,42.2],[21.9,23.4,24,22.8,24,25.6,22.9,23.2,23.8,25,25.2,25.9,21.5,23.7,23.1,21.7,23.1,24.4,22.9,24,22.1,24.6,22.2,22.1,22.9,22.5,21.4,21,21.3,22.6,33.9,21.5,24.5,25.2,26.3,21.9,22.6,24.7,29.6,24.7,26.6,23.7,25,23,21,23.4,22.5,21.9,20.6,21.9,22.4,21,21.4,22.6,22.5,22,22.5,22.7,22.5,22.9,23.1,22.1,23.6,22.7,24.7,22.7,21.7,21.5,22.4,21.6,22.4,21.6,23.1,19.1,20.9,21.4,24.2,21.8,21.5,22.7,23.7,22,23,24.1,22,33.7,21.8,23.3,23.8,24.4,22.5,23.1,22.1,24.5,24.6,23.2,22.9,23.3,21.9,22.3,22.3,22.4,23.2,25.4,22,24.8,23.5,23.4,24.8,22.8,22.3,23.6,23.9,21.9,21.8,22.1,22.8,23.3,24.8,24.5,24.6,23.2,22.6,22.1,23.5,21.9,22.2,20.8,21.8,22.2,23.2,23,22.6,20.5,23,22.7,22.4,23.8,22.5,24.5,21.6,22,22.3,22.7,23.2,22,25.2,24.6,22,24.7,20.4,25.1,23.8,22.8,24.9,21.7,25.2,25,21.8,24.6,21,25,23.5,20.4,23.4,25.6,21.9,24.6,24,23.4,22.1,22,22.4,21.5,24,22.2,22,24.8,24.7,25.8,24.1,20.2,21.8,22.7,22.4,22.6,24.7,23.5,24.8,22.9,21.8,25.5,24.7,26,22.4,24.1,24,22.3,22.5,24.1,23.8,22,23.7,24,21.8,24.1,21.4,23.3,22.5,22.6,21.7,23.2,23,25.5,21.8,23.6,21.5,22.6,22.9,22,23.9,27,23.4,22.5,21.8,19.7,22.6,23.4,22.5,23.2,21.3,22.8,23.6,21,21,22.7,23,22.3,22.3,24,20.1,23.4,24.6,25.4,24.1,22.6,22.6,21.5,23.2,21.5,22.7,24.6],[32,30.5,28.8,32.4,32.2,35.7,31.9,30.5,35.9,35.6,32.8,37.2,32.5,36.9,36.1,31.1,36.2,38.2,37.2,37.1,32.5,33,27.9,29.8,31.1,29.9,28.7,29.2,30.5,30.1,32.5,30.1,29,37.5,37.3,32,35.1,33.2,45,34.1,36.4,33.6,36.7,35.8,26.1,29.7,27.9,28.8,28.8,26.8,26,26.7,29.6,38.5,27.7,35.9,33.1,37.7,31.6,34.5,36.2,32.5,32.7,33.6,35.3,34.8,29.6,32.8,32.6,27.3,31.5,30.3,29.7,29.3,29.4,29.3,30.2,30.8,31.4,30.3,29.4,29.9,34.3,31.2,29.7,32.4,32.6,29.2,30.2,28.8,29.1,31.4,30.1,33.3,30.3,32.9,31.6,30.6,31.6,35.3,32.2,27.9,31,31,30.1,31,30.5,35.1,35.1,32.1,33.3,33.5,35.3,30.7,31.8,29.8,29.9,33.4,33.6,32.1,33.9,33,34.4,30.6,34.4,35.6,33.8,33.9,33.3,31.6,27.5,31.2,33.5,33.6,34,30.9,32.7,34.3,31.7,35.5,30.8,32,31.6,30.5,31.8,33.5,36.1,33.3,25.8,36,31.6,38.5,27.8,30.6,33.7,32.2,35.2,31.6,27,30.1,27,31.3,33.5,28.3,34,36.4,30.2,37.2,36.1,32.5,30.4,24.8,31,32.4,35.4,31,31.5,34.4,34.8,39.1,35.6,27.7,31.4,30,30.5,32.9,37.2,36.4,36.6,33.4,29.6,37.1,34,33.7,31.7,32.9,31.2,30.8,30.6,33.8,31.7,29.4,32.1,32.9,34.8,37.3,33.1,36.7,31.4,29,30.9,36.8,29.5,32.7,28.6,34.7,31.3,27.5,35.7,28.5,31.4,38.4,27.9,29.4,34.1,25.3,33.4,33.2,34.6,32.4,31.7,31.8,30.9,27.8,31.3,30.3,32.6,35.1,29.8,35.9,28.5,34.9,35.6,35.3,32.1,29.2,37.5,25.6,35.2,31.3,30.5,33.7],[27.4,28.9,25.2,29.4,27.7,30.6,27.8,29,31.1,30,29.4,30.2,28.6,31.6,30.5,26.4,30.8,31.6,30.5,30.1,30.3,32.8,25.9,26.7,28,28.2,27,26.6,27.9,26.7,27.7,26.4,30,31.5,31.7,29.8,30.6,30.5,29,31,32.7,28.7,29.8,31.5,23.1,27.4,26.2,26.8,25.5,25.8,25.8,26.1,26,27.4,27.5,30.2,28.3,30.9,28.8,29.6,31.8,29.8,29.9,29,31.1,30.1,27.4,27.4,28.1,27.1,27.3,27.3,27.3,25.7,27,27,29.2,25.7,26.8,28.7,27.2,25.2,29.6,27.3,26.3,27.7,28,28.4,29.3,29.6,27.7,28.4,28.2,29.6,27.9,30.8,30.1,27.8,27.5,30.9,31,26.2,29.2,30.3,27.2,29.4,28.5,29.6,30.7,26,28.2,27.8,31.1,27.6,27.3,26.3,28,29.8,29.5,28.6,31.2,29.6,28,27.5,29.2,30.2,30.3,28.2,29.6,27.8,26.5,28.4,28.6,29.3,29.8,28.8,28.3,28.4,27.4,29.8,27.9,28.5,27.5,27.2,29.7,28.3,30.3,29.7,25.2,30.4,29,33.8,26.3,28.3,29.2,27.7,30.7,28,34.9,28.2,26.3,29.2,30.6,26.2,31.2,33.7,28.7,33.1,31.8,29.8,27.4,25.9,28.7,28.4,21,26.9,26.6,29.5,30.3,32.5,29,24.6,28.3,26.4,28.9,29.3,31.8,30.4,32.4,29.2,27.3,31.2,30.1,29.9,27.1,29.8,27.3,27.8,30,28.8,28.4,26.6,28.9,29.2,30.7,23.1,29.5,31.6,27.5,26.2,28.8,31,27.9,30,26.7,29.1,26.3,25.9,30.4,25.7,29.9,32,27,26.8,31.1,22,29.3,30,30.1,29.7,27.3,29.1,29.6,26.1,28.7,26.3,28.5,29.6,28.9,30.5,24.8,30.1,30.7,29.8,29.3,27.3,32.6,25.7,28.6,27.2,29.4,30],[17.1,18.2,16.6,18.2,17.7,18.8,17.7,18.8,18.2,19.2,18.5,19,17.7,18.8,18.2,16.9,17.3,19.3,18.5,18.2,18.4,19.9,16.7,17.1,17.6,17.7,16.5,17,17.2,17.6,18.4,17.9,18.8,18.7,19.7,17,19,19.4,21.4,18.3,21.4,17.4,18.4,18.8,16.1,18.3,17.3,17.9,16.3,16.8,17.3,17.2,16.9,18.5,18.5,18.9,18.5,19.2,18.2,18.5,20.2,18.3,19.1,18.8,18.4,18.7,17.4,18.7,18.1,17.3,18.6,18.3,18.2,16.9,16.8,18.3,18.1,18.8,18.3,19,19,17.7,19,19.2,18,18.2,18.8,18.1,18.8,18.7,17.7,18.8,18.4,19.1,17.8,20.4,18.5,18.2,18.2,18.3,18.6,17,18.4,19.7,17.7,18.8,18.1,19.1,19.2,17.3,18.1,17.4,19.8,17.4,17.5,17.3,18.1,19.5,18.5,18,19.5,18.4,17.6,17.6,18,17.6,17.2,17.4,18.1,17.7,17.6,17.1,17.9,17.3,18.1,17.6,17.1,17.7,17.6,18.7,16.6,17.8,17.9,18.2,18.3,17.3,18.7,18.4,16.9,18.4,17.8,19.6,17.4,17.9,19.4,17.7,19.1,18.6,16.9,18.2,16.5,19.1,19.7,16.5,18.5,19.4,17.7,19.8,18.8,17.4,17.7,16.9,17.7,17.8,20.1,16.9,16.7,18.4,18.1,19.9,19,16.5,17.2,17.4,17.7,18.2,20,19.1,18.8,18.5,17.9,19.9,18.7,18.5,17.1,18.8,17.4,16.9,18.5,18.8,18.6,17.4,18.7,18.4,17.4,19.4,17.3,18.4,17.7,17.6,17.4,18.9,18.6,19,18,18.4,17.8,18.6,19.2,17.1,18.9,19.6,17.8,17,19.2,15.8,18.8,18.4,18.8,19,16.9,19,18,17.6,18.3,18.3,19,18.5,18.3,19.1,16.5,19.4,19.5,19.5,18.5,18.5,18.8,18.5,20.1,18,19.8,20.9]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>brozek<\/th>\n <th>siri<\/th>\n <th>density<\/th>\n <th>age<\/th>\n <th>weight<\/th>\n <th>height<\/th>\n <th>adipos<\/th>\n <th>free<\/th>\n <th>neck<\/th>\n <th>chest<\/th>\n <th>abdom<\/th>\n <th>hip<\/th>\n <th>thigh<\/th>\n <th>knee<\/th>\n <th>ankle<\/th>\n <th>biceps<\/th>\n <th>forearm<\/th>\n <th>wrist<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]},{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
<p>Alternatively, we can use the <code>is.na()</code> function to check if there are any missing observations in particular with respect to the response variable <code>bronzek</code>. For a given input vector the <code>is.na()</code> function returns a vector of <code>TRUE</code> and <code>FALSE</code> value, with a value of <code>TRUE</code> for any elements that are missing, and a <code>FALSE</code> value for non-missing elements. By using the <code>sum()</code> function we can then count all of the missing elements, which can be easily removed from data by using either the <code>na.omit()</code> function or the <code>drop_na</code> function.</p>
<pre class="r"><code>bodyfat %&gt;%
select(brozek) %&gt;%
is.na() %&gt;%
sum() </code></pre>
<pre><code>## [1] 0</code></pre>
</div>
<div id="the-correlation-matrix-and-its-visualisation" class="section level3">
<h3>The correlation matrix and its visualisation</h3>
<p>Previously (see the section on <a href="https://dataliteracy.rbind.io/module4/mlregression/">Multiple Regression</a>), we learnt how to examine multivariate data by creating a scatter plot matrix to obtain an in-depth vision into its behaviour. We used the <code>GGally::ggpairs()</code> function that produces a pairwise comparison of multivariate data for both data types: measured and attribute. Creating such a matrix for a data set with a considerable number of variables could be a taxing task with a resulting display that may not be easy to make sense of.</p>
<p>Fabulous <a href="https://www.vumc.org/biostatistics/person/frank-e-harrell-jr-phd">Frank Harrell</a> (check his blog <a href="https://www.fharrell.com/">Statistical Thinking</a> if you haven’t stumbled upon it yet!) has created the <a href="https://cran.r-project.org/web/packages/Hmisc/index.html"><code>Hmisc</code></a> package with many useful functions for data analysis and high-level graphics. It contains the <code>rcorr</code> function for the computation of Pearson or Spearman correlation matrix with pairwise deletion of missing data. It generates one table of correlation coefficients, i.e. the correlation matrix and another table of the <span class="math inline">\(p\)</span>-values.</p>
<pre class="r"><code>bodyfat.rcorr = Hmisc::rcorr(as.matrix(bodyfat))
bodyfat.coeff = bodyfat.rcorr$r
bodyfat.p = bodyfat.rcorr$P
bodyfat.coeff</code></pre>
<pre><code>## brozek age weight height adipos neck
## brozek 1.00000000 0.28917352 0.61315611 -0.08910641 0.72799418 0.4914889
## age 0.28917352 1.00000000 -0.01274609 -0.17164514 0.11885126 0.1135052
## weight 0.61315611 -0.01274609 1.00000000 0.30827854 0.88735216 0.8307162
## height -0.08910641 -0.17164514 0.30827854 1.00000000 -0.02489094 0.2537099
## adipos 0.72799418 0.11885126 0.88735216 -0.02489094 1.00000000 0.7778569
## neck 0.49148893 0.11350519 0.83071622 0.25370988 0.77785691 1.0000000
## chest 0.70288516 0.17644968 0.89419052 0.13489181 0.91179865 0.7848350
## abdom 0.81370622 0.23040942 0.88799494 0.08781291 0.92388010 0.7540774
## hip 0.62569993 -0.05033212 0.94088412 0.17039426 0.88326922 0.7349579
## thigh 0.56128438 -0.20009576 0.86869354 0.14843561 0.81270609 0.6956973
## knee 0.50778587 0.01751569 0.85316739 0.28605321 0.71365983 0.6724050
## ankle 0.26678256 -0.10505810 0.61368542 0.26474369 0.50031664 0.4778924
## biceps 0.49303089 -0.04116212 0.80041593 0.20781557 0.74638418 0.7311459
## forearm 0.36327744 -0.08505555 0.63030143 0.22864922 0.55859425 0.6236603
## wrist 0.34757276 0.21353062 0.72977489 0.32206533 0.62590659 0.7448264
## chest abdom hip thigh knee ankle
## brozek 0.7028852 0.81370622 0.62569993 0.5612844 0.50778587 0.2667826
## age 0.1764497 0.23040942 -0.05033212 -0.2000958 0.01751569 -0.1050581
## weight 0.8941905 0.88799494 0.94088412 0.8686935 0.85316739 0.6136854
## height 0.1348918 0.08781291 0.17039426 0.1484356 0.28605321 0.2647437
## adipos 0.9117986 0.92388010 0.88326922 0.8127061 0.71365983 0.5003166
## neck 0.7848350 0.75407737 0.73495788 0.6956973 0.67240498 0.4778924
## chest 1.0000000 0.91582767 0.82941992 0.7298586 0.71949640 0.4829879
## abdom 0.9158277 1.00000000 0.87406618 0.7666239 0.73717888 0.4532227
## hip 0.8294199 0.87406618 1.00000000 0.8964098 0.82347262 0.5583868
## thigh 0.7298586 0.76662393 0.89640979 1.0000000 0.79917030 0.5397971
## knee 0.7194964 0.73717888 0.82347262 0.7991703 1.00000000 0.6116082
## ankle 0.4829879 0.45322269 0.55838682 0.5397971 0.61160820 1.0000000
## biceps 0.7279075 0.68498272 0.73927252 0.7614774 0.67870883 0.4848545
## forearm 0.5801727 0.50331609 0.54501412 0.5668422 0.55589819 0.4190500
## wrist 0.6601623 0.61983243 0.63008954 0.5586848 0.66450729 0.5661946
## biceps forearm wrist
## brozek 0.49303089 0.36327744 0.3475728
## age -0.04116212 -0.08505555 0.2135306
## weight 0.80041593 0.63030143 0.7297749
## height 0.20781557 0.22864922 0.3220653
## adipos 0.74638418 0.55859425 0.6259066
## neck 0.73114592 0.62366027 0.7448264
## chest 0.72790748 0.58017273 0.6601623
## abdom 0.68498272 0.50331609 0.6198324
## hip 0.73927252 0.54501412 0.6300895
## thigh 0.76147745 0.56684218 0.5586848
## knee 0.67870883 0.55589819 0.6645073
## ankle 0.48485454 0.41904999 0.5661946
## biceps 1.00000000 0.67825513 0.6321264
## forearm 0.67825513 1.00000000 0.5855883
## wrist 0.63212642 0.58558825 1.0000000</code></pre>
<pre class="r"><code>bodyfat.p</code></pre>
<pre><code>## brozek age weight height adipos
## brozek NA 3.044832e-06 0.000000e+00 1.584527e-01 0.00000000
## age 3.044832e-06 NA 8.404326e-01 6.303967e-03 0.05956386
## weight 0.000000e+00 8.404326e-01 NA 5.989982e-07 0.00000000
## height 1.584527e-01 6.303967e-03 5.989982e-07 NA 0.69415116
## adipos 0.000000e+00 5.956386e-02 0.000000e+00 6.941512e-01 NA
## neck 0.000000e+00 7.206646e-02 0.000000e+00 4.614130e-05 0.00000000
## chest 0.000000e+00 4.966465e-03 0.000000e+00 3.231539e-02 0.00000000
## abdom 0.000000e+00 2.249495e-04 0.000000e+00 1.646071e-01 0.00000000
## hip 0.000000e+00 4.263055e-01 0.000000e+00 6.701323e-03 0.00000000
## thigh 0.000000e+00 1.408117e-03 0.000000e+00 1.838840e-02 0.00000000
## knee 0.000000e+00 7.820168e-01 0.000000e+00 3.927576e-06 0.00000000
## ankle 1.770467e-05 9.609999e-02 0.000000e+00 2.062620e-05 0.00000000
## biceps 0.000000e+00 5.153980e-01 0.000000e+00 9.038055e-04 0.00000000
## forearm 2.808188e-09 1.783211e-01 0.000000e+00 2.519644e-04 0.00000000
## wrist 1.446366e-08 6.442181e-04 0.000000e+00 1.721428e-07 0.00000000
## neck chest abdom hip thigh
## brozek 0.000000e+00 0.000000e+00 0.000000e+00 0.000000000 0.000000000
## age 7.206646e-02 4.966465e-03 2.249495e-04 0.426305473 0.001408117
## weight 0.000000e+00 0.000000e+00 0.000000e+00 0.000000000 0.000000000
## height 4.614130e-05 3.231539e-02 1.646071e-01 0.006701323 0.018388401
## adipos 0.000000e+00 0.000000e+00 0.000000e+00 0.000000000 0.000000000
## neck NA 0.000000e+00 0.000000e+00 0.000000000 0.000000000
## chest 0.000000e+00 NA 0.000000e+00 0.000000000 0.000000000
## abdom 0.000000e+00 0.000000e+00 NA 0.000000000 0.000000000
## hip 0.000000e+00 0.000000e+00 0.000000e+00 NA 0.000000000
## thigh 0.000000e+00 0.000000e+00 0.000000e+00 0.000000000 NA
## knee 0.000000e+00 0.000000e+00 0.000000e+00 0.000000000 0.000000000
## ankle 8.881784e-16 4.440892e-16 3.597123e-14 0.000000000 0.000000000
## biceps 0.000000e+00 0.000000e+00 0.000000e+00 0.000000000 0.000000000
## forearm 0.000000e+00 0.000000e+00 0.000000e+00 0.000000000 0.000000000
## wrist 0.000000e+00 0.000000e+00 0.000000e+00 0.000000000 0.000000000
## knee ankle biceps forearm wrist
## brozek 0.000000e+00 1.770467e-05 0.000000e+00 2.808188e-09 1.446366e-08
## age 7.820168e-01 9.609999e-02 5.153980e-01 1.783211e-01 6.442181e-04
## weight 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00
## height 3.927576e-06 2.062620e-05 9.038055e-04 2.519644e-04 1.721428e-07
## adipos 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00
## neck 0.000000e+00 8.881784e-16 0.000000e+00 0.000000e+00 0.000000e+00
## chest 0.000000e+00 4.440892e-16 0.000000e+00 0.000000e+00 0.000000e+00
## abdom 0.000000e+00 3.597123e-14 0.000000e+00 0.000000e+00 0.000000e+00
## hip 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00
## thigh 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00
## knee NA 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00
## ankle 0.000000e+00 NA 4.440892e-16 3.886669e-12 0.000000e+00
## biceps 0.000000e+00 4.440892e-16 NA 0.000000e+00 0.000000e+00
## forearm 0.000000e+00 3.886669e-12 0.000000e+00 NA 0.000000e+00
## wrist 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00 NA</code></pre>
<p>Looking at the figures above, we notice that the response variable brozek is in strong association with all of the explanatory variables from the data set. In fact, it looks as if all variables are in strong correlation with one another, with a very few exceptions such as in the case of <code>height</code> and <code>adipos</code>, <code>hip</code> and <code>age</code>, <code>weight</code> and <code>age</code> to name a few of the most obvious ones. In an example like this one, where there are a high number of variables to consider, it is useful to visualise the correlation matrix. There are several packages available for the visualisation of a correlation matrix in R:</p>
<ul>
<li><a href="https://cran.r-project.org/web/packages/GGally/index.html"><code>GGally</code></a> - extension to <a href="https://ggplot2.tidyverse.org"><code>ggplot2</code></a>
<ul>
<li><a href="https://cran.r-project.org/web/packages/GGally/GGally.pdf"><code>ggpairs()</code></a> function</li>
</ul></li>
<li><a href="https://cran.r-project.org/web/packages/ggcorrplot/index.html"><code>ggcorrplot</code></a> - visualisation of a correlation matrix using <a href="https://ggplot2.tidyverse.org"><code>ggplot2</code></a>
<ul>
<li><a href="https://cran.r-project.org/web/packages/ggcorrplot/ggcorrplot.pdf"><code>ggcorrplot()</code></a> function</li>
</ul></li>
<li><a href="https://cran.r-project.org/web/packages/corrplot/index.html"><code>corrplot</code></a> - rich visualisation of a correlation matrix
<ul>
<li><a href="https://cran.r-project.org/web/packages/corrplot/corrplot.pdf"><code>corrplot.mixed()</code></a> function</li>
</ul></li>
<li><a href="https://cran.r-project.org/web/packages/corrr/index.html"><code>corrr</code></a> - a tool for exploring correlations
<ul>
<li><a href="https://cran.r-project.org/web/packages/corrr/corrr.pdf"><code>network_plot()</code></a> function</li>
</ul></li>
<li><a href="https://cran.r-project.org/web/packages/corrgram/index.html"><code>corrgram</code></a> - calculates correlation of variables and displays the results graphically, based on the <a href="http://euclid.psych.yorku.ca/datavis/papers/corrgram.pdf">Corrgrams: Exploratory displays for correlation matrices</a> paper (Friendly, 2002)
<ul>
<li><a href="https://cran.r-project.org/web/packages/corrgram/corrgram.pdf"><code>corrgram</code></a> function</li>
</ul></li>
<li><a href="https://cran.r-project.org/web/packages/PerformanceAnalytics/Performance+%20Analytics.pdf"><code>PerformanceAnalytics</code></a> - package of econometric functions for performance and risk analysis of financial instruments or portfolios
<ul>
<li><a href="https://cran.r-project.org/web/packages/PerformanceAnalytics/PerformanceAnalytics.pdf"><code>chart.Correlation()</code></a> function</li>
</ul></li>
<li><a href="https://cran.r-project.org/web/packages/psych/index.html"><code>psych</code></a> - a toolbox for personality, psychometric theory and experimental psychology
<ul>
<li><a href="https://cran.r-project.org/web/packages/psych/psych.pdf">`pairs.panels()</a></li>
</ul></li>
</ul>
<p>We will visualise the correlation matrix by using the <code>corrplot</code> package that offers many customization options that have been neatly presented in this short <a href="https://cran.r-project.org/web/packages/corrplot/vignettes/corrplot-intro.html"><code>An Introduction to corrplot Package</code></a> tutorial.</p>
<pre class="r"><code># If you don&#39;t have the &quot;corrplot&quot; package installed yet, uncomment and run the line below
#install.packages(&quot;corrplot&quot;)
suppressPackageStartupMessages(library(corrplot))
corrplot(cor(bodyfat),
method = &quot;ellipse&quot;,
order = &quot;hclust&quot;)</code></pre>
<p><img src="/post/breg_cv/index_files/figure-html/unnamed-chunk-6-1.png" width="672" /></p>
<p>For larger and more complex datasets, the construction of a correlogram has obvious advantages for exploratory purposes, because it shows all the correlations in an uncomplicated manner. For example, the information provided through this correlogram is easy to absorb as positive correlations are displayed in blue and negative correlations in red, with the correlation coefficients and the corresponding colours displayed in the legend. Colour intensity and the width of the ellipse are proportional to the correlation coefficients, making it altogether easier to read and understand.</p>
<p>Here we see, among other things, that a) the majority of variables are positively correlated with one another and that b) <code>age</code> and <code>height</code> are the two variables with weak correlations to most other variables.</p>
<p>Without a doubt this is an example in which <strong>Stepwise regression</strong> and <strong>Best Subsets regression</strong> (<strong>BREG</strong>) procedures can be deployed as the effective tools in helping us to identify useful predictors.</p>
</div>
<div id="best-subsets-regression" class="section level3">
<h3>Best Subsets regression</h3>
<p>The <a href="https://cran.r-project.org/web/packages/leaps/index.html"><code>leaps</code></a> package enables the best subset selection through the application of the <code>regsubsets()</code> function. It identifies the best model that contains a given number of predictors, where <strong>best</strong> is quantified using residual sum of squares (RSS). The syntax is the same as for the <code>lm()</code> function and the <code>summary()</code> command outputs the best set of variables for each model size.</p>
<p>The <code>regsubsets()</code> function (part of the <a href="https://cran.r-project.org/web/packages/leaps/index.html"><code>leaps</code></a> library) performs the best subset selection by identifying the best model that contains a given number of predictors, where <strong>best</strong> is quantified using RSS. The syntax is the same as for <code>lm()</code>. The <code>summary()</code> command outputs the best set of variables for each model size. We will save the results of the call of the <code>summary()</code> function as <code>breg_summary</code>, which will allow us to access the parts we need to focus on.</p>
<pre class="r"><code># # If you don&#39;t have the &quot;leaps&quot; package installed yet, uncomment and run the line below
#install.packages(&quot;leaps&quot;)
library(leaps)
breg_full = regsubsets(brozek ~ ., data = bodyfat)
breg_summary = summary(breg_full)
breg_summary</code></pre>
<pre><code>## Subset selection object
## Call: regsubsets.formula(brozek ~ ., data = bodyfat)
## 14 Variables (and intercept)
## Forced in Forced out
## age FALSE FALSE
## weight FALSE FALSE
## height FALSE FALSE
## adipos FALSE FALSE
## neck FALSE FALSE
## chest FALSE FALSE
## abdom FALSE FALSE
## hip FALSE FALSE
## thigh FALSE FALSE
## knee FALSE FALSE
## ankle FALSE FALSE
## biceps FALSE FALSE
## forearm FALSE FALSE
## wrist FALSE FALSE
## 1 subsets of each size up to 8
## Selection Algorithm: exhaustive
## age weight height adipos neck chest abdom hip thigh knee ankle biceps
## 1 ( 1 ) &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 4 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 5 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## forearm wrist
## 1 ( 1 ) &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot;
## 4 ( 1 ) &quot;*&quot; &quot;*&quot;
## 5 ( 1 ) &quot;*&quot; &quot;*&quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot;</code></pre>
<p>Note that the summary prints out the asterisks (“*”). The presence of a “*” indicates that a given variable is included in the corresponding model. For instance, this output indicates that the best three-variable model contains only <code>weight</code>, <code>abd</code> and <code>wrist</code>. Note that by default, <code>regsubsets()</code> only reports results up to the best eight-variable model. But the <code>nvmax</code> argument option can be used in order to return as many variables as are desired. As we would like to use all the available explanatory variables we will request a fit up to a 14-variable model, that is:</p>
<p><span class="math display">\[Y=f(X_1, X_2, X_3,..., X_{14}) \]</span></p>
<pre class="r"><code>breg_full &lt;- regsubsets(brozek ~., data = bodyfat, nvmax = 14)
breg_summary &lt;- summary(breg_full)
breg_summary</code></pre>
<pre><code>## Subset selection object
## Call: regsubsets.formula(brozek ~ ., data = bodyfat, nvmax = 14)
## 14 Variables (and intercept)
## Forced in Forced out
## age FALSE FALSE
## weight FALSE FALSE
## height FALSE FALSE
## adipos FALSE FALSE
## neck FALSE FALSE
## chest FALSE FALSE
## abdom FALSE FALSE
## hip FALSE FALSE
## thigh FALSE FALSE
## knee FALSE FALSE
## ankle FALSE FALSE
## biceps FALSE FALSE
## forearm FALSE FALSE
## wrist FALSE FALSE
## 1 subsets of each size up to 14
## Selection Algorithm: exhaustive
## age weight height adipos neck chest abdom hip thigh knee ankle biceps
## 1 ( 1 ) &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 4 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 5 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 9 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot;
## 10 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 11 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 12 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 13 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 14 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot;
## forearm wrist
## 1 ( 1 ) &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot;
## 4 ( 1 ) &quot;*&quot; &quot;*&quot;
## 5 ( 1 ) &quot;*&quot; &quot;*&quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot;
## 9 ( 1 ) &quot;*&quot; &quot;*&quot;
## 10 ( 1 ) &quot;*&quot; &quot;*&quot;
## 11 ( 1 ) &quot;*&quot; &quot;*&quot;
## 12 ( 1 ) &quot;*&quot; &quot;*&quot;
## 13 ( 1 ) &quot;*&quot; &quot;*&quot;
## 14 ( 1 ) &quot;*&quot; &quot;*&quot;</code></pre>
<p>💡Remember that we can use the <span class="math inline">\(\bar{R}^2\)</span> to select the best model! We need to discover the other pieces of information contained in the <code>breg_summary</code>.</p>
<pre class="r"><code>names(breg_summary)</code></pre>
<pre><code>## [1] &quot;which&quot; &quot;rsq&quot; &quot;rss&quot; &quot;adjr2&quot; &quot;cp&quot; &quot;bic&quot; &quot;outmat&quot; &quot;obj&quot;</code></pre>
<p>In addition to the output that indicates the inclusion of the variables in the given models we get when we print the summary in the console, the <code>summary()</code> function also returns the necessary statistics for the best model selection. It provides <span class="math inline">\(R^2\)</span>, <span class="math inline">\(RSS\)</span>, <span class="math inline">\(\bar{R}^2\)</span>, <span class="math inline">\(C_p\)</span>, and <span class="math inline">\(BIC\)</span>, which will help us in the selection of the best overall model.</p>
<p>Knowing that the <span class="math inline">\(R^2\)</span> statistic increases monotonically as more variables are included, it will not be effective to use it in the model selection procedure. However, we are going to examine it to see the level to which it increases.</p>
<pre class="r"><code>t(t(sprintf(&quot;%0.2f%%&quot;, breg_summary$rsq * 100)))</code></pre>
<pre><code>## [,1]
## [1,] &quot;66.21%&quot;
## [2,] &quot;71.87%&quot;
## [3,] &quot;72.76%&quot;
## [4,] &quot;73.51%&quot;
## [5,] &quot;73.80%&quot;
## [6,] &quot;74.10%&quot;
## [7,] &quot;74.45%&quot;
## [8,] &quot;74.67%&quot;
## [9,] &quot;74.77%&quot;
## [10,] &quot;74.84%&quot;
## [11,] &quot;74.89%&quot;
## [12,] &quot;74.90%&quot;
## [13,] &quot;74.90%&quot;
## [14,] &quot;74.90%&quot;</code></pre>
<p>As expected, the <span class="math inline">\(R^2\)</span> statistic increases monotonically from 66.21% when only one variable is included in the model to almost 75% with the inclusion of 12 or more variables.</p>
<p>Creating the line plots to visualise <span class="math inline">\(RSS\)</span>, <span class="math inline">\(\bar{R}^2\)</span>, <span class="math inline">\(C_p\)</span>, and <span class="math inline">\(BIC\)</span>, for all of the models at once will help us to decide which model to select. We will keep things simple and create the visualisations using the <code>plot</code> function in <code>base</code> R with argument option <code>type = "l"</code> onto which we will superimpose the corresponding minimal or maximal values using the <code>point</code> function in respect of the statistic displayed.</p>
<pre class="r"><code>par(mfrow = c(2,2)) # Set up a 2x2 grid for display of 4 plots at once
plot(breg_summary$rss, xlab = &quot;Number of Variables&quot;, ylab = &quot;RSS&quot;, type = &quot;l&quot;)
# line plot of adjusted R^2 statistic
plot(breg_summary$adjr2, xlab = &quot;Number of Variables&quot;, ylab = &quot;Adjusted RSq&quot;, type = &quot;l&quot;)
# identify the location of the maximum point
adj_r2_max = which.max(breg_summary$adjr2)
# plot a red point to indicate the model with the largest adjusted R^2 statistic
points(adj_r2_max, breg_summary$adjr2[adj_r2_max], col =&quot;red&quot;, cex = 1, pch = 20)
# line plot of C_p and BIC, but this time in a search of the models with the SMALLEST statistic
# line plot of C_p statistic
plot(breg_summary$cp, xlab = &quot;Number of Variables&quot;, ylab = &quot;Cp&quot;, type = &quot;l&quot;)
cp_min = which.min(breg_summary$cp)
points(cp_min, breg_summary$cp[cp_min], col = &quot;red&quot;, cex = 1, pch = 20)
# line plot of BIC statistic
plot(breg_summary$bic, xlab = &quot;Number of Variables&quot;, ylab = &quot;BIC&quot;, type = &quot;l&quot;)
bic_min = which.min(breg_summary$bic)
points(bic_min, breg_summary$bic[bic_min], col = &quot;red&quot;, cex = 1, pch = 20)</code></pre>
<p><img src="/post/breg_cv/index_files/figure-html/unnamed-chunk-11-1.png" width="672" /></p>
<p>We see that the measures are not in sync with one another and we realise that no one measure is going to give us an entirely accurate picture. According to <span class="math inline">\(\bar{R}^2\)</span> and <span class="math inline">\(C_p\)</span> the best performing model is the one with 8 variables and according to <span class="math inline">\(BIC\)</span> the best performing model has only 4 variables. This outcome suggests that the models with fewer than 4 predictors is insufficient, while a model with more than 8 predictors will overfit.</p>
<p>The <code>regsubsets()</code> function has a built-in <code>plot()</code> command which can be used to display the selected variables for the best model with a given number of predictors, ranked according to the <span class="math inline">\(R^2\)</span>, <span class="math inline">\(\bar{R}^2\)</span>, <span class="math inline">\(C_p\)</span>, and <span class="math inline">\(BIC\)</span> statistic. <em>The top row</em> of each plot contains a black square for each variable selected according to the optimal model associated with that statistic. That is, when creating this plot for the display of the <span class="math inline">\(R^2\)</span> value it is not a surprise to see that the top row indicates inclusion of all 14 predictors for the <span class="math inline">\(R^2\)</span> of 75%.</p>
<pre class="r"><code>plot(breg_full, scale=&quot;r2&quot;)</code></pre>
<p><img src="/post/breg_cv/index_files/figure-html/unnamed-chunk-12-1.png" width="672" /></p>
<p>We can now identify the suggested 8 predictors for the best model by observing the top row and checking which of the variables has a black square.<br />
In the following plot by observing the top row and checking which of the variable has a black square we can identify the exact 8 predictors to be used for the best model as suggested by <span class="math inline">\(\bar{R}^2\)</span>.</p>
<pre class="r"><code>plot(breg_full, scale =&quot;adjr2&quot;)</code></pre>
<p><img src="/post/breg_cv/index_files/figure-html/unnamed-chunk-13-1.png" width="672" /></p>
<p>It suggests that <span class="math display">\[brozek = f(age, weight, neck, abdom, hip, thigh, forearm, wrist)\]</span>
We can use the <code>coef()</code> function to see the coefficient estimates associated with this model.</p>
<pre class="r"><code>coef(breg_full, 8)</code></pre>
<pre><code>## (Intercept) age weight neck abdom hip
## -20.06213373 0.05921577 -0.08413521 -0.43189267 0.87720667 -0.18641032
## thigh forearm wrist
## 0.28644340 0.48254563 -1.40486912</code></pre>
<pre class="r"><code>plot(breg_full, scale=&quot;Cp&quot;)</code></pre>
<p><img src="/post/breg_cv/index_files/figure-html/unnamed-chunk-15-1.png" width="672" /></p>
<p><span class="math inline">\(C_p\)</span> suggests the same set of predictors as <span class="math inline">\(\bar{R}^2\)</span>.</p>
<pre class="r"><code>plot(breg_full, scale=&quot;bic&quot;)</code></pre>
<p><img src="/post/breg_cv/index_files/figure-html/unnamed-chunk-16-1.png" width="672" /></p>
<p>We see that two models share a BIC close to -310. However, the model with the lowest <span class="math inline">\(BIC\)</span> is the four-variable model that contains only <code>weight</code>, <code>abdom</code>, <code>forearm</code> and <code>wrist</code></p>
<pre class="r"><code>coef(breg_full, 4)</code></pre>
<pre><code>## (Intercept) weight abdom forearm wrist
## -31.2967858 -0.1255654 0.9213725 0.4463824 -1.3917662</code></pre>
</div>
<div id="forward-and-backward-stepwise-selection" class="section level3">
<h3>Forward and Backward Stepwise Selection</h3>
<p>To perform stepwise selection we will also use the <code>regsubsets()</code> function, but this time with the argument <code>method</code> set to either “<code>forward</code>” or “<code>backward</code>” depending on which of the two stepwise selections we wish to perform.</p>
<pre class="r"><code># Forward stepwise selection
stepw_fwd &lt;- regsubsets(brozek ~ ., data = bodyfat, nvmax = 14, method = &quot;forward&quot;)
summary(stepw_fwd)</code></pre>
<pre><code>## Subset selection object
## Call: regsubsets.formula(brozek ~ ., data = bodyfat, nvmax = 14, method = &quot;forward&quot;)
## 14 Variables (and intercept)
## Forced in Forced out
## age FALSE FALSE
## weight FALSE FALSE
## height FALSE FALSE
## adipos FALSE FALSE
## neck FALSE FALSE
## chest FALSE FALSE
## abdom FALSE FALSE
## hip FALSE FALSE
## thigh FALSE FALSE
## knee FALSE FALSE
## ankle FALSE FALSE
## biceps FALSE FALSE
## forearm FALSE FALSE
## wrist FALSE FALSE
## 1 subsets of each size up to 14
## Selection Algorithm: forward
## age weight height adipos neck chest abdom hip thigh knee ankle biceps
## 1 ( 1 ) &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 4 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 5 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 9 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot;
## 10 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 11 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 12 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 13 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 14 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot;
## forearm wrist
## 1 ( 1 ) &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot;
## 4 ( 1 ) &quot;*&quot; &quot;*&quot;
## 5 ( 1 ) &quot;*&quot; &quot;*&quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot;
## 9 ( 1 ) &quot;*&quot; &quot;*&quot;
## 10 ( 1 ) &quot;*&quot; &quot;*&quot;
## 11 ( 1 ) &quot;*&quot; &quot;*&quot;
## 12 ( 1 ) &quot;*&quot; &quot;*&quot;
## 13 ( 1 ) &quot;*&quot; &quot;*&quot;
## 14 ( 1 ) &quot;*&quot; &quot;*&quot;</code></pre>
<pre class="r"><code>stepw_bwd &lt;- regsubsets(brozek ~ ., data = bodyfat, nvmax = 14, method = &quot;backward&quot;)
summary(stepw_bwd)</code></pre>
<pre><code>## Subset selection object
## Call: regsubsets.formula(brozek ~ ., data = bodyfat, nvmax = 14, method = &quot;backward&quot;)
## 14 Variables (and intercept)
## Forced in Forced out
## age FALSE FALSE
## weight FALSE FALSE
## height FALSE FALSE
## adipos FALSE FALSE
## neck FALSE FALSE
## chest FALSE FALSE
## abdom FALSE FALSE
## hip FALSE FALSE
## thigh FALSE FALSE
## knee FALSE FALSE
## ankle FALSE FALSE
## biceps FALSE FALSE
## forearm FALSE FALSE
## wrist FALSE FALSE
## 1 subsets of each size up to 14
## Selection Algorithm: backward
## age weight height adipos neck chest abdom hip thigh knee ankle biceps
## 1 ( 1 ) &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 4 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 5 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 9 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot;
## 10 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 11 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 12 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 13 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 14 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot;
## forearm wrist
## 1 ( 1 ) &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot;
## 4 ( 1 ) &quot;*&quot; &quot;*&quot;
## 5 ( 1 ) &quot;*&quot; &quot;*&quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot;
## 9 ( 1 ) &quot;*&quot; &quot;*&quot;
## 10 ( 1 ) &quot;*&quot; &quot;*&quot;
## 11 ( 1 ) &quot;*&quot; &quot;*&quot;
## 12 ( 1 ) &quot;*&quot; &quot;*&quot;
## 13 ( 1 ) &quot;*&quot; &quot;*&quot;
## 14 ( 1 ) &quot;*&quot; &quot;*&quot;</code></pre>
<pre class="r"><code>stepw_new &lt;- regsubsets(brozek ~ ., data = bodyfat, nvmax = 14, method = &quot;seqrep&quot;)
summary(stepw_new)</code></pre>
<pre><code>## Subset selection object
## Call: regsubsets.formula(brozek ~ ., data = bodyfat, nvmax = 14, method = &quot;seqrep&quot;)
## 14 Variables (and intercept)
## Forced in Forced out
## age FALSE FALSE
## weight FALSE FALSE
## height FALSE FALSE
## adipos FALSE FALSE
## neck FALSE FALSE
## chest FALSE FALSE
## abdom FALSE FALSE
## hip FALSE FALSE
## thigh FALSE FALSE
## knee FALSE FALSE
## ankle FALSE FALSE
## biceps FALSE FALSE
## forearm FALSE FALSE
## wrist FALSE FALSE
## 1 subsets of each size up to 14
## Selection Algorithm: &#39;sequential replacement&#39;
## age weight height adipos neck chest abdom hip thigh knee ankle biceps
## 1 ( 1 ) &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 4 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 5 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 9 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot;
## 10 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 11 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot;
## 12 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot;
## 13 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot;
## 14 ( 1 ) &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot;*&quot;
## forearm wrist
## 1 ( 1 ) &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot;
## 4 ( 1 ) &quot;*&quot; &quot;*&quot;
## 5 ( 1 ) &quot;*&quot; &quot;*&quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot;
## 7 ( 1 ) &quot; &quot; &quot; &quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot;
## 9 ( 1 ) &quot;*&quot; &quot;*&quot;
## 10 ( 1 ) &quot;*&quot; &quot;*&quot;
## 11 ( 1 ) &quot; &quot; &quot; &quot;
## 12 ( 1 ) &quot;*&quot; &quot;*&quot;
## 13 ( 1 ) &quot;*&quot; &quot; &quot;
## 14 ( 1 ) &quot;*&quot; &quot;*&quot;</code></pre>
<p>We can see that the best one-variable through four-variable models are each identical for best subset and forward and backward selection. Difference occurs for the five-variable model for which the backward selection method selects different set of predictors from the other two.</p>
<pre class="r"><code>coef(breg_full, 5)</code></pre>
<pre><code>## (Intercept) weight neck abdom forearm wrist
## -27.4118045 -0.1136977 -0.3382162 0.9325574 0.4964234 -1.1519642</code></pre>
<pre class="r"><code>coef(stepw_fwd, 5)</code></pre>
<pre><code>## (Intercept) weight neck abdom forearm wrist
## -27.4118045 -0.1136977 -0.3382162 0.9325574 0.4964234 -1.1519642</code></pre>
<pre class="r"><code>coef(stepw_bwd, 5)</code></pre>
<pre><code>## (Intercept) age weight abdom forearm wrist
## -27.89197896 0.03594628 -0.10392579 0.87162722 0.47755107 -1.67587889</code></pre>
<p>Another difference in selection happens again for the six-factor model for which this time forward selection suggests a different set of predictors from the other two.</p>
<pre class="r"><code>coef(breg_full, 6)</code></pre>
<pre><code>## (Intercept) age weight abdom thigh forearm
## -34.8393955 0.0566420 -0.1280528 0.8459061 0.2081457 0.4592612
## wrist
## -1.6278694</code></pre>
<pre class="r"><code>coef(stepw_fwd, 6)</code></pre>
<pre><code>## (Intercept) age weight neck abdom forearm
## -23.21489668 0.04059830 -0.08818394 -0.36881380 0.87738614 0.53615286
## wrist
## -1.45115127</code></pre>
<pre class="r"><code>coef(stepw_bwd, 6)</code></pre>
<pre><code>## (Intercept) age weight abdom thigh forearm
## -34.8393955 0.0566420 -0.1280528 0.8459061 0.2081457 0.4592612
## wrist
## -1.6278694</code></pre>
<p>For the models with seven predictors and more all three methods suggest the same variables.</p>
</div>
</div>
<div id="the-validation-set-approach-and-cross-validation-for-the-selection-of-the-best-model" class="section level2">
<h2>The Validation Set Approach and Cross-Validation for the selection of the best model</h2>
<p>In this section validation set and cross-validation approaches will be used to chose the best model among a set of models of different sizes. When applying these approaches we split the given data into two subsets:</p>
<ul>
<li><strong>training data</strong>: a subset to train a model</li>
<li><strong>test data</strong>: a subset to test the model</li>
</ul>
<p>The training data is used for model estimation and variable selection, whilst the remaining test data is put aside and reserved for testing the accuracy of the model (see section on <a href="http://dataliteracy.rbind.io/module4/what_is_ml/">Machine Learning</a>). That is, the best model variable selection is performed using only the training observations that are randomly selected from the original data.</p>
<p>We begin the application of validation approach in R by splitting the <code>bodyaft</code> data into a training set and a test set. First, we set a random seed to ensure the same data split each time we run the randomised process of splitting the <code>bodyfat</code> data into training set and test set. The <code>set.seed()</code> function sets the starting number used to generate a sequence of random numbers (it ensures that you get the same result if you start with that same seed each time you run the same process). Next, we create a random vector, <code>train</code>, of elements equal to TRUE if the corresponding observation is in the training set, and FALSE otherwise. A random vector <code>test</code> is obtained by using the <code>!</code> command that creates TRUEs to be switched to FALSEs and vice versa. Once both <code>train</code> and <code>test</code> vectors are obtained we perform best subset on the <code>training</code> set of the <code>bodyfat</code> data using the now familiar <code>regsubset()</code> function.</p>
<pre class="r"><code>set.seed(111)
train &lt;- sample(c(TRUE, FALSE), nrow(bodyfat), rep = TRUE)
test &lt;- (!train )
breg &lt;- regsubsets(brozek ~., data = bodyfat[train, ], nvmax = 14)</code></pre>
<p>We now compute the validation set error for the best model of each model size. We first make a model matrix from the test data. The <code>model.matrix()</code> function is used in many regression packages for building an <strong><span class="math inline">\(X\)</span></strong> matrix from data.</p>
<p><span style="color: gray"></p>
<p>In the multiple regression setting because of the potentially large number of predictors, it is more convenient to write the expressions using matrix notation. To develop the analogy let us quickly look at the matrix notation for the 14-variable linear regression function:</p>
<p><span style="color: gray"><span class="math display">\[y_i = \beta_0 + \beta_1x_1 + \epsilon_i \;\;\;\;\;\; \text{for} \;\; i=1, 2, ...n\]</span>
<span style="color: gray">If we actually let <span class="math inline">\(i = 1, ..., n\)</span>, we see that we obtain <span class="math inline">\(n\)</span> equations:
<span style="color: gray"><span class="math display">\[y_i = \beta_0 + \beta_1x_1 + \epsilon_1\]</span>
<span style="color: gray"><span class="math display">\[y_i = \beta_0 + \beta_1x_2 + \epsilon_2\]</span>
<span style="color: gray"><span class="math display">\[ \vdots\]</span>
<span style="color: gray"><span class="math display">\[y_i = \beta_0 + \beta_1x_n + \epsilon_n\]</span>
<span style="color: gray">It is more efficient to use matrices to define the regression model which can be formulated for the above simple linear regression function in matrix notation as:
<span style="color: gray"><span class="math display">\[
\begin{bmatrix}
y_1\\y_2\\ \vdots\\y_n
\end{bmatrix} =
\begin{bmatrix}
1 &amp; x_1 \\
1 &amp; x_2 \\
\vdots &amp; \vdots \\
1 &amp; x_n \\
\end{bmatrix}
\begin{bmatrix}
\beta_0\\\beta_1\\
\end{bmatrix}
+ \begin{bmatrix}
\varepsilon_1\\\varepsilon_2\\ \vdots\\\varepsilon_n
\end{bmatrix}
\]</span>
<span style="color: gray">Instead of writing out the <span class="math inline">\(n\)</span> equations, using matrix notation, the simple linear regression function reduces to a short and simple statement:
<span style="color: gray">
<span class="math display">\[\mathbf{Y} = \mathbf{X\beta} + \mathbf{\varepsilon},\]</span>
where matrices <strong><span class="math inline">\(Y\)</span></strong>, <span class="math inline">\(\mathbf{\varepsilon}\)</span> are of the size <span class="math inline">\(n \times 1\)</span>,
<span class="math inline">\(\mathbf{\beta}\)</span> of <span class="math inline">\(2 \times 1\)</span> and finally matrix <strong><span class="math inline">\(Y\)</span></strong> of <span class="math inline">\(n \times 2\)</span>.
</span></p>
<p>Once we have created <strong><span class="math inline">\(X\)</span></strong> matrix we run a loop, and for each <span class="math inline">\(i\)</span>, we</p>
<ol style="list-style-type: decimal">
<li>extract the coefficients from <code>breg</code> for the best model of that size</li>
<li>multiply them into the appropriate columns of the test model matrix to form the predictions and</li>
<li>compute the test MSE</li>
</ol>
<pre class="r"><code>test.mat &lt;- model.matrix(brozek ~., data = bodyfat[test, ])
val.errors &lt;- rep(NA, 14)
for(i in 1:14){
coefi &lt;- coef(breg, id = i)
pred &lt;- test.mat[ ,names(coefi)] %*% coefi
val.errors[i] &lt;- mean((bodyfat$brozek[test] - pred)^2)
}
val.errors</code></pre>
<pre><code>## [1] 20.43699 16.90849 16.75413 16.20119 16.04250 17.82335 17.46721 17.64863
## [9] 17.55764 17.06395 16.60744 17.00481 16.94781 16.84543</code></pre>
<pre class="r"><code>coef(breg, which.min(val.errors))</code></pre>
<pre><code>## (Intercept) weight neck abdom forearm wrist
## -25.45466859 -0.09711483 -0.39312889 0.95767906 0.38200504 -1.25643570</code></pre>
<p>It looks as if the best model is the one that contains five variables. As there is no <code>predict()</code> method for <code>regsubsets()</code> we can capture our steps above and write our own predict method.</p>
<pre class="r"><code>predict.regsubsets &lt;- function(object, newdata, id,...){
form &lt;- as.formula(object$call[[2]])
mat &lt;- model.matrix(form, newdata)
coefi &lt;- coef(object, id = id)
xvars &lt;- names(coefi)
mat[,xvars ] %*% coefi
}</code></pre>
<p>The function almost does what we did above. The only tricky bit is the extraction of the formula used in the call to <code>regsubsets()</code>. We are going to use this function to do cross-validation. We start by performing best subset selection on the full data set, and select the best five-variable model. In order to get more accurate coefficient estimates we make use of the full data set, rather than using the variables that were obtained from the training set. We should realise that the best five-variable model on the full data set may differ from the corresponding five-variable model on the training set.</p>
<pre class="r"><code>breg &lt;- regsubsets(brozek ~., data = bodyfat , nvmax = 14)
coef(breg, 5)</code></pre>
<pre><code>## (Intercept) weight neck abdom forearm wrist
## -27.4118045 -0.1136977 -0.3382162 0.9325574 0.4964234 -1.1519642</code></pre>
<p>We are in luck as the best five-variable model on the full data set has the same set of variables as identified for the best five-variable model on the training set.</p>
<p>We now try to choose among the models of different sizes using cross-validation and perform best subset selection within each of the <span class="math inline">\(k\)</span> training sets. This taxing task can be easily performed in R by creating a vector that allocates each observation to one of k = 10 folds and a matrix that will hold the results.</p>
<pre class="r"><code>k &lt;- 10
set.seed(111)
folds &lt;- sample (1:k, nrow(bodyfat), replace = TRUE)
cv.errors &lt;- matrix(NA, k, 14, dimnames = list(NULL, paste (1:14)))</code></pre>
<p>Now we write a <code>for</code> loop that performs cross-validation:</p>
<ol style="list-style-type: decimal">
<li>in the <span class="math inline">\(j^{\text{th}}\)</span> fold, the elements of folds that equal <span class="math inline">\(j\)</span> are in the <em>test set</em>, and the remainder are in the <em>training set</em>.</li>
<li>next,</li>
</ol>
<ul>
<li>make our predictions for each model size (using our new <code>predict()</code> method)</li>
<li>compute the test errors on the appropriate subset, and</li>
<li>store them in the appropriate slot in the matrix <code>cv.errors</code>.</li>
</ul>
<pre class="r"><code>for(j in 1:k){
best.fit &lt;- regsubsets(brozek ~., data = bodyfat[folds!=j, ], nvmax = 14)
for(i in 1:14){
pred &lt;- predict(best.fit, bodyfat[folds==j, ], id = i)
cv.errors[j, i] &lt;- mean((bodyfat$brozek[folds==j] - pred)^2)
}
}</code></pre>
<p>This creates a 10 <span class="math inline">\(\times\)</span> 14 matrix, where the <span class="math inline">\((i, j)^{\text{th}}\)</span> element represents the test MSE for the <span class="math inline">\(i^{\text{th}}\)</span> cross-validation fold for the best <span class="math inline">\(j\)</span>-variable model. We use <code>apply()</code> to average over the columns of this matrix in order to obtain a vector for which the <span class="math inline">\(j^{\text{th}}\)</span> element is the cross-validation error for the <span class="math inline">\(j\)</span>-variable model.</p>
<pre class="r"><code>mean.cv.errors &lt;- apply(cv.errors, 2, mean)
mean.cv.errors</code></pre>
<pre><code>## 1 2 3 4 5 6 7 8
## 20.34108 17.51758 17.10110 17.13365 17.53639 17.85260 17.54328 17.54522
## 9 10 11 12 13 14
## 31.93117 35.00292 35.25575 34.60876 34.99476 34.96073</code></pre>
<pre class="r"><code>cv.errors</code></pre>
<pre><code>## 1 2 3 4 5 6 7 8
## [1,] 12.32946 9.416819 9.903653 11.10810 12.19012 13.10063 13.37630 13.56035
## [2,] 12.48594 13.771297 13.852726 12.72790 12.99102 13.28162 13.21003 13.36145
## [3,] 15.55276 13.954025 14.715055 14.75803 15.87441 14.79482 15.16436 13.98156
## [4,] 31.60131 23.661676 21.820352 20.87529 22.30423 22.44055 23.90745 22.11288
## [5,] 21.10524 24.206510 21.984589 21.35164 22.34848 23.97423 20.04423 20.86420
## [6,] 21.92108 16.523561 16.195050 15.11413 15.57349 15.48747 14.61914 14.04020
## [7,] 19.97793 21.266331 20.512493 20.99681 20.69149 20.58539 20.26534 19.82136
## [8,] 17.86496 12.694993 12.796383 13.62946 13.55600 13.63481 12.65788 12.28301
## [9,] 15.83071 15.834488 14.922103 15.21815 14.95377 15.17270 14.62951 15.25427
## [10,] 34.74140 23.846053 24.308575 25.55695 24.88093 26.05378 27.55856 30.17297
## 9 10 11 12 13 14
## [1,] 13.60245 13.58774 13.41241 13.46334 13.44878 13.45282
## [2,] 14.11558 14.21637 13.99672 13.94406 14.02302 13.99533
## [3,] 14.09255 13.99584 14.08496 14.16646 14.69319 14.91532
## [4,] 22.23907 21.92254 22.40096 22.45297 22.49736 22.50527
## [5,] 165.51126 197.37461 200.09727 193.90667 196.86868 196.97994
## [6,] 13.87184 13.77324 13.69918 13.80556 13.97567 13.98346
## [7,] 19.94554 20.10140 20.08304 20.15145 20.17206 20.19522
## [8,] 12.32055 12.29925 12.06850 12.06478 12.04933 12.05878
## [9,] 15.56205 15.55901 15.21091 15.30418 15.32023 15.32022
## [10,] 28.05086 27.19924 27.50356 26.82817 26.89926 26.20099</code></pre>
<pre class="r"><code>par(mfrow=c(1, 1))
plot(mean.cv.errors, type=&quot;b&quot;)</code></pre>
<p><img src="/post/breg_cv/index_files/figure-html/unnamed-chunk-27-1.png" width="672" /></p>
<p>We see that cross-validation selects a three-variable model. We will use <code>breg_full</code> in order to obtain the three-variable model.</p>
<pre class="r"><code>#breg &lt;- regsubsets(brozek ~., data = bodyfat, nvmax = 4)
coef(breg_full, 3)</code></pre>
<pre><code>## (Intercept) weight abdom wrist
## -24.7612893 -0.1055832 0.9019083 -1.1456968</code></pre>
</div>
<div id="caret-stepwise-and-cross-validation-put-together" class="section level2">
<h2><code>caret</code>: Stepwise and Cross-Validation put together</h2>
<p>The <a href="https://cran.r-project.org/web/packages/caret/index.html"><code>caret</code></a> package makes modelling in R easier. It combines two in one:</p>
<ul>
<li>automatically resamples the models</li>
<li>while conducting parameter tuning</li>
</ul>
<p><code>caret</code>’s most affable feature is its consistent modelling syntax which enables you to build and compare models with very little splurge. By simply changing the <code>method</code> argument, you can easily investigate varying applicable models adopted from the pre-existing R packages. In total there are <a href="https://topepo.github.io/caret/available-models.html">over two hundred different models</a> available in <code>caret</code>.</p>
<p>Another sublime feature of <code>caret</code> lies in its <code>train()</code> function, which enables the use of the same function to run all of the competing models. It accepts several caret-specific arguments that provide capabilities for different resampling methods, performance measures, and algorithms for choosing the best model. Running the <code>train()</code> function will create a train object with estimated parameters for a selected model from a set of given data with many other useful results.</p>
<p>We will use <code>caret</code> to adopt 10-fold cross-validation, that will split <code>bodyfat</code> data into 10 approximately equal chunks to perform stepwise selections using the <code>leapBackward</code> method adopted from the <code>leaps</code> package.
The model will be developed based on 9 chunks and predict the <span class="math inline">\(\text{$10^{th}$}\)</span> until all of the folds are treated as a validation set, while the model is fitted on the remaining 10-1 folds. <code>caret</code> takes care of setting up this resampling with the help of the <code>trainControl()</code> function and the <code>trControl</code> argument of its <code>train()</code> function. As the data set contains 14 predictors, we will vary <code>nvmax</code> from 1 to 14 resulting in the identification of the 14 best models with different sizes: the best 1-variable model, the best 2-variables model, and so on up to the best 14-variables model.</p>
<pre class="r"><code>#If you don&#39;t have the &quot;caret&quot; package installed yet, uncomment and run the line below
#install.packages(&quot;caret&quot;)
# Train the model
suppressPackageStartupMessages(library(caret))
set.seed(111)
# Set up repeated k-fold cross-validation
train.control &lt;- trainControl(method = &quot;cv&quot;, number = 10)
# Train the model
stepwb &lt;- train(brozek ~., data = bodyfat,
method = &quot;leapBackward&quot;,
tuneGrid = data.frame(nvmax = 1:14),
trControl = train.control
)
class(stepwb)</code></pre>
<pre><code>## [1] &quot;train&quot; &quot;train.formula&quot;</code></pre>
<pre class="r"><code>stepwb$results</code></pre>
<pre><code>## nvmax RMSE Rsquared MAE RMSESD RsquaredSD MAESD
## 1 1 4.500901 0.6779062 3.667717 0.8055119 0.11240896 0.5675990
## 2 2 4.260896 0.7022554 3.501954 0.7003423 0.11479575 0.6140745
## 3 3 4.257837 0.7103567 3.493261 0.6797350 0.09896541 0.5393138
## 4 4 4.173503 0.7218316 3.444086 0.6125089 0.09224353 0.5335659
## 5 5 4.227939 0.7159505 3.511170 0.6146446 0.08503459 0.5306893
## 6 6 4.186677 0.7200482 3.465637 0.6156029 0.08406015 0.5571481
## 7 7 4.139519 0.7272454 3.387318 0.6905944 0.08203603 0.5906919
## 8 8 4.130202 0.7290699 3.379290 0.6346024 0.07918020 0.5620786
## 9 9 4.138647 0.7279039 3.397510 0.6191154 0.08006510 0.5422292
## 10 10 4.151920 0.7245898 3.410971 0.6212871 0.08515454 0.5505494
## 11 11 4.140745 0.7268756 3.396724 0.6400128 0.08463738 0.5612896
## 12 12 4.183797 0.7214162 3.423895 0.6614864 0.08789564 0.5794326
## 13 13 4.196041 0.7197241 3.434989 0.6747641 0.09008652 0.5887054
## 14 14 4.199835 0.7192288 3.437822 0.6768053 0.09037830 0.5945132</code></pre>
<p>The output above shows different metrics and their standard deviation for comparing the accuracy of the 14 best models for a different number of variables.</p>
<ul>
<li><code>nvmax</code>: the number of variables; in the best model for the given number of variables</li>
<li><code>RMSE</code> and <code>MAE</code> are two different metrics measuring the prediction error of the model, implying that the lower their value, the better the model.</li>
<li><code>Rsquared</code> indicates the correlation between the observed outcome values and the values predicted by the model, meaning that the higher the <code>Rsquared</code>, the better the model.</li>
</ul>
<p>Using the 10-fold cross-validation we have estimated average prediction error (<code>RMSE</code>) for each of the 14 best models for the given number of variables. The <code>RMSE</code> statistical metric is used to compare the 14 selected models and to automatically choose the model that minimises the <code>RMSE</code> as the best model. It can be seen that the model with 8 variables (<code>nvmax</code> = 8) is the one that has the lowest <code>RMSE</code>. We can call <code>bestTune</code> from the created <code>stepwb</code> object of class <code>train</code> to directly display the “best tuning values” (<code>nvmax</code>) and <code>finalModel</code> to see selected variables of the best model.</p>
<pre class="r"><code># Final model coefficients
stepwb$finalModel</code></pre>
<pre><code>## Subset selection object
## 14 Variables (and intercept)
## Forced in Forced out
## age FALSE FALSE
## weight FALSE FALSE
## height FALSE FALSE
## adipos FALSE FALSE
## neck FALSE FALSE
## chest FALSE FALSE
## abdom FALSE FALSE
## hip FALSE FALSE
## thigh FALSE FALSE
## knee FALSE FALSE
## ankle FALSE FALSE
## biceps FALSE FALSE
## forearm FALSE FALSE
## wrist FALSE FALSE
## 1 subsets of each size up to 8
## Selection Algorithm: backward</code></pre>
<pre class="r"><code># Summary of the model
stepwb$bestTune</code></pre>
<pre><code>## nvmax
## 8 8</code></pre>
<pre class="r"><code>summary(stepwb$finalModel)</code></pre>
<pre><code>## Subset selection object
## 14 Variables (and intercept)
## Forced in Forced out
## age FALSE FALSE
## weight FALSE FALSE
## height FALSE FALSE
## adipos FALSE FALSE
## neck FALSE FALSE
## chest FALSE FALSE
## abdom FALSE FALSE
## hip FALSE FALSE
## thigh FALSE FALSE
## knee FALSE FALSE
## ankle FALSE FALSE
## biceps FALSE FALSE
## forearm FALSE FALSE
## wrist FALSE FALSE
## 1 subsets of each size up to 8
## Selection Algorithm: backward
## age weight height adipos neck chest abdom hip thigh knee ankle biceps
## 1 ( 1 ) &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 4 ( 1 ) &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 5 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot;*&quot; &quot; &quot; &quot;*&quot; &quot;*&quot; &quot;*&quot; &quot; &quot; &quot; &quot; &quot; &quot;
## forearm wrist
## 1 ( 1 ) &quot; &quot; &quot; &quot;
## 2 ( 1 ) &quot; &quot; &quot; &quot;
## 3 ( 1 ) &quot; &quot; &quot;*&quot;
## 4 ( 1 ) &quot;*&quot; &quot;*&quot;
## 5 ( 1 ) &quot;*&quot; &quot;*&quot;
## 6 ( 1 ) &quot;*&quot; &quot;*&quot;
## 7 ( 1 ) &quot;*&quot; &quot;*&quot;
## 8 ( 1 ) &quot;*&quot; &quot;*&quot;</code></pre>
<p>💡Typing <code>?caret::train</code> inside R’s console will bring a more detailed explanation on how to use the <code>train()</code> function and information about the expected outcomes.</p>
<div id="summary" class="section level3">
<h3>Summary</h3>
<p>All of the above approaches have suggested different variable selection for the best fitted model. We have noticed by examining the correlogram high multicollinearity amongst the variables in the <code>bodyfat</code>. It reveals a weak correlation only between three to four variables. This is a frequently encountered problem in multiple regression analysis, where such an interrelationship among explanatory variables obscures their relationship with the response variable. This causes computational instability in model estimation resulting in the different suggestions of the best model for different variable selection approaches:</p>
<ul>
<li>Best Subsets Regression: has indicated that the models with a fewer than 4 predictors is insufficient, while a model with more than 8 predictors will overfit</li>
<li>Stepwise Forward and Stepwise Backward selections: has suggested different sets of variables for the five-factor model and the six-factor model</li>
<li>Validation Set: has recommended the five-factor model as the best one</li>
<li>10-Fold Cross-Validation: has argued the case for the three-factor model</li>
<li>Backward Selection with automatic 10-Fold Cross-Validation adopted by the application of the <code>caret</code> package: identifies the eight-factor model as the best one</li>
</ul>
<p>Our main goal was to derive a model to predict body fat (variable <code>brozek</code>) using all predictors except for <code>siri</code>, <code>density</code> and <code>free</code> from the <code>fat</code> data available from the <code>faraway</code> package. Guided by the outcomes of the above analysis and a desire to describe the behaviour of this multivariate data set parsimoniously (see Miller, 2002) we suggest the four-factor model for which <span class="math inline">\(\bar{R}^2\)</span> = 73.51%. Adding another 4 variables as suggested by Backward Selection with automatic 10-Fold Cross-Validation would increase <span class="math inline">\(\bar{R}^2\)</span> statistic by only 1.16%.</p>
<pre class="r"><code>coef(breg_full, 4)</code></pre>
<pre><code>## (Intercept) weight abdom forearm wrist
## -31.2967858 -0.1255654 0.9213725 0.4463824 -1.3917662</code></pre>
<p><span class="math display">\[brozek = -31.2967858 - 0.1255654 \; weight + 0.9213725 \; abdom + 0.4463824 \; forearm - 1.3917662 \; wrist\]</span>
Applying different subset selection approaches we have pruned a list of plausible explanatory variables down to a parsimonious collection of the “most useful” variables. We need to realise that subset selection approaches for multiple regression modelling should be used as a tool that can help us avoid the tiresome process of trying all possible combinations of explanatory variables, by testing variables one by one. However, as statistical models can serve different purposes we need to incorporate prior knowledge when it is possible. Solely reliance on subset selection approaches is wrong and it could be misleading (Smith, 2018).</p>