This class has been extremely informative! I think the best part of this class is the fact that everything taught at a conceptual level was reinforced through practice. Every week there was an assignment due that encompassed the learning material covered in the lectures. This is the way to do it, especially when the concepts involve programming. Learning SAS and R together was a bit challenging, but learning the power of what these tools can do from analytical standpoint was extremely valuable. I really enjoyed the flexibility and support offered from Mike Ames. I think he brings a lot of insight not just from an academic standpoint, but from a vocational standpoint as well. With the combination of these perspectives, the students can get a strong understanding of what needs to be learned and what can be considered “extra noise” in the analytics field. Additionally, the broad spectrum of analytical topics covered in this class should certainly provide a good base for moving into other more focused classes. Thanks for a great first semester!

This week’s focus was mostly about logistic regression – which is super useful, and different than our typical linear regression model which focuses on continuous variable prediction. Logistic regression, on the other hand, falls under the binomial category – and predicts essentially a boolean value (either yes or no / 1 vs 0) In other words, you can predict whether something is likely to happen or not as opposed to predicting a numeric value. This is a type of analysis that is super useful and practical and real world scenarios where individuals want to know if something is likely to happen or not…it’s a more definitive approach to performing an analysis and helps to answer a different question. I think the most challenging part of logistic regression is understanding the math behind the calculations. I’m looking forward to diving into this in future classes so I can have a more robust understanding in the future.

This week’s lesson was all about text analytics. This was a really interesting and applicable lesson because most people can relate to social media and all the information that exists on the web (and in other various areas). Most of this information is in fact text, which illuminates the importance of this particular topic. R has some robust packages used for text analytics, and I enjoyed pulling trump tweets to understand sentiments and topic modeling. although we only scratched the surface of text analytics, we can really start to uncover the power of insights even by doing something as simple as pulling a document and counting the number of times a word exists. I think the biggest challenge for this going forward will be the level of difficulty and complexity involved in text analytics; but with discovery and constant practice, it’s certainly a topic of high interest to me.

In this week’s lesson, we covered functions and clusters. Over the past few weeks, there have been several instances where we would copy and paste certain chunks of code and re-use them (modifying them slightly). By using functions, we can build our own code specifying the data, columns, and various other inputs to eliminate redundantly copying and pasting code. For example, we wrote a function for proc means and proc freq in R, something that can’t be done in 1 or 2 steps. Now, we ca re-use that function and others in the future to help eliminate unneeded extra work. Lastly, we talked about K Means clustering in R: very similar to SAS, but we used to analyze wine data and explore the different natural groupings of continuous variables within the dataset.

This week, our focus was on regression in R. We’ve been through regression before in SAS and excel, so conceptually, it was mostly review; but when we started to dive into the various capabilities in R such as plotting residuals, fitting the model, etc., we can see that R has more complexities here than SAS, and some of the regression work we did, I still don’t fully understand. Furthermore, the syntax really starts to get tricky as some of the functions seem counter-intuitive to the functions we’ve already seen in R. As usual, I think the key here is practice. There are so many different packages in R which can prove to be an advantage and downfall. Some of the open questions I have is: how can we track packages i.e. when were they last updated? are they no longer active? which package should we choose for different scenarios? and how can we see when new packages are expected to be released?

This week, our focus was on manipulating data in R, that is using joins and other data wrangling techniques to combine and aggregate data in R. To date, this was the most time consuming assignment, mainly due to the the level of joining and aggregating we had to do for each step. However, during this assignment, using the pipes really became second nature to me, only because we’ve done it so much. As I move forward with these assignments, I’m discovering that there are parts about R that I like better than SAS and vice versa. In SAS, I like the convenience of using proc means and proc freq in SAS to be able to quickly gather essential information and make inferences at a high level, but from a data manipulation standpoint, pipes seem far more efficient than creating data step after data step / even using proc SQL. Although R is much more sensitive than SAS and can be much more frustrating at times, I enjoy the flexibility / robustness that it has in terms of creating graphs and manipulating data structures.

This week, we really started to dive into R. R is a very powerful statistical analysis tool, and I think after exploring the intricacies of how it operates, I realized the complexities of what it can do. However, there were some real challenges (and still are some real challenges) to pivoting from SAS and Excel to R. With my experience thus far, I’ve found that the syntax is much more sensitive than SAS. R operates at a much more detailed level than SAS and has thousands of packages. We’ve only scratched the surface of packages in R, but learning how to apply the appropriate packages, incorporate R syntax in loops, data manipulation, etc. has certainly been challenging. The other challenge that I encountered was the graphing component. Graphing, while more advanced and specific than SAS, seems to be much more difficult. For example, creating a simple plot with GGplot 2 took some additional research / trial and error before everything came together.

All that being said, I’m looking forward to the challenge of learning R and becoming adept at performing core statistical analysis functions, as this tool / statistical language is used ubiquitously in the workplace for statistical analysis.

This week, we focused on the midterm project, which covered everything we’ve done up until this point. The objective of this assignment was to take a large data set of property values and leverage other variables to build an optimal regression model. Our first task was all about structuring, exploring and transforming our data. The next step was to perform clusters and identify outliers. Lastly, we used correlation and data step techniques to build our first model, Then we used a more advanced approach to proc reg to include all the variables that had correlation of > .10, and stripped out the remaining variables with p values of > .05 from there.

The challenging part of this assignment was that it encompassed everything we’ve learned from soup to nuts, and remembering the syntax of how to write everything was taxing at times – going back to separate assignments and lectures, etc. The part of that project that perplexed me a little bit was how to appropriately identify and treat outliers to increase the efficacy of the model. e.g. normalizing through logs, then identifying large variances, box plots (IQRs), etc. Lastly, proc GLM Select I found to be a little confusing, mainly because I was used to using proc Reg. I think the lengthy options threw me off.

This week was really a review of everything we’ve learned so far. everything from proc import to proc reg and some of the more advanced regression procedures. In preparation for the midterm, the class used office hours to run through the techniques we would be required to use for the midterm – predicting housing prices.

There are lots of benefits and lessons learned that can be taken away from the first 7 weeks of this course. I think the most satisfying part for me was learning to lean on SAS and capitalizing on its utility versus leaning on Excel (something that I’ve worked with using smaller data sets my entire life). Through repetition and exploration, I finally feel comfortable using SAS software to structure, standardize, clean, analyze, and model data. As far as challenges go, I think the most challenging piece is the syntax part: knowing what you want to do but not knowing how to write the code to do it. Sometimes, it can even be difficult to find the answer through Google. I suppose that this again, just takes time and repetition as well as feedback from those much more familiar with the program.

I’m excited to finish up this mid term with an all inclusive project – one that encompasses and tests all the skills we’ve learned up to this point. I also like the idea of bonus points based on top 3 Kaggle submissions – this gives the class incentive to treat this project as if it were a real scenario or problem. Looking forward to the end results and starting the half of the class in R.

This week, we shifted our focus to regression and prediction, as well as T-tests. We started the lessons with a quick review of all the past procedures we should be familiar with by now, including proc means, proc sort, proc freq, proc rank, proc sgplot, proc fastclus, datastep, etc. We then moved on to proc T test – a test used to compare means of two separate groups e.g. are males taller than females, or does one type of beer contain more alcohol than another. These questions are very prevalent and applicable in the real world, so while this was something I was familiar with from my undergrad days, it was a good refresher. Furthermore it made me realize the simple, but powerful functions of SAS to conduct T tests on data.

The 2nd piece, and really the bulk of our focus this week, was on regression / prediction. We started by reviewing an article by a prediction skeptic, whereby the author talks about the prevalence of bias in everything we do and predict, especially when it comes to data science. he certainly raised some good points that encouraged the class to think critically, but I think more importantly, it informed the class of bias and other inherent flaws when performing data analysis.

When we actually dove into the code, we analyzed wine quality with several contributing variables including sugar, alcohol content, various residuals, density, etc. We then built out a training model in SAS and tested the results using our test set; then, we put the code through a final scoring model and reviewed the results – seeing the analysis come to life and understanding the power of prediction with existing data was probably the most satisfying portion of this week’s assignments.