Software engineer's guide to getting started with data science


Dec 31, 2012

Software engineer's guide to getting started with data science


Many of my software engineer friends ask me about learning data science. There are many articles on this subject from renowned data scientists (Dataspora, Gigaom, Quora, Hilary Mason). This post captures my journey (a software engineer) on learning Statistics and Data Visualization.

I'm mid-way in my 5 year journey to become proficient in data science and my learning program has included self-learning (books, blogs, toy problems), projects at work, class-room training (Stanford), teaching/presentations, conferences (UseR, Strata). Here's what I've done so far and what worked and what didn't...


1. GETTING STARTED

a) Self-learning (2 - 4 months)
Explore if data science is for you

This is the key to getting started. Two years ago some of us at work formed a study group to review Stats 202 class material. This is what got me excited and started with data analytics. Only 2 of the 5 members of our study group chose to dive deeper into this field (data science is not for everyone).
Learn basic statistics: Stats 202 coursework is perfect for this
Learn a statistical tool: I spent 3 months heads-down learning R as a new-bee and had the most fun doing so. Why learn R?
Solve toy problems: Curiosity is key to data science. If you've questions about your country's economy, crime stats, sports performance, get the data and start answering your questions
Learn Unix tools: I picked O'Reilly's Data Analysis with Open Source Tools (A hands-on guide for programmers and data scientists) book to read.
Learn SQL and scripting languages: I know Java, Ruby and SQL. Python is on my list.

There's a lot of training material available online
Stats 202
Caltech Data Science course
Coursera: Introduction to Data Science, Machine learning, Data Analysis, Computing for Data Analysis
University of California Berkeley - Introduction to Data Science
Knight Center for Journalism's course on Introduction to Infographics and Data Visualization
Stats 101: Udacity (Intro to Stats), Khan academy, Carnegie Mellon's stats course
Learn R

b) Class-room training (9 - 12 months)
If you're serious about learning, enroll into a formal program

If you're serious about picking this skill, then opt for a course. The rigor of the class ensured that I didn't slack. Stanford offers great coursework to get started. They are far superior compared to many week-long training courses I've been to...
Data Mining and Analysis STATS202
Linear and Nonlinear Optimization MS&E211
Mining Massive Data Sets CS246
Modern Applied Statistics: Learning STATS315A
Statistical Methods in Finance STATS240P
Modern Applied Statistics: Data Mining STATS315B


2. GETTING FOCUSED

a) Spend 100% of my time on data science
Once I was hooked on data science, it was difficult to spend only 20% of my time on it to build expertise. I needed to spend 100% of my time on it, so I found work problems related to data science (big data analysis, healthcare, marketing & sales and retail analytics, optimization problems). 

b) Work on interesting problems
I aligned my learning goal with my passion. I found it energizing and engaging to solve interesting problems while learning new techniques. I was interested in retail, healthcare and sports (cricket) data analysis. 

c) Accelerate learning: 
Teach: I taught R and data mining introductory classes to colleagues and friends. This helped me reinforce my learning and get others excited on this topic. This is also a great way for me to give back to the open source community. Blogging is another medium to contribute and learn
Follow the leaders in data science and network with data scientists: DJ Patil, Hillary Mason, Jeff Hammerbacher, Carla Gentry, Monica Rogati, Cathy O'Neil. There are many others in this space. Apologies for missing out many of them. These are the people I look up to.
Follow interesting blogs: http://datascience101.wordpress.com, http://columbiadatascience.com/blog, http://www.r-bloggers.com, http://www.datawrangling.com, http://flowingdata.com (Quora's best data blog list)
Attend conferences/meetups periodically: Local data science/R meetups, O'Reilly Strata is great! Given how rapidly this field is evolving, I go there at least every other year. UseR is wonderful to see what's happening in the world of R
Learn Big Data techniques: MapReduce/Hadoop, Cloud computing. I avoided picking any commercial, vendor technology and in retrospect, it was a good decision.

d) Learn business domains
I'm lucky to have access to internal and external experts in data science, and they've helped me understand their approach to data science problems (how they think, hypothesize and test/assess/reject solutions). I've learned from them the importance of "Hypothesis-driven data analysis" rather than "blind/brute-force data analysis". This highlighted the importance of understanding the business domains really well before trying to extract meaningful insights from the data. This led me to understand operations research and marketing topics, retail, travel & logistics (revenue management) and healthcare industries. NY Times recently published an article highlighting the need for intuition.



3. DATA SCIENCE BOOKS I FOUND USEFUL
Introduction to Data Mining by Tan, Steinback and Kumar This is the textbook used in many introductory data science courses, including Stats 202 at Stanford. Great guide to keep handy
R in a nutshell
Data Analysis by using Open Source tools
Beautiful visualization
See more books on data science: O'Reilly, Manning



4. WHAT DIDN'T WORK FOR ME
Learning multiple Statistical tools: A year ago, I started getting some work requests for SAS programming, so I wanted to learn it. I tried to learn it for a month or so but could not do it. The main reason was learning inertia and my love for the statistical tool I knew already - R. I really didn't need another statistical tool. I could solve most of my data science problems with R and other software tools I knew. So my advice is that if you already know SAS, Stata, Matlab, SPSS, Statistica very well, stick to it. However if you're learning a new statistical tool, pick R. R is open source while most others are commercial software (expensive and complex).
Auditing courses: I tried to follow self-paced coursework from Coursera and other MOOCs but it wasn't effective for me. I needed the routine, the pressure of a formal course with proper grading to go through the rigor
Increasing academic workload: Manage work-life balance and work-commitments well. Earlier this year, I tried to take multiple difficult courses at the same time and quickly realized that I wasn't enjoying and learning as I should.
Sticking to course text book only: Many of the books in these classes are too "dense" for me (a software engineer). So I used other material to understand the concepts. E.g. regression from Carnegie Mellon notes