Difference between revisions of "Course: Massive Data Analysis 2014"
Jump to navigation
Jump to search
(→News) |
|||
| Line 162: | Line 162: | ||
**Dynamic Itemset Counting and Implication Rules for Market Basket Data. Brin et al., SIGMOD 1997. http://www-db.stanford.edu/~sergey/dic.html | **Dynamic Itemset Counting and Implication Rules for Market Basket Data. Brin et al., SIGMOD 1997. http://www-db.stanford.edu/~sergey/dic.html | ||
== Week 12 -- Dec 1: | == Week 12 -- Dec 1: Project Updates == | ||
* Lecture notes: | * Lecture notes: | ||
Revision as of 00:51, 2 December 2014
CS-GY 6333 Massive Data Analysis: Tentative Schedule -- subject to change
- Course Web page: http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/
- Instructor: Professor Juliana Freire (http://vgc.poly.edu/~juliana/)
- Lecture: Mondays, 1:00pm-3:25pm at 2MTC, room 9.011.
News
- Massive Data Analysis 2014: Class project
- Aditi Nakta, our TA, will hold office hours on Tuesdays from 1 - 3 pm @ 2 MTC room 10.98D
- Your Gradiance assignment on MapReduce has been posted: http://www.newgradiance.com/services. If you haven't registered yet, do so and use the class token 1AEF5F24. Make sure to use your official NYU email and id when you register.
- On Sept 22nd, I distributed AWS tokens that will be needed for your assignments. If you have not received your token, let me know.
- Your first assignment has been posted -- see details below and in NYU Classes.
- Instructions on how to set up your AWS account: http://www.vistrails.org/index.php/AWS_Setup
- You should get an NYU HPC account so that you can use the NYU Hadoop cluster. To submit a request for an account, follow the instructions in: https://wikis.nyu.edu/display/NYUHPC/HPC+at+NYU+-+Access. You can find instructions on how to login and use the NYU Hadoop cluster at: http://vgc.poly.edu/~juliana/courses/BigData2014/Lectures/MapReduceExample/readme-nyu-hadoop.txt
Background (4 weeks)
Week 1 -- Sept 8: Course Overview; the evolution of Data Management
- Lecture notes: http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Lectures/course-overview.pdf (http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Lectures/course-overview-6p.pdf)
- Reading: Chapter 1 of Mining of Massive Data Sets (version 1.1)
- Course survey: https://docs.google.com/spreadsheet/embeddedform?formkey=dFpwTjROVzhLUWY2NVNXb0xvNTVLMnc6MA
Week 2 -- Sept 15: Provenance and Reproducibility
- Lecture notes: http://vgc.poly.edu/~fchirigati/mda-class/provenance-reproducibility.pdf
- The class will have a lab component. Please bring your laptops.
- Before class, follow the instructions below to install and set up VisTrails as well as github
- VisTrails setup:
- Download VisTrails 2.1.4 from http://www.vistrails.org/index.php/Downloads and follow the installation instructions. Start the system and then quit.
- Download the following packages:
- After you extract the content of the zip files, place them under $HOME/.vistrails/userpackages
- Github setup:
- Create a github account (https://github.com/join)
- Learn how to set up git and create a public repository.
- During class, you will add the trail of your analysis to github, and submit the link to your public github repo using this form: https://docs.google.com/forms/d/17OScN8Ea-El20AC4mHIb32S3e62mAbGEiU-BET0PyX8/viewform?usp=send_form
Week 3 -- Sept 22: Introduction to Databases; Relational Model and SQL
- Lecture notes:
- http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Lectures/intro-to-db.pdf
- http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Lectures/relational-algebra.pdf
- http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Lectures/sql-intro.pdf
- http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Lectures/sql-more.pdf
- Other useful reading:
Week 4 -- Sept 29: Overview: Advanced SQL and Query Optimization
- Lecture notes:
- In-class exercise: http://vistrails.org/index.php/Big_Data_Lab_SQL
Big Data Foundations and Infrastructure (3 weeks)
Week 5 -- Oct 6: Cloud computing, Map Reduce and Hadoop
- Lab: after the lecture, you will work on an in-class exercise. For this you need to install Hadoop on your laptop and have your account setup on AWS. See instructions below.
- You will use two different Hadoop configurations:
- Local (on your laptop)
- Amazon AWS: Each student should have received a token with $100 credit towards computing time at AWS. If you have not received the token yet, contact us immediately! When using AWS, always remember to terminate your instances! If you don't, you will be charged and you are responsible for the charges beyond your credit.
- See installation instructions for Hadoop on your local machine and how to setup your AWS account in http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Lectures/HadoopExerciseInstructions.pdf
- Warning: Install Hadoop in your machine and setup your AWS account before class starts. There will be no time for installing software during our in-class exercise.
- In-Class Exercise: Hadoop Exercise
- Required reading:
- Data-Intensive Text Processing with MapReduce, Chapters 1 and 2
- Mining of Massive Datasets (2nd Edition), Chapter 2 - 2.1 and 2.2 (Large-Scale File Systems and Map-Reduce).
- Other useful reading:
- Hadoop: The Definitive Guide. http://www.amazon.com/Hadoop-Definitive-Guide-Tom-White/dp/1449311520
Week 6 -- Oct 13: Fall Break
Week 7 -- Oct 20: Big Data Analysis with Myria
- Lecture notes:
- Useful reading:
Week 7 -- Oct 27: Algorithm Design for MapReduce
- Lecture notes:
- Required reading:
- Data-Intensive Text Processing with MapReduce, Chapters 1 and 2
- Mining of Massive Datasets (2nd Edition), Chapter 2.
Week 8 -- Nov 3: Parallel Databases vs MapReduce, Query Processing on Mapreduce and High-level Languages
- Lecture notes:
- Discussion about project
- Assignment: check Gradiance!
- Required reading:
- Data-Intensive Text Processing with MapReduce (Jan 27, 2013), Chapter 6 -- Processing Relational Data (this chapter appears in the 2013 version of the textbook -- I have placed this version in http://vgc.poly.edu/~juliana/courses/MassiveDataAnalysis2014/Textbooks/MapReduce-algorithms-Jan2013-draft.pdf)
- Benchmark DBMS vs MapReduce (2009): http://database.cs.brown.edu/sigmod09/benchmarks-sigmod09.pdf
- MapReduce: A Flexible Data Processing Tool: http://cacm.acm.org/magazines/2010/1/55744-mapreduce-a-flexible-data-processing-tool/fulltext
- Additional reading:
- Pig Latin: A Not-So-Foreign Language for Data Processing: http://pages.cs.brandeis.edu/~olga/cs228/Reading%20List_files/piglatin.pdf
- Hive - A Warehousing Solution Over a Map-Reduce Framework: http://www.vldb.org/pvldb/2/vldb09-938.pdf
Big Data Algorithms, Techniques, and Visualization (3 weeks)
Week 9 -- Nov 10: Visualization and Big Data -- Invited lecture by Dr. Huy Vo (NYU CUSP)
- Lecture notes:
Week 10 -- Nov 17: Visualization Techniques -- Invited lecture by Dr. Lauro Lins (AT&T Research)
- Project status report due!
- Lecture notes:
- Reading:
- Nanocubes for real-time exploration of spatiotemporal datasets. Lins et al. http://nanocubes.net/assets/pdf/nanocubes_paper.pdf
Week 11 -- Nov 25 Association Rules
- Lecture notes:
- Assignment: Check http://www.newgradiance.com/services
- Reading: Chapter 6 Mining of Massive Datasets
- Suggested additional reading:
- Fast algorithms for mining association rules, Agrawal and Srikant, VLDB 1994.
- Data Mining Concepts and Techniques, Jiawei Han and Micheline Kamber, Morgan Kaufmann
- Dynamic Itemset Counting and Implication Rules for Market Basket Data. Brin et al., SIGMOD 1997. http://www-db.stanford.edu/~sergey/dic.html
Week 12 -- Dec 1: Project Updates
- Reading: Chapter 3 Mining of Massive Datasets
Week 13 -- Dec 8: Graph Analysis
- Lecture notes: