Difference between revisions of "Course: Big Data Analysis"
Jump to navigation
Jump to search
| Line 157: | Line 157: | ||
Your assignment is in http://www.newgradiance.com/services. Please see http://vgc.poly.edu/~juliana/courses/cs9223 for instructions on how to access this service. | Your assignment is in http://www.newgradiance.com/services. Please see http://vgc.poly.edu/~juliana/courses/cs9223 for instructions on how to access this service. | ||
== Week 11: Monday Nov 18th- | == Week 11: Monday Nov 18th- MapReduce Algorithm Design and Graph Processing == | ||
** Lecture notes: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/ | ** Lecture notes: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/mapreduce-indexing-graph.pdf | ||
=== Homework Assignment === | |||
Your Mapreduce/Pig assignment is available from Blackboard. '''It is Due December 1st'''. | |||
=== Required Reading === | === Required Reading === | ||
* Mining of Massive Datasets, Chapter | * Chapter 5, Data-Intensive Text Processing with MapReduce | ||
=== Additional Reading === | |||
* [http://infolab.stanford.edu/pub/papers/google.pdf 1998 PageRank Paper] | |||
* [http://lintool.github.com/MapReduceAlgorithms/MapReduce-book-final.pdf Data-Intensive Text Processing with MapReduce, Chapter 4 (Inverted Indexing for Text Retrieval) and 5(Graph Algorithms)] | |||
* [http://infolab.stanford.edu/~ullman/mmds/ch5.pdf Mining of Massive Datasets, Chapter 5 (Link Analysis)] | |||
* Pregel: A System for Large-Scale Graph Processing. Google. [http://kowshik.github.com/JPregel/pregel_paper.pdf] | |||
== | == Week 12: Monday Nov. 25th - Large-Scale Visualization == | ||
* Invited lectures by: | |||
** Dr. Lauro Lins (AT&T Research) | |||
** Dr. Huy Vo (NYU Center for Urban Science and Progress) | |||
* Lecture notes: | * Lecture notes: | ||
** | ** https://www.dropbox.com/s/7t2vqryj5zgs44n/intro-to-visualization.pdf | ||
** | ** https://www.dropbox.com/s/btb3ocupkmpgefi/nanocubes.pdf | ||
=== Required Reading === | |||
The Value of Visualization, Jarke Van Wijk | |||
http://www.win.tue.nl/~vanwijk/vov.pdf | |||
Tamara Munzner's Book draft 2 available online | |||
http://www.cs.ubc.ca/~tmm/courses/533/book/ | |||
Nanocubes Paper | |||
http://nanocubes.net | |||
http://nanocubes.net/assets/pdf/nanocubes_paper_preprint.pdf | |||
== | === Additional Reading === | ||
imMens Paper (to contrast with nanocubes) | |||
http://vis.stanford.edu/papers/immens | |||
== Week 13: Monday Dec. 2nd - | == Week 13: Monday Dec. 2nd - Frequent Itemsets == | ||
** Lecture notes: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/assoc-rules1.pdf, http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/assoc-rules1.pdf | |||
** Lecture notes: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/ | |||
=== | === Additional Reading === | ||
* Mining association rules between sets of items in large databases. Agrawal et al., SIGMOD 1993. http://delivery.acm.org/10.1145/180000/170072/p207-agrawal.pdf?ip=128.238.251.32&acc=ACTIVE%20SERVICE&CFID=198467341&CFTOKEN=23537886&__acm__=1352747519_b80a516e0f5e294b36dc021f13f55bbb | |||
* Fast algorithms for mining association rules. Agrawal and Srikant, VLDB 1994. https://www.seas.upenn.edu/~jstoy/cis650/papers/Apriori.pdf | |||
* An effective hash-based algorithm for mining association rules. Park et al., SIGMOD 1995. http://dl.acm.org/citation.cfm?id=223813 | |||
=== Homework Assignment === | |||
'''Due Dec 8th''' | |||
== Week 14: Monday Dec. 9th - - | == Week 14: Monday Dec. 9th - - Clustering == | ||
* Lecture notes: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/clustering.pdf, http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/form-clustering-icde2007.pdf | |||
=== Readings === | === Readings === | ||
* | * Chapter 7 (Clustering), Mining of Massive Data Sets | ||
=== Homework Assignment === | |||
'''Due Dec 15th''' | |||
== Week 15 Monday Dec. 16th - Final Exam == | == Week 15 Monday Dec. 16th - Final Exam == | ||
| Line 222: | Line 233: | ||
* [http://vgc.poly.edu/~juliana/pub/freire-cise2008.pdf Provenance for Computational Tasks: A Survey] | * [http://vgc.poly.edu/~juliana/pub/freire-cise2008.pdf Provenance for Computational Tasks: A Survey] | ||
Juliana Freire, David Koop, Emanuele Santos, and Claudio T. Silva. In IEEE Computing in Science & Engineering, 2008. | Juliana Freire, David Koop, Emanuele Santos, and Claudio T. Silva. In IEEE Computing in Science & Engineering, 2008. | ||
Revision as of 15:21, 22 November 2013
Fall 2013
This schedule is tentative and subject to change
Make sure to check my.poly.edu for course announcements
News
- Nov 7th: New quizzes have been assigned. Please see http://www.newgradiance.com/services/servlet/COTC
The deadline is Nov 15th. Please make sure that you have your correct name and Poly ID in your Gradiance account.
- Dr. C Mohan's presentation is now available at http://bit.ly/CMnMDS
For frequently asked questions about the course and homework assignments, please check our BigDataAnalysisFAQ.
Week 1: Monday Sept. 9th - Course Overview
- Course overview and introduction to Big Data Analysis
- Lecture notes: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/intro.pdf
- Student survey -- to be filled out today!
Required Reading
Additional References
- Dilbert's BigData
- New York Time's "How BigData Became so Big"
- World Economic Forum: Big Data, Big Impact
- The Analytics Journey
- BigData Analytics Usecases
Week 2: Monday Sept. 16th - Map-Reduce/Hadoop
- Introduction to Map-Reduce and high-level data processing languages
- Lecture notes: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/hadoop.pdf
- Hand out AWS tokens. Notes on using AWS.
- Introduction to Hadoop
- The Map-Reduce ecosystem: Pig, Hive, Jaql, Mahout, BigInsights
Assignment
- cs9223 Mapreduce Assignment
- This is an individual assignment. You may not collude with any other individual, or plagiarise their work.
For more details see http://cis.poly.edu/policies.
- You assignment is due on Sun Sept 29th. Make sure you can login and access my.poly.edu!
- If you have questions about the assignment, we will hold office hours on Sept 23, 2013 from 2:30-3:30pm at 2 Metrotech, room 10.018
Required Reading
- Mining of Massive Datasets, Chapter 2
- Data-Intensive Text Processing with MapReduce, Chapter 2 and Chapter 3
- original google map-reduce paper
Week 3: Monday Sept. 23rd - Data Management for Big Data
- Databases and Big Data: Persistence, Querying, Indexing, Transactions
- Lecture notes: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/paralleldb-vs-hadoop.pdf
Related Topics
- BigTables and NoSQL stores. Tuple store vs. column stores: HBase, MongoDB, Cassandra
- HBase book HBase: The Definitive Guide. Random Access to Your Planet-Size Data: http://shop.oreilly.com/product/0636920014348.do
- HBase book. Chapter 8 Architecture for information about transactional processing, WriteAhead Log notably, and how consistency is being maintained.
- Transactions in NoSQL stores. Google's percolator, [1].
- "NewSQL" stores: more on Hive, VoltDB, HadoopDB,
- Beyond MapReduce: Berkeley's Spark, UC Irvine's Asterix, Google's Dremel
Required Reading
- PDMBS vs. MapReduce
- http://cacm.acm.org/magazines/2010/1/55744-mapreduce-a-flexible-data-processing-tool/fulltext
- Parallel data processing with MapReduce: a survey. Lee et al, SIGMOD Record 2011
- Benchmark DBMS vs MapReduce (2009)
Additional References
- http://www.computerworld.com/s/article/9224180/What_s_the_big_deal_about_Hadoop_
- Bigtable: A Distributed Storage System for Structured Data
- HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads
- Low Overhead Concurrency Control for Partitioned Main Memory Databases
- ASTERIX: Towards a Scalable, Semistructured Data Platform for Evolving-World Models.
- Dremel: Interactive Analysis of Web-Scale Datasets
- Large-scale Incremental Processing Using Distributed Transactions and Notifications
Week 4: Monday Sept 30th - Invited lecture by Dr. C. Mohan (IBM)
- Note that we will meet at a different location: NYU CUSP, 1 Metrotech Center, 19th floor
- Tutorial: An In-Depth Look at Modern Database Systems: http://bit.ly/CMnMDS
- Abstract: This tutorial is targeted at a broad set of database systems and applications people. It is intended to let the attendees better appreciate what is really behind the covers of many of the modern database systems (e.g., NoSQL and NewSQL systems), going beyond the hype associated with these open source and commercial systems. The capabilities and limitations of such systems will be addressed. Modern extensions to decades old relational DBMSs will also be described. Some application case studies will also be presented. An outline of problems for which no adequate solutions exist will be included. Such problems could be fertile grounds for new research work.
- Presenter: Dr. C. Mohan, IBM Fellow, IBM Almaden Research Center, San Jose, CA 95120, USA.
- Bio: Dr. C. Mohan has been an IBM researcher for 31 years in the information management area, impacting numerous IBM and non-IBM products, the research community and standards, especially with his invention of the ARIES family of locking and recovery algorithms, and the Presumed Abort commit protocol. This IBM, ACM and IEEE Fellow has also served as the IBM India Chief Scientist. In addition to receiving the ACM SIGMOD Innovation Award, the VLDB 10 Year Best Paper Award and numerous IBM awards, he has been elected to the US and Indian National Academies of Engineering, and has been named an IBM Master Inventor. This distinguished alumnus of IIT Madras received his PhD at the University of Texas at Austin. He is an inventor of 38 patents. He serves on the advisory board of IEEE Spectrum and on the IBM Software Group Architecture Board’s Council. More information can be found in his home page at http://bit.ly/CMohan
Week 5: Monday Oct. 7th - Query Processing on Mapreduce and High-level Languages
- Pig Latin and Query Processing:
- In-class assignment
Required Reading
Additional References
- Jaql: A Scripting Language for Large Scale Semistructured Data Analysis
- Hive - A Warehousing Solution Over a Map-Reduce Framework
Week 6: Mon Oct. 14th - Fall Break - No class
Week 6: Wed Oct. 16th - Fall Break - Make-up class
- Reproducibility and Data Exploration: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/reproducibility.pdf
- Large-scale information integration: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/web-information-integration.pdf
Week 7: Monday Oct. 21st - Invited Speaker: Alberto Lerner
- Inside MongoDB
Week 8: Monday Oct 28th- Statistics is easy - Invited Speaker: Dennis Shasha
- Guest lecture by Dennis Shasha: Statistics is Easy
- Introduction to Provenance
Required Reading
- http://www.morganclaypool.com/doi/abs/10.2200/S00142ED1V01Y200807MAS001 -- book is available for free for NYU students
- Second edition of the book: http://www.morganclaypool.com/doi/pdf/10.2200/S00295ED1V01Y201009MAS008
- We will cover the material planned for "Week 10: Monday Nov. 11th": Finding Similar Items
Week 9: Monday Nov. 4th - Finding Similar Items, Information Integration
- Similarity: Applications, Measures and Efficiency considerations
- Similarity application: Information integration on the Web:
- Homework presentation and demo
Required Reading
Homework Assignment
Due Nov 15th, 2013 Your assignment is in http://www.newgradiance.com/services. Please see http://vgc.poly.edu/~juliana/courses/cs9223 for instructions on how to access this service.
Week 10: Monday Nov. 11th - MapReduce Algorithm Design
Required Reading
- Chapters 3 and 4 in textbook: Data-Intensive Text Processing with MapReduce, by Lin and Dyer
Homework Assignment
Due Nov 15th, 2013 Your assignment is in http://www.newgradiance.com/services. Please see http://vgc.poly.edu/~juliana/courses/cs9223 for instructions on how to access this service.
Week 11: Monday Nov 18th- MapReduce Algorithm Design and Graph Processing
Homework Assignment
Your Mapreduce/Pig assignment is available from Blackboard. It is Due December 1st.
Required Reading
- Chapter 5, Data-Intensive Text Processing with MapReduce
Additional Reading
- 1998 PageRank Paper
- Data-Intensive Text Processing with MapReduce, Chapter 4 (Inverted Indexing for Text Retrieval) and 5(Graph Algorithms)
- Mining of Massive Datasets, Chapter 5 (Link Analysis)
- Pregel: A System for Large-Scale Graph Processing. Google. [2]
Week 12: Monday Nov. 25th - Large-Scale Visualization
- Invited lectures by:
- Dr. Lauro Lins (AT&T Research)
- Dr. Huy Vo (NYU Center for Urban Science and Progress)
- Lecture notes:
Required Reading
The Value of Visualization, Jarke Van Wijk http://www.win.tue.nl/~vanwijk/vov.pdf
Tamara Munzner's Book draft 2 available online http://www.cs.ubc.ca/~tmm/courses/533/book/
Nanocubes Paper http://nanocubes.net http://nanocubes.net/assets/pdf/nanocubes_paper_preprint.pdf
Additional Reading
imMens Paper (to contrast with nanocubes) http://vis.stanford.edu/papers/immens
Week 13: Monday Dec. 2nd - Frequent Itemsets
Additional Reading
- Mining association rules between sets of items in large databases. Agrawal et al., SIGMOD 1993. http://delivery.acm.org/10.1145/180000/170072/p207-agrawal.pdf?ip=128.238.251.32&acc=ACTIVE%20SERVICE&CFID=198467341&CFTOKEN=23537886&__acm__=1352747519_b80a516e0f5e294b36dc021f13f55bbb
- Fast algorithms for mining association rules. Agrawal and Srikant, VLDB 1994. https://www.seas.upenn.edu/~jstoy/cis650/papers/Apriori.pdf
- An effective hash-based algorithm for mining association rules. Park et al., SIGMOD 1995. http://dl.acm.org/citation.cfm?id=223813
Homework Assignment
Due Dec 8th
Week 14: Monday Dec. 9th - - Clustering
- Lecture notes: http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/clustering.pdf, http://vgc.poly.edu/~juliana/courses/cs9223/Lectures/form-clustering-icde2007.pdf
Readings
- Chapter 7 (Clustering), Mining of Massive Data Sets
Homework Assignment
Due Dec 15th
Week 15 Monday Dec. 16th - Final Exam
Other topics
Provenance
Juliana Freire and Claudio Silva. In Computing in Science and Engineering 14(4): 18-25, 2012.
Juliana Freire, David Koop, Emanuele Santos, and Claudio T. Silva. In IEEE Computing in Science & Engineering, 2008.