Difference between revisions of "Accessing Wikipedia data on S3"
Jump to navigation
Jump to search
| Line 71: | Line 71: | ||
<value>ID</value> | <value>ID</value> | ||
</property> | </property> | ||
<property> | <property> | ||
<name>fs.s3.awsSecretAccessKey</name> | <name>fs.s3.awsSecretAccessKey</name> | ||
Revision as of 19:17, 12 November 2012
There are 2 ways to access the Wikipedia segments:
- By HTTP. Here are the link to 27 files:
- Through Hadoop on EC2. Hadoop supports access to S3 directly, so anyone with an access key and secret key configured in core-sites.xml will be able to access it. For example
bin/hadoop fs -ls s3n://cs9223/enwiki-20121001/
(s3n is the s3 native filesystem)
To access these files from any machine with hadoop installed, open core-site.xml and add the following:
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>ID</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>SECRET</value>
</property>
See http://wiki.apache.org/hadoop/AmazonS3 for more information