Accessing Wikipedia data on S3
Jump to navigation
Jump to search
There are 2 ways to access the Wikipedia segments:
- By HTTP. Here are the link to 27 files:
- Through Hadoop on EC2. Hadoop supports access to S3 directly, so anyone with an access key and secret key configured in core-sites.xml will be able to access it. For example
bin/hadoop fs -ls s3n://cs9223/enwiki-20121001/
(s3n is the s3 native filesystem)
To access these files from any machine with hadoop installed, open core-site.xml and add the following:
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>ID</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>SECRET</value>
</property>
See http://wiki.apache.org/hadoop/AmazonS3 for more information