Update Java to JDK 8 on Amazon Elastic MapReduce
Last updated:The new emr-4.4.0 label on Amazon Elastic MapReduce now supports Java 8.
If you are running older labels, you'll have to download it yourself (
sudo yum install java-1.8.0 java-1.8.0-openjdk-devel)
First off, make sure your cluster has been created with a private key, otherwise you can't SSH into it:
Once you are in, type java -version to see what you current version is:
[hadoop@ip-000-00-00 ~]$ java -version
java version "1.7.0_95"
OpenJDK Runtime Environment (amzn-2.6.4.0.65.amzn1-x86_64 u95-b00)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
If you see a version starting with 1.7 (as in the example above) it means you're running Java 7.
The solution
To change it to Java 8:
$ sudo alternatives --config java
You will be given an option of what JDK you want. Type 2 to select Java 8 and then hit ENTER.
Checking that it worked
[hadoop@ip-000-00-00 ~]$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
Trobleshooting
If your Spark/Hadoop Jobs don't seem to be picking up the new Java version, you may also need to alter $JAVA_HOME in some configuration files, e.g. Spark's and Hadoop's:
$ echo "export JAVA_HOME=/usr/lib/jvm/java-1.8.0" | sudo tee -a /usr/lib/spark/conf/spark-env.sh
$ echo "export JAVA_HOME=/usr/lib/jvm/java-1.8.0" | sudo tee -a /usr/lib/hadoop/etc/hadoop/hadoop-env.sh