One of the reasons for this error is lack of disk space on the master node.
Is the disk almost full?
$ hadoop fs -df -h Filesystem Size Used Available Use% hdfs://ip-172-16-1-188.ec2.internal:8020 143.9 G 114.2 G 17.7 G 79%
Which directory is responsible?
In my case it's
$ hadoop fs -du -h /var/log/spark 108.2 G /var/log/spark/apps
This may happen if you run many applications in the same cluster.
Just delete everything under
/var/log/spark/apps/ but not the directory itself:
$ hadoop fs -rm -r -f /var/log/spark/apps/ $ hadoop fs -mkdir /var/log/spark/apps/