1335228 - ATMO V2: fix log permissions

Status: RESOLVED INVALID

(Cloud Services :: Metrics: Data Tools, defect, P1)

Description

[Sam Penrose]

spark-history-server.out now complains of a permissions failure and doesn't log.

[hadoop@ip-172-31-10-177 ~]$ ls -al /mnt/var/log/spark/
total 12
drwxr-xr-x  2 spark  spark    53 Jan 30 22:59 .
drwxr-xr-x 22 hadoop hadoop 4096 Jan 30 22:58 ..
-rw-r--r--  1 spark  spark  1904 Jan 30 22:59 spark-history-server.out
-rw-rw-r--  1 hadoop hadoop  739 Jan 30 23:04 spark.log

hadoop@ip-172-31-10-177 ~]$ cat /mnt/var/log/spark/spark-history-server.out 
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /mnt/var/log/spark/spark.log (Permission denied)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
	at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
	at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.apache.spark.internal.Logging$class.initializeLogging(Logging.scala:117)
	at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:102)
	at org.apache.spark.deploy.history.HistoryServer$.initializeLogIfNecessary(HistoryServer.scala:239)
	at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
	at org.apache.spark.deploy.history.HistoryServer$.log(HistoryServer.scala:239)
	at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:245)
	at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)

Comment 1

[Sam Penrose]

Jannis could you triage this one or ni? someone who will?

Comment 2

[Jannis Leidel [:jezdez]]

This seems like a simple permissin issue, where Spark tries to write to a file (/mnt/var/log/spark/spark.log) that it can't since it was created by the hadoop user: https://github.com/mozilla/emr-bootstrap-spark/blob/dd44344f4e43e46aedacbc7fbc6782ec2935b95c/ansible/files/telemetry.sh#L115-L118

I've looked through the recent changes to telemetry.sh and couldn't find something glaring, so I'm wondering if it has anything to do with the recent EMR update. Have you by any chance seen this issue with different EMR versions?

Comment 3

[Sam Penrose]

(In reply to Jannis Leidel [:jezdez] from comment #2)
> This seems like a simple permissin issue, where Spark tries to write to a
> file (/mnt/var/log/spark/spark.log) that it can't since it was created by
> the hadoop user:
> https://github.com/mozilla/emr-bootstrap-spark/blob/
> dd44344f4e43e46aedacbc7fbc6782ec2935b95c/ansible/files/telemetry.sh#L115-L118
> 
> I've looked through the recent changes to telemetry.sh and couldn't find
> something glaring, so I'm wondering if it has anything to do with the recent
> EMR update. Have you by any chance seen this issue with different EMR
> versions?

When I start a fresh 5.0.0 cluster the permissions are the same. I do not often look at the log, so I can't say when the issue started.

Comment 4

[Jannis Leidel [:jezdez]]

https://github.com/mozilla/emr-bootstrap-spark/pull/61

Comment 5

[Jannis Leidel [:jezdez]]

Moved to https://github.com/mozilla/telemetry-analysis-service/issues/216