How do you connect to the Spark UI? Is it enabled by Seahorse?


#1

Which service in the dockder-compose.yaml file should have the Spark UI port (4040) mapped to allow outside access to the Spark UI?


#2

SparkUI for first started workflow can be found on http://localhost:4040. Second started workflow is on port 4041.

Spark local is being run in the sessionmanager container. Sessionmanager uses host network driver and exposes all ports on localhost.


#3

For others with this problem: the “http://localhost:4040” approach only works if a) the session manager is running in host network mode, and b) your browser is on the same machine as the session manager. The first issue is due to 4040, 4041, etc. not being EXPOSEd in the session manager Docker image. The second is due to a bug outlined here:

Currently the only known way around b) is to modify seahorse and rebuild (the fix above is unlikely to be the correct longterm solution, but it’s a hack that works).