How to install and configure a standalone TEZ UI with HDP 3.x

Preamble

We recently upgraded our Hadoop platform to HortonWorks Data Platform (HDP) 3.1.4 (latest free available edition) and Ambari 2.7.4. We had tons of issue when doing and even after the migration so I might publish few articles around this painful period. One annoying drawback we had is TEZ UI that has gone in Ambari 2.7.x:

tez-ui01
tez-ui01

I have found many Cloudera and Stackoverflow forum discussion explaining partially what to do but they are most of the time contradictory so decided to try to write one on my own with bare minimum of what is really required.

TEZ UI installation

The official Apache Tez UI installation page steps are very easy I have personally downloaded latest available at the time of writing this article so TEZ UI 0.9.2 at this url: https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-ui/0.9.2/

i have taken tez-ui-0.9.2.war file.

I have installed the Apache of my RedHat release with yum command (start it with systemctl start httpd, ensure firewalld is stopped of well configured):

[root@ambari_server tez-ui]# yum list httpd
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Installed Packages
httpd.x86_64                                                          2.4.6-67.el7_4.6                                         @rhel-7-server-rpms
Available Packages
httpd.x86_64                                                          2.4.6-80.el7                                             Server

Then I have unzipped the tez-ui-0.9.2.war file in /var/www/html/tez-ui directory (/var/www/html being default Apache DocumentRoot directory bu this can be changed in Apache configuration):

[root@ambari_server tez-ui]# pwd
/var/www/html/tez-ui
[root@ambari_server tez-ui]# ll
total 4312
drwxr-xr-x 4 root root     151 Mar 19  2019 assets
drwxr-xr-x 2 root root      25 Feb 21 16:20 config
drwxr-xr-x 2 root root    4096 Mar 19  2019 fonts
-rw-r--r-- 1 root root    1777 Mar 19  2019 index.html
drwxr-xr-x 3 root root      75 Mar 19  2019 META-INF
-rw-r--r-- 1 root root 2200884 Feb 21 16:20 tez-ui-0.9.2.war
drwxr-xr-x 3 root root      36 Mar 19  2019 WEB-INF

Edit /var/www/html/tez-uiconfig/configs.env fiel and change only two parameters (you need to un comment them as well, remove trailing //):

  • timeline must be set to Ambari value of yarn.timeline-service.webapp.address (YARN configuration)
  • rm must be set to Ambari value of yarn.resourcemanager.webapp.address (YARN configuration)

The two above parameter values are like server.domain.com:8188 and server.domain.com:8088. Ensure with a web browser that they return the list of competed and running applications onto your cluster.

Ambari values to copy/paste:

tez-ui02
tez-ui02

And voila installtion done you can access to TEZ UI at this url: http://ambari_server.domain.com/tez-ui/

TEZ UI configuration

Previous part was really straightforward but the Hadoop configuration was quite cumbersome as many articles are saying opposite things…

Some are even quite dangerous with HDP 3.1.4… For example hive.exec.failure.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook, hive.exec.post.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.atlas.hive.hook.HiveHook, hive.exec.pre.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook have corrupted beeline access to databases…

What I have set in Custom tez-site part is this below parameter list:

  • tez.history.logging.service.class = org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService
  • tez.tez-ui.history-url.base = http://ambari_server.domain.com/tez-ui/
  • tez.am.tez-ui.history-url.template = __HISTORY_URL_BASE__?viewPath=/#/tez-app/__APPLICATION_ID__

Those other parameters must also have below values but as those are default should not be an issue:

  • yarn.timeline-service.enabled = true (in YARN and in TEZ configuration)
  • hive_timeline_logging_enabled = true
  • yarn.acl.enable = false

Remark:
If you want to keep yarn.acl.enable = true (which sounds a good idea) you might need to add something to yarn.admin.acl so set it to activity_analyzer,yarn,admin,operator,…. The only issue is that so far I have not found what to add. See next chapter for a trick to solve this issue…

If you have multiple servers in your landscape and your Resource Manager server is not the one running TEZ UI you might need to set as well:

  • yarn.timeline-service.http-cross-origin.allowed-origins = *
  • yarn.timeline-service.hostname = Timeline Service server name (FQDN)

To be able to display queries in Hive Queries panel of TEZ Ui home page I have also set:

  • hive.exec.failure.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook
  • hive.exec.post.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook
  • hive.exec.pre.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook

Then going to http://ambari_server.domain.com/tez-ui/ I was only seeing old queries pre-upgrade. To unlock the situation I have been to YARN resource manager and in on of the running queries I selected ApplicationMaster link:

tez-ui03
tez-ui03

And I got redirected to TEZ UI:

tez-ui04
tez-ui04

And starting from this the TEZ UI home page (http://ambari_server.domain.com/tez-ui/) was correctly displaying running figures.

With the extra hook parameters the Hive Queries panel is also displaying query list:

tez-ui05
tez-ui05

Or more precisely:

tez-ui06
tez-ui06

TEZ UI bug correction trick

If at a point in time you encounter an issue and no figures are displayed you can get some help from the developer tool of your browser (Firefox or Chrome). Here is a screenshot using Firefox:

tez-ui07
tez-ui07

Here using the Network tab and the response from my NameNode I see that http://namenode.domain.com:8188/ws/v1/timeline/TEZ_DAG_ID?limit=11&_=1583236988485 is returning an empty json answer:

{"entities":[]}

So obviously nothing is displayed… This is how I have seen that yarn.acl.enable was playing an important role to display this resource…

I also noticed that trying to access to something like http://namenode.domain.com:8188/ws/v1/timeline/TEZ_DAG_ID?user.name=whatever is resolving the issue for… Well, some time, not even the current session of your web browser… There is something weird here…

I also have to say that the documentation on this part is not very well done. Strange knowing the increasing interest on security nowadays…

References

About Post Author

11 thoughts on “How to install and configure a standalone TEZ UI with HDP 3.x

  1. Hi,

    About this remark below
    #####
    Remark:
    If you want to keep yarn.acl.enable = true (which sounds a good idea) you might need to add something to yarn.admin.acl so set it to activity_analyzer,yarn,admin,operator,…. The only issue is that so far I have not found what to add. See next chapter for a trick to solve this issue…
    #####
    I want to keep “yarn.acl.enable = true” but, tez-ui failed to display. Any suggestion?
    I already used “yarn.admin.acl: “activity_analyzer,yarn,admin,knox,hive”, but no way !

    Thanks

  2. Hi Yannick

    In my case to display “Hive Queries” Panel. this setting make beeline not works:

    hive.exec.failure.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook
    hive.exec.post.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook
    hive.exec.pre.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook

    Instead I have to keep the original hooks, like this:

    hive.exec.failure.hooks = org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook,org.apache.hadoop.hive.ql.hooks.ATSHook
    hive.exec.post.hooks = org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook,org.apache.hadoop.hive.ql.hooks.ATSHook
    hive.exec.pre.hooks = org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook,org.apache.hadoop.hive.ql.hooks.ATSHook

    Beeline works normally but “Hive Queries” Panel display wrong information like this: https://chanthel.solusi247.com/index.php/s/Kbriwa3xPkjNJst/preview

    Any suggestion? Thank you.

    • Hi Kholis,

      Really difficult to say with no access to your cluster…

      Try the trick of my blog post in developer tool to see what’s going on. Have you restarted components ? Sometimes it helps…

      Keeping the two hooks might also cause an issue, try with only one to see if Tez UI works fine or not then focus on Beeline.

      Needless to say that a test platform is mandatory to test all this !

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>