Preamble

We recently upgraded our Hadoop platform to HortonWorks Data Platform (HDP) 3.1.4 (latest free available edition) and Ambari 2.7.4. We had tons of issue when doing and even after the migration so I might publish few articles around this painful period. One annoying drawback we had is TEZ UI that has gone in Ambari 2.7.x:

I have found many Cloudera and Stackoverflow forum discussion explaining partially what to do but they are most of the time contradictory so decided to try to write one on my own with bare minimum of what is really required.

TEZ UI installation

The official Apache Tez UI installation page steps are very easy I have personally downloaded latest available at the time of writing this article so TEZ UI 0.9.2 at this url: https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-ui/0.9.2/

i have taken tez-ui-0.9.2.war file.

I have installed the Apache of my RedHat release with yum command (start it with systemctl start httpd, ensure firewalld is stopped of well configured):

[root@ambari_server tez-ui]# yum list httpd
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Installed Packages
httpd.x86_64                                                          2.4.6-67.el7_4.6                                         @rhel-7-server-rpms
Available Packages
httpd.x86_64                                                          2.4.6-80.el7                                             Server

Then I have unzipped the tez-ui-0.9.2.war file in /var/www/html/tez-ui directory (/var/www/html being default Apache DocumentRoot directory bu this can be changed in Apache configuration):

[root@ambari_server tez-ui]# pwd
/var/www/html/tez-ui
[root@ambari_server tez-ui]# ll
total 4312
drwxr-xr-x 4 root root     151 Mar 19  2019 assets
drwxr-xr-x 2 root root      25 Feb 21 16:20 config
drwxr-xr-x 2 root root    4096 Mar 19  2019 fonts
-rw-r--r-- 1 root root    1777 Mar 19  2019 index.html
drwxr-xr-x 3 root root      75 Mar 19  2019 META-INF
-rw-r--r-- 1 root root 2200884 Feb 21 16:20 tez-ui-0.9.2.war
drwxr-xr-x 3 root root      36 Mar 19  2019 WEB-INF

Edit /var/www/html/tez-uiconfig/configs.env fiel and change only two parameters (you need to un comment them as well, remove trailing //):

• timeline must be set to Ambari value of yarn.timeline-service.webapp.address (YARN configuration)
• rm must be set to Ambari value of yarn.resourcemanager.webapp.address (YARN configuration)

The two above parameter values are like server.domain.com:8188 and server.domain.com:8088. Ensure with a web browser that they return the list of competed and running applications onto your cluster.

Ambari values to copy/paste:

And voila installtion done you can access to TEZ UI at this url: http://ambari_server.domain.com/tez-ui/

TEZ UI configuration

Previous part was really straightforward but the Hadoop configuration was quite cumbersome as many articles are saying opposite things…

Some are even quite dangerous with HDP 3.1.4… For example hive.exec.failure.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook, hive.exec.post.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.atlas.hive.hook.HiveHook, hive.exec.pre.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook have corrupted beeline access to databases…

What I have set in Custom tez-site part is this below parameter list:

• tez.history.logging.service.class = org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService
• tez.tez-ui.history-url.base = http://ambari_server.domain.com/tez-ui/
• tez.am.tez-ui.history-url.template = __HISTORY_URL_BASE__?viewPath=/#/tez-app/__APPLICATION_ID__

Those other parameters must also have below values but as those are default should not be an issue:

• yarn.timeline-service.enabled = true (in YARN and in TEZ configuration)
• hive_timeline_logging_enabled = true
• yarn.acl.enable = false

Remark:
If you want to keep yarn.acl.enable = true (which sounds a good idea) you might need to add something to yarn.admin.acl so set it to activity_analyzer,yarn,admin,operator,…. The only issue is that so far I have not found what to add. See next chapter for a trick to solve this issue…

If you have multiple servers in your landscape and your Resource Manager server is not the one running TEZ UI you might need to set as well:

• yarn.timeline-service.http-cross-origin.allowed-origins = *
• yarn.timeline-service.hostname = Timeline Service server name (FQDN)

To be able to display queries in Hive Queries panel of TEZ Ui home page I have also set:

• hive.exec.failure.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook
• hive.exec.post.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook
• hive.exec.pre.hooks = org.apache.hadoop.hive.ql.hooks.ATSHook

Then going to http://ambari_server.domain.com/tez-ui/ I was only seeing old queries pre-upgrade. To unlock the situation I have been to YARN resource manager and in on of the running queries I selected ApplicationMaster link:

And I got redirected to TEZ UI:

And starting from this the TEZ UI home page (http://ambari_server.domain.com/tez-ui/) was correctly displaying running figures.

With the extra hook parameters the Hive Queries panel is also displaying query list:

Or more precisely:

TEZ UI bug correction trick

If at a point in time you encounter an issue and no figures are displayed you can get some help from the developer tool of your browser (Firefox or Chrome). Here is a screenshot using Firefox:

Here using the Network tab and the response from my NameNode I see that http://namenode.domain.com:8188/ws/v1/timeline/TEZ_DAG_ID?limit=11&_=1583236988485 is returning an empty json answer:

{"entities":[]}

So obviously nothing is displayed… This is how I have seen that yarn.acl.enable was playing an important role to display this resource…

I also noticed that trying to access to something like http://namenode.domain.com:8188/ws/v1/timeline/TEZ_DAG_ID?user.name=whatever is resolving the issue for… Well, some time, not even the current session of your web browser… There is something weird here…

I also have to say that the documentation on this part is not very well done. Strange knowing the increasing interest on security nowadays…

References

• Tez UI Overview
• YARN Timeline Background
• The YARN Timeline Service v.2
• The YARN Timeline Server
• Tez History/UI
• ACLs and the Tez UI
• TEZ-UI (self hosted) not showing any jobs in HDP 3.0
• How to install Tez UI Standalone and use it to debug Hive Queries
• can not see any dags in Tez ui
• Configuring Tez

About Post Author

Yannick Jaquier

administrator

Find more about me on social media.

See author's posts