Table of contents
Preamble
Straight from the beginning when I have landed to our Hadoop project (HortonWorks) I have seen all our Python scripts directly connecting to our Hive server, with PyHive, bypassing Zookeeper coordination. I also noticed that every Beeline client connection where well using (obviously) the HiveServer2 JDBC URL. I have left this point open for later until we decided to improve our High Availability (HA) by making few components running on multiple servers. And when it came to our HiveServer2 that is now running on two edge nodes of our Hadoop cluster I have decided to dig out this thread…
The good way of connecting to HiveServer2 is to first get current status and configuration from Zookeeper and then use this information in PyHive (for example) to make a Hive connection. Zookeeper is acting here as a configuration keeper as well as an availability watcher, means Zookeeper will not return a dead HiveServer2 information.
Digging a bit on Internet I came quickly to the obvious conclusion that Kazoo Python package was a must try !
This blog post has been written using kazoo 2.5.0, Python 3.7.3. My Hadoop cluster is HortonWorks Data Platform (HDP) 2.6.4. All developed scripts are running on a Fedora 30 virtual machine.
Kazoo development environment installation
Anaconda is the preferred Python 3.7 environment management so started by downloading and installing it for Python 3.7 on my Fedora virtual machine. The release I have installed is:
[root@fedora1 ~]# anaconda --version anaconda 30.25.6 |
It also gives you access to conda that is command line environment management:
(base) [root@fedora1 ~]# conda --version conda 4.6.14 (base) [root@fedora1 ~]# conda info active environment : base active env location : /opt/anaconda3 shell level : 1 user config file : /root/.condarc populated config files : /root/.condarc conda version : 4.6.14 conda-build version : 3.17.8 python version : 3.7.3.final.0 base environment : /opt/anaconda3 (writable) channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/linux-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /opt/anaconda3/pkgs /root/.conda/pkgs envs directories : /opt/anaconda3/envs /root/.conda/envs platform : linux-64 user-agent : conda/4.6.14 requests/2.21.0 CPython/3.7.3 Linux/5.0.11-300.fc30.x86_64 fedora/30 glibc/2.29 UID:GID : 0:0 netrc file : None offline mode : False |
As I am behind a corporate proxy I had to customize a little bit my .condarc profile:
(base) [root@fedora1 ~]# cat .condarc ssl_verify: False proxy_servers: http: http://proxy_user:proxy_password@proxy_server:proxy_port https: https://proxy_user:proxy_password@proxy_server:proxy_port |
Remark
I have also been obliged to set ssl_verify to false to avoid any certificate issues that are not in my proxy server…
I create a kazoo conda environment with:
(base) [root@fedora1 ~]# conda create -n kazoo Collecting package metadata: done Solving environment: done ## Package Plan ## environment location: /opt/anaconda3/envs/kazoo Proceed ([y]/n)? y Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate kazoo # # To deactivate an active environment, use # # $ conda deactivate |
Activate it with:
[root@fedora1 ~]# conda activate kazoo (kazoo) [root@fedora1 ~]# |
As you have seen I am working with root which is a very bad idea so better to use a normal account (and keep root for more important tasks see below), to do so initialize Conda with (my shell is obviously bash):
[yjaquier@fedora1 ~]$ /opt/anaconda3/bin/conda init bash no change /opt/anaconda3/condabin/conda no change /opt/anaconda3/bin/conda no change /opt/anaconda3/bin/conda-env no change /opt/anaconda3/bin/activate no change /opt/anaconda3/bin/deactivate no change /opt/anaconda3/etc/profile.d/conda.sh no change /opt/anaconda3/etc/fish/conf.d/conda.fish no change /opt/anaconda3/shell/condabin/Conda.psm1 no change /opt/anaconda3/shell/condabin/conda-hook.ps1 no change /opt/anaconda3/lib/python3.7/site-packages/xonsh/conda.xsh no change /opt/anaconda3/etc/profile.d/conda.csh modified /home/yjaquier/.bashrc ==> For changes to take effect, close and re-open your current shell. <== |
Logoff and logon again and use the newly created environment with:
(base) [yjaquier@fedora1 ~]$ conda activate kazoo (kazoo) [yjaquier@fedora1 ~]$ |
You also need to configure your conda environment (.condarc) same as above…
For package management and search your reference will be https://anaconda.org. Here is an example of a search for Python (direct link is https://anaconda.org/search?q=python):
If you enter in the Python package most downloaded (good practice in my opinion) you will find the command to install it:
conda install -c conda-forge python |
But then you cannot modify the environment with your own account and packages add must be done by root, which is, in my opinion, a very good practice:
EnvironmentNotWritableError: The current user does not have write permissions to the target environment. environment location: /opt/anaconda3/envs/kazoo uid: 1000 gid: 100 |
So executing with root account (in kazoo conda environment):
(kazoo) [root@fedora1 ~]# conda install -c conda-forge python Collecting package metadata: done Solving environment: done ## Package Plan ## environment location: /opt/anaconda3/envs/kazoo added / updated specs: - python The following packages will be downloaded: package | build ---------------------------|----------------- bzip2-1.0.6 | h14c3975_1002 415 KB conda-forge certifi-2019.3.9 | py37_0 149 KB conda-forge pip-19.1 | py37_0 1.8 MB conda-forge python-3.7.3 | h5b0a415_0 35.7 MB conda-forge setuptools-41.0.1 | py37_0 616 KB conda-forge wheel-0.33.1 | py37_0 34 KB conda-forge ------------------------------------------------------------ Total: 38.7 MB The following NEW packages will be INSTALLED: bzip2 conda-forge/linux-64::bzip2-1.0.6-h14c3975_1002 ca-certificates conda-forge/linux-64::ca-certificates-2019.3.9-hecc5488_0 certifi conda-forge/linux-64::certifi-2019.3.9-py37_0 libffi conda-forge/linux-64::libffi-3.2.1-he1b5a44_1006 libgcc-ng pkgs/main/linux-64::libgcc-ng-8.2.0-hdf63c60_1 libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-8.2.0-hdf63c60_1 ncurses conda-forge/linux-64::ncurses-6.1-hf484d3e_1002 openssl conda-forge/linux-64::openssl-1.1.1b-h14c3975_1 pip conda-forge/linux-64::pip-19.1-py37_0 python conda-forge/linux-64::python-3.7.3-h5b0a415_0 readline conda-forge/linux-64::readline-7.0-hf8c457e_1001 setuptools conda-forge/linux-64::setuptools-41.0.1-py37_0 sqlite conda-forge/linux-64::sqlite-3.26.0-h67949de_1001 tk conda-forge/linux-64::tk-8.6.9-h84994c4_1001 wheel conda-forge/linux-64::wheel-0.33.1-py37_0 xz conda-forge/linux-64::xz-5.2.4-h14c3975_1001 zlib conda-forge/linux-64::zlib-1.2.11-h14c3975_1004 Proceed ([y]/n)? y Downloading and Extracting Packages python-3.7.3 | 35.7 MB | #################################################################################################################################################################### | 100% certifi-2019.3.9 | 149 KB | #################################################################################################################################################################### | 100% wheel-0.33.1 | 34 KB | #################################################################################################################################################################### | 100% setuptools-41.0.1 | 616 KB | #################################################################################################################################################################### | 100% pip-19.1 | 1.8 MB | #################################################################################################################################################################### | 100% bzip2-1.0.6 | 415 KB | #################################################################################################################################################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done |
I have also installed Kazoo using:
(kazoo) [root@fedora1 ~]# conda install -c conda-forge kazoo |
And I have also installed PyHive to connect to Hive, Pandas to manipulate data structures (finally I have not used it at the end):
conda install -c anaconda pyhive conda install -c conda-forge pandas |
Python source code
The small source code (kazoo_testing.py file name in below) I have written is mainly coming (for the Kazoo part) from the official documentation so no restriction to visit it:
from kazoo.client import KazooClient,KazooState def my_listener(state): if state == KazooState.LOST: # Register somewhere that the session was lost print('Connection lost !!') elif state == KazooState.SUSPENDED: # Handle being disconnected from Zookeeper print('Connection suspended !!') else: # Handle being connected/reconnected to Zookeeper print('Connected !!') zk = KazooClient(hosts='zookeeper_server01.domain.com:2181,zookeeper_server02.domain.com:2181,zookeeper_server03.domain.com:2181') zk.add_listener(my_listener) #zk.start() zk.start(timeout=5) # Display Zookeeper information print(zk.get_children('/')) #print(zk.get_children('hiveserver2')[0]) print(zk.get_children(path='hiveserver2')) for hiveserver2 in zk.get_children(path='hiveserver2'): array01=hiveserver2.split(';')[0].split('=')[1].split(':') hive_hostname=array01[0] hive_port=array01[1] print('Hive hostname: ' + hive_hostname) print('Hive port: ' + hive_port) |
The list of Zookeeper server can be taken from the Hive Ambari page where you can copy/paste the so called HIVESERVER2 JDBC URL.
The above source code does not include the PyHive connection but once you get the Hive host name and port you can easily connect with something like (configuration parameter is optional):
from pyhive import hive # Hive connection connection=hive.connect( host = hive_hostname, port = hive_port, configuration={'tez.queue.name': 'your_yarn_queue_name'}, username = "your_account" ) pandas01=pd.read_sql("select * from ...", connection) print(pandas01.sample(10)) |
Kazoo testing
I have the chance to have configured two HiveServer2 in my Hortonworks Hadoop cluster. Which is, by the way, strongly suggested if you aim to be Highly Available (HA). When the two HiveServer2 processes are up and running I get below result:
(kazoo) [yjaquier@fedora1 ~]$ python kazoo_testing.py Connected !! ['registry', 'cluster', 'brokers', 'storm', 'zookeeper', 'infra-solr', 'hbase-unsecure', 'tracers', 'hadoop-ha', 'admin', 'isr_change_notification', 'accumulo', 'logsearch', 'controller_epoch', 'hiveserver2', 'druid', 'rmstore', 'ambari-metrics-cluster', 'consumers', 'config'] ['serverUri=hiveserver201.domain.com:10000;version=1.2.1000.2.6.4.0-91;sequence=0000000042', 'serverUri=hiveserver202.domain.com:10000;version=1.2.1000.2.6.4.0-91;sequence=0000000043'] Hive hostname: hiveserver201.domain.com Hive port: 10000 Hive hostname: hiveserver202.domain.com Hive port: 10000 |
If I stop the first HiverServer2, after a while, I suppose the time for Zookeeper to get and propagate the information I finally get:
(kazoo) [yjaquier@fedora1 ~]$ python kazoo_testing.py Connected !! ['registry', 'cluster', 'brokers', 'storm', 'zookeeper', 'infra-solr', 'hbase-unsecure', 'tracers', 'hadoop-ha', 'admin', 'isr_change_notification', 'accumulo', 'logsearch', 'controller_epoch', 'hiveserver2', 'druid', 'rmstore', 'ambari-metrics-cluster', 'consumers', 'config'] ['serverUri=hiveserver202.domain.com:10000;version=1.2.1000.2.6.4.0-91;sequence=0000000042'] Hive hostname: hiveserver202.domain.com Hive port: 10000 |
References
