IT World https://blog.yannickjaquier.com RDBMS, Unix and many more... Fri, 26 Jul 2019 13:55:29 +0000 en-US hourly 1 https://wordpress.org/?v=5.2.2 ORC versus Parquet compression and response time https://blog.yannickjaquier.com/hadoop/orc-versus-parquet-compression-and-response-time.html https://blog.yannickjaquier.com/hadoop/orc-versus-parquet-compression-and-response-time.html#respond Fri, 02 Aug 2019 07:52:44 +0000 https://blog.yannickjaquier.com/?p=4596 Preamble For their open source position we have chosen to install an Hortonworks HDP 2.6 Hadoop cluster. At the initial phase of our Hadoop project ORC storage has been chosen as the default storage engine for our very first Hive tables. Performance of our queries is, obviously, a key factor we consider. This is why […]

The post ORC versus Parquet compression and response time appeared first on IT World.

]]>

Table of contents

Preamble

For their open source position we have chosen to install an Hortonworks HDP 2.6 Hadoop cluster. At the initial phase of our Hadoop project ORC storage has been chosen as the default storage engine for our very first Hive tables.

Performance of our queries is, obviously, a key factor we consider. This is why we have started to consider Live Long And Process (LLAP) and realized it was not so easy to handle in our small initial cluster. Then the merge between Hortonworks and Cloudera happened and we decided to move all our tables to Parquet storage engine with the clear objective to use Impala from Cloudera.

But at a point in time we have started to study a bit the disk space usage (again linked to our small initial cluster) and realized that Parquet tables were much bigger than their ORC counterpart. All our Hive tables are highly using partitioning for performance and to ease cleaning by simply dropping old partitions…

There are plenty of articles comparing Parquet and ORC (and others) storage engines but if you read them carefully till the end there will most probably be a disclaimer stating that the comparison is tightly linked to data nature. In other words your data model and figures is unique and you have really no other option than testing it by yourself and this blog post is here to provide few tricks to achieve this…

Our cluster is running HDP-2.6.4.0 with Ambari version 2.6.1.0.

ORC versus Parquet compression

On one partition of one table we observed:

  • Parquet = 33.9 G
  • ORC = 2.4 G

Digging further we saw that ORC compression can be easily configured in Ambari and we have set it to zlib:

orc_vs_parquet01
orc_vs_parquet01

While the default Parquet compression is (apparently) uncompressed that is obviously not really good from compression perspective.

Digging in multiple (contradictory) blog posts and official documentation and personal testing I have been able to draw below table:

Hive SQL property Default Values
orc.compress ZLIB NONE, ZLIB or SNAPPY
parquet.compression UNCOMPRESSED UNCOMPRESSED, GZIP or SNAPPY

Remark:
I have seen many blog posts suggesting to use parquet.compress for Parquet compression algorithm but in my opinion this one does not work…

To change compression algorithm when creating a table use TBLPROPERTIES keyword like:

STORED AS PARQUET TBLPROPERTIES("parquet.compression"="GZIP");
STORED AS ORC TBLPROPERTIES("orc.compress"="SNAPPY")

So, as an example, the test table I have built for my testing is something like:

drop table default.test purge;

CREATE TABLE default.test(
  column01 string,
  column02 string,
  column03 int,
  column04 array)
PARTITIONED BY (column01 string, column02 string, column03 int)
STORED AS PARQUET
TBLPROPERTIES("parquet.compression"="SNAPPY");

To get the size of your test table (replace database_name and table_name by real values) just use something like (check the value of hive.metastore.warehouse.dir for /apps/hive/warehouse):

[hdfs@server01 ~]$ hdfs dfs -du -s -h /apps/hive/warehouse/database_name/table_name

Then I have copied my source Parquet table to this test table using the six combination of storage engine and compression algorithms and the result is:

ORC Parquet
orc.compress parquet.compression
NONE ZLIB SNAPPY UNCOMPRESSED GZIP SNAPPY
12.9 G 2.4 G 3.2 G 33.9 G 7.3 G 11.5 G

Or graphically:

orc_vs_parquet02
orc_vs_parquet02

To do the copy I have written a shell script that is dynamically copying each partition of the source table to its destination table that has the same layout:

#!/bin/bash
#
# Y.JAQUIER  06/02/2019  Creation
#
# -----------------------------------------------------------------------------
#
# This job copy one table on another one
# Useful when migrating from Parquet to ORC for example
#
# Destination table must have been created before using this script
#
# -----------------------------------------------------------------------------
#

###############################################################################
# Function to execute a query on hive
# Param 1 : Query string
###############################################################################
function execute_query
{
  #echo "Execute query: $1"
  MYSTART=$(date)
  beeline -u "jdbc:hive2://${HIVE_CONNEXION}?tez.queue.name=${HIVE_QUEUE}" -n ${HIVE_USER} --incremental=true --silent=true --fastConnect=true -e "$1"
  status=$?
  MYEND=$(date)
  MYDURATION=$(expr $(date -d "$MYEND" +%s) - $(date -d "$MYSTART" +%s))
  echo "Executed in $MYDURATION second(s)"
  #echo "MEASURE|$FAB|${INGESTION_DATE}|$1|$(date -d "$MYSTART" +%Y%m%d%H%M%S)|$(date -d "$MYEND" +%Y%m%d%H%M%S)|$MYDURATION"
  if [[ $status != "0" ]]
  then
    echo " !!!!!!!! Error Execution Query $1 !!!!!!!! "
    exit -1
  fi
}

###############################################################################
# Function to execute a query on hive in CSV output file
# Param 1 : Query string
# Param 2 : output file
###############################################################################
function execute_query_to_csv
{
  #echo "Execute query to csv: $1 => $2"
  MYSTART=$(date)
  beeline -u "jdbc:hive2://${HIVE_CONNEXION}?tez.queue.name=${HIVE_QUEUE}" -n ${HIVE_USER} --outputformat=csv2 --silent=true --verbose=false \
  --showHeader=false --fastConnect=true -e "$1" > $2
  status=$?
  MYEND=$(date)
  MYDURATION=$(expr $(date -d "$MYEND" +%s) - $(date -d "$MYSTART" +%s))
  #echo "MEASURE|$FAB|${INGESTION_DATE}|$1|$(date -d "$MYSTART" +%Y%m%d%H%M%S)|$(date -d "$MYEND" +%Y%m%d%H%M%S)|$MYDURATION"
  if [[ $status != "0" ]]
  then
    echo " !!!!!!!! Error Execution Query to csv : $1 => $2 !!!!!!!! "
    exit -1
  fi
}

###############################################################################
# Print the help
###############################################################################
function print_help
{
  echo "syntax:"
  echo "$0 source_database.source_table destination_database.destination_table partition_filter_pattern_and_option (not mandatory)"
  echo "Source and destination table must exists"
  echo "Destination table data will be overwritten !!"
}

###############################################################################
# Main
###############################################################################
HIVE_CONNEXION="..."
HIVE_QUEUE=...
HIVE_USER=...

TABLE_SOURCE=$1
TABLE_DESTINATON=$2
PARTITION_FILTER=$3

if [[ $# < "2" ]]
then
  print_help
  exit 0 
fi

echo "This will overwrite $TABLE_DESTINATON table by $TABLE_SOURCE table data !"
echo "The destination table MUST be created first !"
read -p "Do you wish to continue [Y | N] ? " answer
case $answer in
  [Yy]* ) ;;
  [Nn]* ) exit 0;;
  * ) echo "Please answer yes or no."; exit 0;;
esac

# Generate partitions list
execute_query_to_csv "show partitions $1;" partition_list.$$.csv

# Filter partiion list base on regular expression given
if [[ $PARTITION_FILTER != "" ]]
then
  grep $PARTITION_FILTER partition_list.$$.csv > partition_list1.$$.csv
  mv partition_list1.$$.csv partition_list.$$.csv
fi

partition_number=$(cat partition_list.$$.csv | wc -l)

# Generate column list (with partition columns which must be removed)
execute_query_to_csv "show columns from $1;" column_list.$$.csv

# First partition column
while read line
do
  first_partition_column=$(echo $line | awk -F "=" '{print $1}')
  break
done < partition_list.$$.csv

# Columns list without partition columns
columns_list_without_partitions=""
while read line
do
  if [[ $line = $first_partition_column ]]
  then
    break
  fi
  columns_list_without_partitions+="$line,"
done < column_list.$$.csv

# Remove trailing comma
columns_length=${#columns_list_without_partitions}
columns_list_without_partitions=${columns_list_without_partitions:0:$(($columns_length-1))}

echo "The source table has $partition_number partition(s)"

# Generate list of all insert partition by partition
i=1
while read line
do
  #echo $line
  echo "Partition ${i}:"
  j=1
  query1="insert overwrite table $TABLE_DESTINATON partition ("
  query2=""
  query3=""
  IFS="/"
  read -r -a partition_list <<< "$line"

  # We fetch all partition columns
  for partition_column_list in "${partition_list[@]}"
  do
    IFS="="
    read -r -a partition_columns <<< "$partition_column_list"
    # First insert is with WHERE and we must enclosed columns value with double quote
    if [[ $j -eq 1 ]]
    then
      query3+="where ${partition_columns[0]}=\"${partition_columns[1]}\" "
    else
      query3+="and ${partition_columns[0]}=\"${partition_columns[1]}\" "
    fi
    query2+="${partition_columns[0]}=\"${partition_columns[1]}\","
    j=$((j+1))
  done
  IFS=""
  i=$((i+1))
  query2_length=${#query2}
  query2_length=$((query2_length-1))
  query2=${query2:0:$query2_length}
  final_query=$query1$query2") select "$columns_list_without_partitions" from $TABLE_SOURCE "$query3
  #Executing the query comment out the execute query to test it before running
  echo $final_query
  execute_query $final_query
done < partition_list.$$.csv
rm partition_list.$$.csv
rm column_list.$$.csv

So clearly for our data nature the ORC storage engine cannot be beaten when it comes to disk usage...

I have taken additional figures when we have migrated our live tables of our Spotfire data model:

[hdfs@server01 ~]$ hdfs dfs -du -s -h /apps/hive/warehouse/database.db/table01*
564.2 G  /apps/hive/warehouse/database.db/table01_orc
3.6 T  /apps/hive/warehouse/database.db/table01_pqt
[hdfs@server01 ~]$ hdfs dfs -du -s -h /apps/hive/warehouse/database.db/table02*
121.3 M  /apps/hive/warehouse/database.db/table02_pqt
5.6 M  /apps/hive/warehouse/database.db/table02_orc

ORC versus Parquet response time

But what about response time ? To do this I have extracted a typical query used by Spotfire and executed it on Parquet (UNCOMPRESSED) and on ORC (ZLIB) tables:

Iteration Parquet (s) ORC (s)
Run 1 6.553 0.077
Run 2 5.291 0.066
Run 3 1.915 0.065
Run 4 2.987 0.074
Run 5 1.825 0.070
Run 6 2.720 0.092
Run 7 3.989 0.062
Run 8 4.526 0.079
Run 9 3.385 0.082
Run 10 3.588 0.176
Average 3.6779 0.0843

So on average over my ten runs we have a factor of 43-44 time faster for ORC...

I would explain this that I have much less data to read on disk for the ORC tables and, again, this is linked to our data nodes hardware where we have much more CPU than disk axis (ratio of one thread per physical disk is not at all followed). If you are low on CPU and have plenty of disks (which is also not a good practice for an Hadoop cluster) you might experience different results...

References

The post ORC versus Parquet compression and response time appeared first on IT World.

]]>
https://blog.yannickjaquier.com/hadoop/orc-versus-parquet-compression-and-response-time.html/feed 0
HDFS balancer options to speed up balance operations https://blog.yannickjaquier.com/hadoop/hdfs-balancer-options-to-speed-up-balance-operations.html https://blog.yannickjaquier.com/hadoop/hdfs-balancer-options-to-speed-up-balance-operations.html#respond Fri, 05 Jul 2019 06:54:28 +0000 https://blog.yannickjaquier.com/?p=4604 Preamble We have started to receive the below Ambari alerts: Percent DataNodes With Available Spaceaffected: [2], total: [5] DataNode StorageRemaining Capacity:[4476139956751], Total Capacity:[77% Used, 19675609717760] In itself the DataNode Storage alert is not super serious because, first, it is sent far in advance (> 75%) but it anyways tells you that you are reaching the […]

The post HDFS balancer options to speed up balance operations appeared first on IT World.

]]>

Table of contents

Preamble

We have started to receive the below Ambari alerts:

  • Percent DataNodes With Available Space
    affected: [2], total: [5]
  • DataNode Storage
    Remaining Capacity:[4476139956751], Total Capacity:[77% Used, 19675609717760]

In itself the DataNode Storage alert is not super serious because, first, it is sent far in advance (> 75%) but it anyways tells you that you are reaching the storage limit of your cluster. One drawback we have seen is the impacted DataNodes are loosing contact with Ambari server and we are often obliged to restart the process.

On our small Hadoop cluster two nodes have more fill than the three others…

Should be easy to solve with below command:

[hdfs@clientnode ~]$ hdfs balancer

HDFS Balancer

We issued the HDFS balancer command with no options but after a very long run (almost a week) we end up with a still unbalanced situation. We even try to rerun the command but at the end the command completed very quickly (less than 2 seconds) but left us with two Datanodes still more filled than the three others.

[hdfs@clientnode ~]$ hdfs dfsadmin -report
Configured Capacity: 98378048588800 (89.47 TB)
Present Capacity: 93358971611260 (84.91 TB)
DFS Remaining: 31894899799432 (29.01 TB)
DFS Used: 61464071811828 (55.90 TB)
DFS Used%: 65.84%
Under replicated blocks: 24
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (5):

Name: 192.168.1.3:50010 (datanode03.domain.com)
Hostname: datanode03.domain.com
Rack: /AH/26
Decommission Status : Normal
Configured Capacity: 19675609717760 (17.89 TB)
DFS Used: 11130853114413 (10.12 TB)
Non DFS Used: 0 (0 B)
DFS Remaining: 7534254091791 (6.85 TB)
DFS Used%: 56.57%
DFS Remaining%: 38.29%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 25
Last contact: Tue Jan 08 12:51:44 CET 2019
Last Block Report: Tue Jan 08 06:52:34 CET 2019


Name: 192.168.1.2:50010 (datanode02.domain.com)
Hostname: datanode02.domain.com
Rack: /AH/26
Decommission Status : Normal
Configured Capacity: 19675609717760 (17.89 TB)
DFS Used: 11269739413291 (10.25 TB)
Non DFS Used: 0 (0 B)
DFS Remaining: 7403207769673 (6.73 TB)
DFS Used%: 57.28%
DFS Remaining%: 37.63%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 33
Last contact: Tue Jan 08 12:51:44 CET 2019
Last Block Report: Tue Jan 08 11:30:59 CET 2019


Name: 192.168.1.4:50010 (datanode04.domain.com)
Hostname: datanode04.domain.com
Rack: /AH/27
Decommission Status : Normal
Configured Capacity: 19675609717760 (17.89 TB)
DFS Used: 14226431394146 (12.94 TB)
Non DFS Used: 0 (0 B)
DFS Remaining: 4448006323316 (4.05 TB)
DFS Used%: 72.30%
DFS Remaining%: 22.61%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 14
Last contact: Tue Jan 08 12:51:43 CET 2019
Last Block Report: Tue Jan 08 12:12:55 CET 2019


Name: 192.168.1.1:50010 (datanode01.domain.com)
Hostname: datanode01.domain.com
Rack: /AH/26
Decommission Status : Normal
Configured Capacity: 19675609717760 (17.89 TB)
DFS Used: 10638187881052 (9.68 TB)
Non DFS Used: 0 (0 B)
DFS Remaining: 8035048514823 (7.31 TB)
DFS Used%: 54.07%
DFS Remaining%: 40.84%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 20
Last contact: Tue Jan 08 12:51:43 CET 2019
Last Block Report: Tue Jan 08 09:38:50 CET 2019


Name: 192.168.1.5:50010 (datanode05.domain.com)
Hostname: datanode05.domain.com
Rack: /AH/27
Decommission Status : Normal
Configured Capacity: 19675609717760 (17.89 TB)
DFS Used: 14198860008926 (12.91 TB)
Non DFS Used: 0 (0 B)
DFS Remaining: 4474383099829 (4.07 TB)
DFS Used%: 72.16%
DFS Remaining%: 22.74%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 29
Last contact: Tue Jan 08 12:51:45 CET 2019
Last Block Report: Tue Jan 08 11:50:32 CET 2019

From NameNode UI it gives the clean graphical picture:

hdfs_balancer01
hdfs_balancer01

Two datanodes are still more filled than the three others.

Then digging inside HDFS balancer official documentation we found two interesting parameters that are -source and -threshold.

-source is easily understandable with below example from official documentation (that I prefer to put it with the acquisition of Hortonworks by Cloudera):

The following table shows an example, where the average utilization is 25% so that D2 is within the 10% threshold. It is unnecessary to move any blocks from or to D2. Without specifying the source nodes, HDFS Balancer first moves blocks from D2 to D3, D4 and D5, since they are under the same rack, and then moves blocks from D1 to D2, D3, D4 and D5.
By specifying D1 as the source node, HDFS Balancer directly moves blocks from D1 to D3, D4 and D5.

Datanodes (with the same capacity) Utilization Rack
D1 95% A
D2 30% B
D3, D4, and D5 0% B

This is also explained in Storage group pairing policy:

The HDFS Balancer selects over-utilized or above-average storage as source storage, and under-utilized or below-average storage as target storage. It pairs a source storage group with a target storage group (source → target) in a priority order depending on whether or not the source and the target storage reside in the same rack.

And this rack awareness story is exactly what we have as displayed in server list of Ambari:

hdfs_balancer02
hdfs_balancer02

-threshold is also an interesting parameter to be more strict with nodes above or below the average…

So we tried unsuccessfully below command:

[hdfs@clientnode ~]$ hdfs balancer -source datanode04.domain.com,datanode05.domain.com -threshold 1

We also found many others “more agressive options” listed below:

DataNode Configuration Properties:

Property Default Background Mode Fast Mode
dfs.datanode.balance.max.concurrent.moves 5 4 x (# of disks) 4 x (# of disks)
dfs.datanode.balance.max.bandwidthPerSec 1048576 (1 MB) use default 10737418240 (10 GB)

Balancer Configuration Properties:

Property Default Background Mode Fast Mode
dfs.datanode.balance.max.concurrent.moves 5 # of disks 4 x (# of disks)
dfs.balancer.moverThreads 1000 use default 20,000
dfs.balancer.max-size-to-move 10737418240 (10 GB) 1073741824 (1GB) 107374182400 (100 GB)
dfs.balancer.getBlocks.min-block-size 10485760 (10 MB) use default 104857600 (100 MB)

So again tried:

[hdfs@clientnode ~]$ hdfs balancer -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=50 -Ddfs.balancer.dispatcherThreads=200 -threshold 1 \
-source datanode04.domain.com,datanode05.domain.com 1>/tmp/balancer-out.log 2>/tmp/balancer-err.log

But again it did not change anything special and they have been both executed very fast…

So clearly in our case the rack awareness story is a blocking factor. One mistake we have done is to have an odd number of datanodes and this 2-3 configuration in two racks is clearly not a good idea. Of course we could remove the rack awareness configuration to have a well balanced cluster but we do not want to loose the extra high availibilty we have with it. SO only available plan is to buy new databases or add more disks to our existing nodes as we have less disks than threads…

References

The post HDFS balancer options to speed up balance operations appeared first on IT World.

]]>
https://blog.yannickjaquier.com/hadoop/hdfs-balancer-options-to-speed-up-balance-operations.html/feed 0
JournalNode Web UI time out critical error on port 8480 https://blog.yannickjaquier.com/hadoop/journalnode-web-ui-critical-error.html https://blog.yannickjaquier.com/hadoop/journalnode-web-ui-critical-error.html#respond Fri, 07 Jun 2019 10:11:29 +0000 https://blog.yannickjaquier.com/?p=4549 Preamble One of our three JournalNodes was constantly failing for an alert on JournalNode Web UI saying the connection to http://journalnode3.domain.com:8480 has timed out. In Ambari the configured alert is this one: JournalNodes is: High-availabilty clusters use JournalNodes to synchronize active and standby NameNodes. The active NameNode writes to each JournalNode with changes, or “edits,” […]

The post JournalNode Web UI time out critical error on port 8480 appeared first on IT World.

]]>

Table of contents

Preamble

One of our three JournalNodes was constantly failing for an alert on JournalNode Web UI saying the connection to http://journalnode3.domain.com:8480 has timed out.

In Ambari the configured alert is this one:

journalnode_web_ui01
journalnode_web_ui01

JournalNodes is:

High-availabilty clusters use JournalNodes to synchronize active and standby NameNodes. The active NameNode writes to each JournalNode with changes, or “edits,” to HDFS namespace metadata. During failover, the standby NameNode applies all edits from the JournalNodes before promoting itself to the active state.

Obviously the URL is also not responding when accessing to it through a web browser…

JournalNode Web UI time out resolution

The port was correctly use by a listening process:

[root@journalnode3 ~]# netstat -an | grep 8480 |grep LISTEN
tcp        0      0 0.0.0.0:8480            0.0.0.0:*               LISTEN

Found this interesting article called Frequent journal node connection timeout alerts.

As suggested and expected curl command do not give anything and end up with a timed out:

[root@journalnode3 ~]# curl -v http://journalnode3.domain.com:8480 --max-time 4 | tail -4 

I also had tens of CLOSE_WAIT connection to this port using below suggested command:

[root@journalnode3 ~]# netstat -putane | grep -i 8480

You can find the JournalNodes directory with dfs.journalnode.edits.dir HDFS variable, which is set to /var/qjn for me.

The files in the directory of the problematic node were all out of date:

.
-rw-r--r-- 1 hdfs hadoop 182246 Oct 31 13:43 edits_0000000000175988225-0000000000175989255
-rw-r--r-- 1 hdfs hadoop 595184 Oct 31 13:45 edits_0000000000175989256-0000000000175992263
-rw-r--r-- 1 hdfs hadoop 216550 Oct 31 13:47 edits_0000000000175992264-0000000000175993354
-rw-r--r-- 1 hdfs hadoop 472885 Oct 31 13:49 edits_0000000000175993355-0000000000175995694
-rw-r--r-- 1 hdfs hadoop 282984 Oct 31 13:51 edits_0000000000175995695-0000000000175997143
-rw-r--r-- 1 hdfs hadoop      8 Oct 31 13:51 committed-txid
-rw-r--r-- 1 hdfs hadoop 626688 Oct 31 13:51 edits_inprogress_0000000000175997144

Versus the directory on one of the two working well JournalNodes:

.
-rw-r--r-- 1 hdfs hadoop  174901 Nov  8 11:28 edits_0000000000184771705-0000000000184772755
-rw-r--r-- 1 hdfs hadoop  418119 Nov  8 11:30 edits_0000000000184772756-0000000000184774838
-rw-r--r-- 1 hdfs hadoop  342889 Nov  8 11:32 edits_0000000000184774839-0000000000184776640
-rw-r--r-- 1 hdfs hadoop  270983 Nov  8 11:34 edits_0000000000184776641-0000000000184778154
-rw-r--r-- 1 hdfs hadoop  593676 Nov  8 11:36 edits_0000000000184778155-0000000000184781027
-rw-r--r-- 1 hdfs hadoop 1048576 Nov  8 11:37 edits_inprogress_0000000000184781028

in /var/log/hadoop/hdfs log directory I have also seen in hadoop-hdfs-journalnode-server1.domain.com.log file below error messages:

2018-11-08 12:18:20,949 WARN  namenode.FSImage (EditLogFileInputStream.java:scanEditLog(359)) - Caught exception after scanning through 0 ops from
/var/qjn/ManufacturingDataLakeHdfs/current/edits_inprogress_0000000000175997144 while determining its valid length. Position was 626688
java.io.IOException: Can't scan a pre-transactional edit log.
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LegacyReader.scanOp(FSEditLogOp.java:4974)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.scanNextOp(EditLogFileInputStream.java:245)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.scanEditLog(EditLogFileInputStream.java:355)
        at org.apache.hadoop.hdfs.server.namenode.FileJournalManager$EditLogFile.scanLog(FileJournalManager.java:551)
        at org.apache.hadoop.hdfs.qjournal.server.Journal.scanStorageForLatestEdits(Journal.java:192)
        at org.apache.hadoop.hdfs.qjournal.server.Journal.(Journal.java:152)
        at org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:90)
        at org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:99)
        at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.startLogSegment(JournalNodeRpcServer.java:165)
        at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.startLogSegment(QJournalProtocolServerSideTranslatorPB.java:186)
        at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25425)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)

I have stopped JournalNodes process via Ambari on problematic server and moves the edits_inprogress_xxx file that was out of date:

[root@journalnode3 current]# pwd /var/qjn/Hdfsproject/current [root@journalnode3 current]# mv edits_inprogress_0000000000175997144 edits_inprogress_0000000000175997144.bak

And restarted via Ambari the JournalNodes process on this server… It has taken a while to recover the situation but after few days the log generation have decreased and problematic JournalNodes was able to cope with latency… You might also delete the entire directory, not tested personally but should work…

Watch out the log directory size on problematic JounalNodes server as it may fill up very fast…

References

  • A Journal Node is in Bad Health in Cloudera Manager with “java.io.IOException: Can’t scan a pre-transactional edit log” (Doc ID 2160881.1)
  • HDFS Metadata Directories Explained

The post JournalNode Web UI time out critical error on port 8480 appeared first on IT World.

]]>
https://blog.yannickjaquier.com/hadoop/journalnode-web-ui-critical-error.html/feed 0
YARN command line for low level management of applications https://blog.yannickjaquier.com/hadoop/yarn-command-line-manage-applications.html https://blog.yannickjaquier.com/hadoop/yarn-command-line-manage-applications.html#respond Fri, 10 May 2019 07:24:26 +0000 https://blog.yannickjaquier.com/?p=4566 Preamble We recently implemented LLAP in our Hadoop cluster. To do this as you can see in the Hortonworks configuration guideline we have dedicated a YARN queue 100% to run LLAP queries and this YARN queue dedicated to LLAP has a greater priority than the other YARN queues. The queue we have used was already […]

The post YARN command line for low level management of applications appeared first on IT World.

]]>

Table of contents

Preamble

We recently implemented LLAP in our Hadoop cluster. To do this as you can see in the Hortonworks configuration guideline we have dedicated a YARN queue 100% to run LLAP queries and this YARN queue dedicated to LLAP has a greater priority than the other YARN queues. The queue we have used was already existing and was used to perform business reporting queries.

We unfortunately have observed a bad side effect of this change. We had plenty of scripts (in Python mainly) connecting to Hive server directly and not specifying any queue. Those scripts were, also, not going through ZooKeeper and in any case not specifying the new JDBC Hive2 URL to connect to HiveServer2 Interactive i.e. LLAP. The (unexpected) result was multiple applications automatically allocated to LLAP queue in a non-LLAP mode so having NO resources available to be launched…

Solution at that stage is to kill them or try to move them to a different queue… Part of this can be done with Resource Manager UI but for fine modification like queue allocation or application priority you must use YARN command line !

The edition we have chosen is the Hortonworks one and we have installed release HDP-2.6.4.0.

Problematic situation and YARN command line first trial

From Resource Manager UI we can see that many application are launched and have landed in the higher priority queue for LLAP (spotfire). I have started to suspect an issue because the applciation are waiting indefinitely. Application ID application_1543251372086_1684, for example, launched multiple hours ago (at the time of the screen shot) and completed at 0%:

yarn_command_line01
yarn_command_line01

The running applications have no allocated resources and we don’t even see them in TEZ view:

yarn_command_line02
yarn_command_line02

You can do (almost all) with Resource Manager UI but it’s always nice to have the YARN command line equivalence. If one day you like to developed few scripts to ease your life (the move queue below is also NOT accessible with graphical interface). YARN command line is also faster if you need to check status of multiple applications. Start by listing all applications, running and waiting to be run (ACCEPTED) with:

[yarn@mgmtserver ~]$ yarn application -list
18/11/28 14:43:04 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 14:43:04 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 14:43:04 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):24
                Application-Id      Application-Name        Application-Type          User           Queue                   State             Final-State             Progress                        Tracking-URL
application_1543251372086_1684  HIVE-bd74b31a-9151-43bb-a787-6159d79b0970                        TEZ      training        spotfire                 RUNNING               UNDEFINED                   0% http://worker03.domain.com:40313/ui/
application_1543251372086_2692  HIVE-3f923b84-4c06-4b4e-b395-e325a4721714                        TEZ          hive        spotfire                 RUNNING               UNDEFINED                   0% http://worker02.domain.com:34894/ui/
application_1543251372086_2735  HIVE-258d99b0-f275-4dae-853b-20da9e803fef                        TEZ      mfgaewsp        spotfire                 RUNNING               UNDEFINED                   0% http://worker01.domain.com:38295/ui/
application_1543251372086_2774  HIVE-abbb82a4-af93-403d-abc2-58fa6053a217                        TEZ      training        spotfire                 RUNNING               UNDEFINED                   0% http://worker05.domain.com:39203/ui/
application_1543251372086_2929  HIVE-c3d5d016-30f9-4237-ae1a-2858945070ae                        TEZ      training        spotfire                 RUNNING               UNDEFINED                   0% http://worker04.domain.com:43126/ui/
application_1543251372086_2930  HIVE-37c2b156-43e2-43a5-86b5-fa96f29486cf                        TEZ      training        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_2954  HIVE-de98d006-f1ee-43b6-9928-99c2b8338c2d                        TEZ    mfgdl_ingestion   spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_2955  HIVE-0ddab9a1-b335-4fd0-a717-db7ba5d7a6fd                        TEZ    mfgdl_ingestion   spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_2968  HIVE-ba458900-e9ee-48ad-b0a1-839be5208a81                        TEZ      training        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_2969  HIVE-e59da0bc-bf0b-428c-ad0c-3e262825bbc2                        TEZ      training        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_2967  HIVE-770714ee-8c6f-46b3-9374-b768b39b7f01                        TEZ      training        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_2952  HIVE-01e31000-829c-44b3-a0ff-7113c5864e86                        TEZ      mfgaewsp        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_2953  HIVE-b2089514-18e1-48e2-b8b1-85c48b407ab2                        TEZ      mfgaewsp        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_2946  HIVE-c04f98fd-b2a4-4c6d-9bc5-f2481e111f8c                        TEZ    mfgdl_ingestion  ingestion                 RUNNING               UNDEFINED                0.91% http://worker01.domain.com:43019/ui/
application_1543251372086_2970  HIVE-a3fdd952-d5ed-424e-861c-7e0ac4572d37                        TEZ      training        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_2971  HIVE-8d10cc75-bdfb-4b41-aa2b-2771f401681e                        TEZ      training        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_3025  HIVE-960aa95c-59f1-4f0f-9feb-6c615b8163cd                        TEZ      mfgaewsp        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_3023  HIVE-17224232-ebc8-4a3b-bd78-c183840c521c                        TEZ      mfgaewsp        spotfire                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_3100  HIVE-d4aabb8b-b239-4b2e-8bcd-e16880114842                        TEZ    mfgdl_ingestion  ingestion                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_3101  HIVE-9e63b3da-af08-40b0-8f74-4fd596089d51                        TEZ    mfgdl_ingestion  ingestion                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_3098  HIVE-04686451-e693-45c7-9d79-038cffa25a80                        TEZ    mfgdl_ingestion  ingestion                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_3099  HIVE-e12314df-8369-44c6-87cd-5b1c56b97a10                        TEZ    mfgdl_ingestion  ingestion                ACCEPTED               UNDEFINED                   0%                                 N/A
application_1543251372086_0019                 llap0       org-apache-slider          hive        spotfire                 RUNNING               UNDEFINED                 100%   http://worker03.domain.com:39758
application_1542883016916_0125  Thrift JDBC/ODBC Server                SPARK          hive       analytics                 RUNNING               UNDEFINED                  10%             http://192.168.1.2:4040

To have only the running ones:

[yarn@mgmtserver ~]$ yarn application -appStates RUNNING -list
18/11/28 17:15:27 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 17:15:27 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 17:15:28 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Total number of applications (application-types: [] and states: [RUNNING]):11
                Application-Id      Application-Name        Application-Type          User           Queue                   State             Final-State             Progress                        Tracking-URL
application_1543251372086_1684  HIVE-bd74b31a-9151-43bb-a787-6159d79b0970                        TEZ      training        spotfire                 RUNNING               UNDEFINED                   0% http://worker03.domain.com:40313/ui/
application_1543251372086_2692  HIVE-3f923b84-4c06-4b4e-b395-e325a4721714                        TEZ          hive        spotfire                 RUNNING               UNDEFINED                   0% http://worker02.domain.com:34894/ui/
application_1543251372086_2735  HIVE-258d99b0-f275-4dae-853b-20da9e803fef                        TEZ      mfgaewsp        spotfire                 RUNNING               UNDEFINED                   0% http://worker01.domain.com:38295/ui/
application_1543251372086_2774  HIVE-abbb82a4-af93-403d-abc2-58fa6053a217                        TEZ      training        spotfire                 RUNNING               UNDEFINED                   0% http://worker05.domain.com:39203/ui/
application_1543251372086_2929  HIVE-c3d5d016-30f9-4237-ae1a-2858945070ae                        TEZ      training        spotfire                 RUNNING               UNDEFINED                   0% http://worker04.domain.com:43126/ui/
application_1543251372086_2946  HIVE-c04f98fd-b2a4-4c6d-9bc5-f2481e111f8c                        TEZ    mfgdl_ingestion  ingestion                 RUNNING               UNDEFINED                0.91% http://worker01.domain.com:43019/ui/
application_1543251372086_0019                 llap0       org-apache-slider          hive        spotfire                 RUNNING               UNDEFINED                 100%   http://worker03.domain.com:39758
application_1542883016916_0125  Thrift JDBC/ODBC Server                SPARK          hive       analytics                 RUNNING               UNDEFINED                  10%             http://192.168.1.2:4040

If I take the LLAP process I get its application attempt with:

[yarn@mgmtserver ~]$ yarn applicationattempt -list application_1543251372086_0019
18/11/28 17:30:39 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 17:30:39 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 17:30:39 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Total number of application attempts :1
         ApplicationAttempt-Id                 State                        AM-Container-Id                            Tracking-URL
appattempt_1543251372086_0019_000001                 RUNNING    container_e185_1543251372086_0019_01_000001     http://masternode01.domain.com:8088/proxy/application_1543251372086_0019/

And the list of (LLAP) containers with:

[yarn@mgmtserver ~]$ yarn container -list appattempt_1543251372086_0019_000001
18/11/28 15:03:41 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 15:03:41 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 15:03:41 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Total number of containers :4
                  Container-Id            Start Time             Finish Time                   State                    Host       Node Http Address                                LOG-URL
container_e185_1543251372086_0019_01_000001     Mon Nov 26 18:05:35 +0100 2018                   N/A                 RUNNING    worker03.domain.com:45454      http://worker03.domain.com:8042        http://worker03.domain.com:8042/node/containerlogs/container_e185_1543251372086_0019_01_000001/hive
container_e185_1543251372086_0019_01_000005     Mon Nov 26 18:05:41 +0100 2018                   N/A                 RUNNING    worker04.domain.com:45454      http://worker04.domain.com:8042        http://worker04.domain.com:8042/node/containerlogs/container_e185_1543251372086_0019_01_000005/hive
container_e185_1543251372086_0019_01_000002     Mon Nov 26 18:05:40 +0100 2018                   N/A                 RUNNING    worker03.domain.com:45454      http://worker03.domain.com:8042        http://worker03.domain.com:8042/node/containerlogs/container_e185_1543251372086_0019_01_000002/hive
container_e185_1543251372086_0019_01_000003     Mon Nov 26 18:05:40 +0100 2018                   N/A                 RUNNING    worker01.domain.com:45454      http://worker01.domain.com:8042        http://worker01.domain.com:8042/node/containerlogs/container_e185_1543251372086_0019_01_000003/hive

At that stage it means that any application not using one of above containers is NOT using LLAP… Is it the case for the waiting indefinitely one we have seen above ?

YARN command line to the rescue

Get more details of a particular application with (one you suspect to be stuck):

[yarn@mgmtserver ~]$ yarn application -status application_1543251372086_1684
18/11/28 14:52:22 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 14:52:22 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 14:52:22 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Application Report :
        Application-Id : application_1543251372086_1684
        Application-Name : HIVE-bd74b31a-9151-43bb-a787-6159d79b0970
        Application-Type : TEZ
        User : training
        Queue : spotfire
        Application Priority : null
        Start-Time : 1543339326789
        Finish-Time : 0
        Progress : 0%
        State : RUNNING
        Final-State : UNDEFINED
        Tracking-URL : http://worker03.domain.com:40313/ui/
        RPC Port : 42387
        AM Host : worker03.domain.com
        Aggregate Resource Allocation : 61513551 MB-seconds, 15017 vcore-seconds
        Log Aggregation Status : NOT_START
        Diagnostics :
        Unmanaged Application : false
        Application Node Label Expression : 
        AM container Node Label Expression : 

Application is in state RUNNING, in YARN queue spotfire (LLAP) and progress status is 0%.

Get the application attempt (in order to get container list) with:

[yarn@mgmtserver ~]$ yarn applicationattempt -list application_1543251372086_1684
18/11/28 14:53:41 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 14:53:41 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 14:53:41 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Total number of application attempts :1
         ApplicationAttempt-Id                 State                        AM-Container-Id                            Tracking-URL
appattempt_1543251372086_1684_000001                 RUNNING    container_e185_1543251372086_1684_01_000001     http://masternode01.domain.com:8088/proxy/application_1543251372086_1684/

Get container list with:

[yarn@mgmtserver ~]$ yarn container -list appattempt_1543251372086_1684_000001
18/11/28 14:53:55 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 14:53:55 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 14:53:55 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Total number of containers :1
                  Container-Id            Start Time             Finish Time                   State                    Host       Node Http Address                                LOG-URL
container_e185_1543251372086_1684_01_000001     Wed Nov 28 10:42:02 +0100 2018                   N/A                 RUNNING    worker03.domain.com:45454      http://worker03.domain.com:8042        http://worker03.domain.com:8042/node/containerlogs/container_e185_1543251372086_1684_01_000001/training

Here we see that this application is in spotfire queue and not running with one of the LLAP container so the issue…

Get the status of a container with:

[yarn@mgmtserver ~]$ yarn container -status container_e185_1543251372086_1684_01_000001
18/11/28 14:54:25 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 14:54:25 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 14:54:25 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Container Report :
        Container-Id : container_e185_1543251372086_1684_01_000001
        Start-Time : 1543398122796
        Finish-Time : 0
        State : RUNNING
        LOG-URL : http://worker03.domain.com:8042/node/containerlogs/container_e185_1543251372086_1684_01_000001/training
        Host : worker03.domain.com:45454
        NodeHttpAddress : http://worker03.domain.com:8042
        Diagnostics : null

Now you can decide to abruptly kill it with:

[yarn@mgmtserver ~]$ yarn application -kill application_1543251372086_1684
18/11/28 17:50:51 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 17:50:51 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 17:50:51 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Killing application application_1543251372086_1684
18/11/28 17:50:51 INFO impl.YarnClientImpl: Killed application application_1543251372086_1684

Or to move it to a non-LLAP queue that have free resources with -movetoqueue option:

[yarn@mgmtserver ~]$ yarn application -queue analytics -movetoqueue application_1543251372086_1684
18/11/28 15:05:53 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 15:05:53 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 15:05:53 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Moving application application_1543251372086_1684 to queue analytics
Successfully completed move.

In above 1543339326789 is Tue Nov 27 18:22:06 +0100 2018. Which is almost 24 hours before I have taken the screen shot. Your can convert the timestamp in human readable date with many available web sites or directly in Bash with something like (man date for more information):

[yarn@mgmtserver ~]$ date --date='@1543339327' +'%c'
Tue 27 Nov 2018 06:22:07 PM CET

After a very short period I finally got below result (I wanted to go faster by changing application priority but my release is too old):

[yarn@mgmtserver ~]$ yarn application -status application_1543251372086_1684
18/11/28 15:08:54 INFO client.AHSProxy: Connecting to Application History server at masternode01.domain.com/192.168.1.1:10200
18/11/28 15:08:54 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/11/28 15:08:54 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
Application Report :
        Application-Id : application_1543251372086_1684
        Application-Name : HIVE-bd74b31a-9151-43bb-a787-6159d79b0970
        Application-Type : TEZ
        User : training
        Queue : analytics
        Application Priority : null
        Start-Time : 1543339326789
        Finish-Time : 1543414091398
        Progress : 100%
        State : FINISHED
        Final-State : SUCCEEDED
        Tracking-URL : http://mgmtserver.domain.com:8080/#/main/view/TEZ/tez_cluster_instance?viewPath=%2F%23%2Ftez-app%2Fapplication_1543251372086_1684
        RPC Port : 42387
        AM Host : worker03.domain.com
        Aggregate Resource Allocation : 67097515 MB-seconds, 16378 vcore-seconds
        Log Aggregation Status : SUCCEEDED
        Diagnostics : Session stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0

        Unmanaged Application : false
        Application Node Label Expression : 
        AM container Node Label Expression : 

References

The post YARN command line for low level management of applications appeared first on IT World.

]]>
https://blog.yannickjaquier.com/hadoop/yarn-command-line-manage-applications.html/feed 0
StreamSets Data Collector replication with Oracle, MySQL and JSON https://blog.yannickjaquier.com/oracle/streamsets-data-collector-oracle-cdc-client.html https://blog.yannickjaquier.com/oracle/streamsets-data-collector-oracle-cdc-client.html#comments Thu, 11 Apr 2019 09:48:45 +0000 https://blog.yannickjaquier.com/?p=4493 Preamble I came across a nice overview article of Franck Pachot and shared it with few teammates and they have all been interested by StreamSets Data Collector product. One of the main reason is the obvious cost cutting versus GoldenGate that we have implemented in a project deployed worldwide. The product is free but has […]

The post StreamSets Data Collector replication with Oracle, MySQL and JSON appeared first on IT World.

]]>

Table of contents

Preamble

I came across a nice overview article of Franck Pachot and shared it with few teammates and they have all been interested by StreamSets Data Collector product. One of the main reason is the obvious cost cutting versus GoldenGate that we have implemented in a project deployed worldwide. The product is free but has some clearly described limitation like managing only INSERT, UPDATE, SELECT_FOR_UPDATE, and DELETE operations for one or more tables in a database. So in other words DDL are not managed as well as few data types (not an issue for us)…

To really handle DDL you would have to check “Produce Events” check box and handle the generated events on target to handle DDL, this is a little bit more complex and outside the scope of this blog post…

I have decided to give a try to the product and implement it for what we currently do with GoldenGate means building a reporting environment of our production database. Target is also an Oracle database but might be in future a MySQL one.

My testing environment is made of three servers:

  • server1.domain.com (192.168.56.101) is the primary database server.
  • server2.domain.com (192.168.56.102) is the secondary database server. The server hosting the databases (Oracle & MySQL) where figures should land.
  • server4.domain.com (192.168.56.104) is the StreamSets server.

Oracle database release is 18c Enterprise Edition Release 18.3.0.0.0. StreamSets is release 3.4.1. MySQL release is 8.0.12 MySQL Community Server.

The three servers are in fact VirtualBox guests running Oracle Linux Server release 7.5.

StreamSets Data Collector installation

I have first created a Linux streamsets account (in users group) with a /streamsets dedicated filesystem:

[streamsets@server4 ~]$ id
uid=1001(streamsets) gid=100(users) groups=100(users)
[streamsets@server4 ~]$ pwd
/streamsets
[streamsets@server4 ~]$ ll
total 246332
-rw-r--r-- 1 streamsets users 248202258 Aug  2 15:32 streamsets-datacollector-core-3.4.1.tgz
[streamsets@server4 streamsets]$ tar xvzf streamsets-datacollector-core-3.4.1.tgz
[streamsets@server4 ~]$ /streamsets/streamsets-datacollector-3.4.1/bin/streamsets dc
Can't find java, please set JAVA_HOME pointing to your java installation

I have installed Java SE Development Kit 8 (jdk-8u181-linux-x64.rpm). Only release 8 is supported so far…

[streamsets@server4 streamsets]$ /streamsets/streamsets-datacollector-3.4.1/bin/streamsets dc
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
Configuration of maximum open file limit is too low: 1024 (expected at least 32768). Please consult https://goo.gl/LgvGFl

At the end of /etc/security/limits.conf I have added:

streamsets        soft    nofile           32768
streamsets        hard    nofile           32768

Now I can launch the process with:

[streamsets@server4 streamsets]$ /streamsets/streamsets-datacollector-3.4.1/bin/streamsets dc
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
Logging initialized @1033ms to org.eclipse.jetty.util.log.Slf4jLog
Running on URI : 'http://server4:18630'

Then from any browser (at this url for me, http://server4.domain.com:18630) you should get this login windows (admin/admin as default username/password):

streamsets01
streamsets01

Once logged you get:

streamsets02
streamsets02

By default Oracle CDC (Change Data Capture) client requires Oracle JDBC thin driver, in top left tool bar click on Package Manager (third icon). If you go in JDBC you see that nothing is there (the error message is because I’m behind a company proxy):

streamsets03
streamsets03

When I tried to import my JDBC thin driver file (ojdbc8.jar, the name for 18c (18.3) is the same as for 12.2.0.1, but the size differ) I have seen that JDBC category is not there:

streamsets04
streamsets04

I have spent a bit of time to see that everything was on StreamSets Data Collector download page:

[streamsets@server4 ~]$ /streamsets/streamsets-datacollector-3.4.1/bin/streamsets stagelibs -list

curl: (6) Could not resolve host: archives.streamsets.com; Unknown error
Failed! running curl -s -f https://archives.streamsets.com/datacollector/3.4.1/tarball/stage-lib-manifest.properties.sha1
-SL -o /tmp/sdc-setup-20988/stage-lib-manifest.properties.sha1 in /home/streamsets

[streamsets@server4 ~]$ export https_proxy='http://proxy_account:proxy_password@proxy_host:proxy_port'
[streamsets@server4 ~]$ echo $https_proxy
http://proxy_account:proxy_password@proxy_host:proxy_port
[streamsets@server4 streamsets]$ /streamsets/streamsets-datacollector-3.4.1/bin/streamsets stagelibs -list



StreamSets Data Collector

Stage Library Repository: https://archives.streamsets.com/datacollector/3.4.1/tarball

    ID                                                           Name                                     Installed
=================================================================================================================
 streamsets-datacollector-aerospike-lib                       Aerospike 3.15.0.2                           NO
 streamsets-datacollector-apache-kafka_0_10-lib               Apache Kafka 0.10.0.0                        NO
 streamsets-datacollector-apache-kafka_0_11-lib               Apache Kafka 0.11.0.0                        NO
 streamsets-datacollector-apache-kafka_0_9-lib                Apache Kafka 0.9.0.1                         NO
 streamsets-datacollector-apache-kafka_1_0-lib                Apache Kafka 1.0.0                           NO
 streamsets-datacollector-apache-kudu_1_3-lib                 Apache Kudu 1.3.0                            NO
 streamsets-datacollector-apache-kudu_1_4-lib                 Apache Kudu 1.4.0                            NO
.
.
 streamsets-datacollector-jdbc-lib                            JDBC                                         NO
.
.
[streamsets@server4 streamsets]$ /streamsets/streamsets-datacollector-3.4.1/bin/streamsets stagelibs -install=streamsets-datacollector-jdbc-lib




Downloading: https://archives.streamsets.com/datacollector/3.4.1/tarball/streamsets-datacollector-jdbc-lib-3.4.1.tgz
######################################################################## 100.0%

Stage library streamsets-datacollector-jdbc-lib installed

Relaunch StreamSets Data Collector and back to web interface I have now seen JDBC as possible library:

streamsets05
streamsets05

So imported JDBC driver:

streamsets06
streamsets06

You should be prompt to restart StreamSets Data Collector and see below screen:

streamsets07
streamsets07

StreamSets Data Collector source Oracle database configuration

This source multitenant Oracle database is common for the three scenario I have decided to test so you have to do it only once. On this source Oracle database you have to create a global account able to manage LogMiner (global because LogMiner is accessible from root pluggable database in a multitenant architecture):

SQL> alter session set container=cdb$root;

Session altered.

SQL> create user c##streamsets identified by "streamsets" container=all;

User created.

SQL> grant create session, alter session, set container, select any dictionary, logmining, execute_catalog_role TO c##streamsets container=all;

Grant succeeded.

SQL> alter session set container=pdb1;

Session altered.

Change source Oracle database to archivelog mode and activate default supplemental log:

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 1048575184 bytes
Fixed Size                  8903888 bytes
Variable Size             729808896 bytes
Database Buffers          301989888 bytes
Redo Buffers                7872512 bytes
Database mounted.
SQL> alter database archivelog;

Database altered.

SQL> alter database open;

Database altered.

SQL> select supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all from v$database;

SUPPLEME SUP SUP
-------- --- ---
NO       NO  NO

SQL> alter database add supplemental log data;

Database altered.

SQL> select supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all from v$database;

SUPPLEME SUP SUP
-------- --- ---
YES      NO  NO

SQL> alter system switch logfile;

System altered.

I have also created a test schema on my pluggable database (pdb1) to handle my test table:

SQL> create user test01 identified by test01;

User created.

SQL> grant connect,resource to test01;

Grant succeeded.

SQL> alter user test01 quota unlimited on users;

User altered.

Create a test table and add supplemental log:

SQL> create table test01.table01 (
  id number not null,
  descr varchar2(50),
  constraint table01_pk primary key (id) enable 
);

Table created.

SQL> alter table test01.table01 add supplemental log data (primary key) columns;

Table altered.

SQL> set lines 200 pages 1000
SQL> col table_name for a15
SQL> col log_group_name for a15
SQL> col owner for a15
SQL> select * from dba_log_groups where owner='TEST01';

OWNER           LOG_GROUP_NAME  TABLE_NAME      LOG_GROUP_TYPE      ALWAYS      GENERATED
--------------- --------------- --------------- ------------------- ----------- --------------
TEST01          SYS_C007365     TABLE01         PRIMARY KEY LOGGING ALWAYS      GENERATED NAME

And insert few rows in it to simulate an already existing environment:

SQL> insert into test01.table01 values(1,'One');

1 row created.

SQL> insert into test01.table01 values(2,'Two');

1 row created.

SQL> commit;

Commit complete.

On source database generate a dictionary in redo log. If you choose to set “Dictionary Source” to Online Catalog then this step is not mandatory. It is also much faster on small ressources to use from Online Catalog so really up to you:

SQL> alter session set container=cdb$root;

Session altered.

SQL> execute dbms_logmnr_d.build(options=> dbms_logmnr_d.store_in_redo_logs);

PL/SQL procedure successfully completed.

SQL> col name for a60
SQL> set lines 200 pages 1000
SQL> select name,dictionary_begin,dictionary_end from v$archived_log where name is not null order by recid desc;

NAME                                                         DIC DIC
------------------------------------------------------------ --- ---
/u01/app/oracle/oradata/ORCL/arch/1_135_983097959.dbf        YES YES
/u01/app/oracle/oradata/ORCL/arch/1_134_983097959.dbf        NO  NO
/u01/app/oracle/oradata/ORCL/arch/1_133_983097959.dbf        NO  NO
/u01/app/oracle/oradata/ORCL/arch/1_132_983097959.dbf        NO  NO
/u01/app/oracle/oradata/ORCL/arch/1_131_983097959.dbf        NO  NO
/u01/app/oracle/oradata/ORCL/arch/1_130_983097959.dbf        NO  NO
/u01/app/oracle/oradata/ORCL/arch/1_129_983097959.dbf        NO  NO
/u01/app/oracle/oradata/ORCL/arch/1_128_983097959.dbf        YES YES
/u01/app/oracle/oradata/ORCL/arch/1_127_983097959.dbf        NO  NO
/u01/app/oracle/oradata/ORCL/arch/1_126_983097959.dbf        NO  NO

StreamSets Data Collector Oracle to Oracle replication

Oracle prerequisites

On target Oracle database I have created an account in my target pluggable database (pdb1):>/p>

SQL> create user test01 identified by test01;

User created.

SQL> grant connect,resource to test01;

Grant succeeded.

SQL> alter user test01 quota unlimited on users;

User altered.

On source database create an export directory and grant read and write on it to test01 account:

SQL> alter session set container=pdb1;

Session altered.

SQL> create or replace directory tmp as '/tmp';

Directory created.

SQL> grant read,write on directory tmp to test01;

Grant succeeded.

Get the current System Change Number (SCN) on source database with:

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
    5424515

Finally export the figures with:

[oracle@server1 ~]$ expdp test01/test01@pdb1 dumpfile=table01.dmp directory=tmp tables=table01 flashback_scn=5424515

Export: Release 18.0.0.0.0 - Production on Wed Sep 5 13:02:23 2018
Version 18.3.0.0.0

Copyright (c) 1982, 2018, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Starting "TEST01"."SYS_EXPORT_TABLE_01":  test01/********@pdb1 dumpfile=table01.dmp directory=tmp tables=table01 flashback_scn=5424515
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER
Processing object type TABLE_EXPORT/TABLE/TABLE
Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
. . exported "TEST01"."TABLE01"                          5.492 KB       2 rows
Master table "TEST01"."SYS_EXPORT_TABLE_01" successfully loaded/unloaded
******************************************************************************
Dump file set for TEST01.SYS_EXPORT_TABLE_01 is:
  /tmp/table01.dmp
Job "TEST01"."SYS_EXPORT_TABLE_01" successfully completed at Wed Sep 5 13:04:25 2018 elapsed 0 00:01:09

[oracle@server1 ~]$ ll /tmp/table01.dmp
-rw-r----- 1 oracle dba 200704 Sep  5 13:04 /tmp/table01.dmp
[oracle@server1 ~]$ scp /tmp/table01.dmp server2.domain.com:/tmp
The authenticity of host 'server2.domain.com (192.168.56.102)' can't be established.
ECDSA key fingerprint is SHA256:hduqTIePPHF3Y+N/ekuZKnnXbocm+PNS7yU/HCf1GEw.
ECDSA key fingerprint is MD5:13:dc:e3:27:bc:4b:08:b8:bf:53:2a:15:3c:86:d7:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'server2.domain.com' (ECDSA) to the list of known hosts.
oracle@server2.domain.com's password:
table01.dmp                                                                      100%  196KB  19.0MB/s   00:00                    

Import the figures on target database with something like (in pdb1 pluggable database):

[oracle@server2 ~]$ impdp test01/test01@pdb1 file=table01.dmp directory=tmp

Import: Release 18.0.0.0.0 - Production on Wed Sep 5 13:06:46 2018
Version 18.3.0.0.0

Copyright (c) 1982, 2018, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Legacy Mode Active due to the following parameters:
Legacy Mode Parameter: "file=table01.dmp" Location: Command Line, Replaced with: "dumpfile=table01.dmp"
Master table "TEST01"."SYS_IMPORT_FULL_01" successfully loaded/unloaded
Starting "TEST01"."SYS_IMPORT_FULL_01":  test01/********@pdb1 dumpfile=table01.dmp directory=tmp
Processing object type TABLE_EXPORT/TABLE/TABLE
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
. . imported "TEST01"."TABLE01"                          5.492 KB       2 rows
Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
ORA-39083: Object type OBJECT_GRANT failed to create with error:
ORA-01917: user or role 'C##STREAMSETS' does not exist

Failing sql is:
GRANT SELECT ON "TEST01"."TABLE01" TO "C##STREAMSETS"

Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER
Job "TEST01"."SYS_IMPORT_FULL_01" completed with 1 error(s) at Wed Sep 5 13:07:47 2018 elapsed 0 00:00:46

StreamSets Data Collector configuration

Create the pipeline:

streamsets08
streamsets08

Choose Oracle CDC Client as a source (from replicating from Oracle this is the de facto option to choose):

streamsets09
streamsets09

Configure all parameters. I have chosen the most reliable solution to get dictionary from redo in case we want to test schema change (DDL). As the simplest test I have chosen From Latest Change – Processes changes that arrive after you start the pipeline. I will re-configure it after the database settings:

streamsets10
streamsets10

Below query can help you to choose correct database time zone:

SQL> SELECT DBTIMEZONE FROM DUAL;

DBTIME
------
+00:00

Set JDBC connection string (I am in a multitenant configuration):

streamsets11
streamsets11

Use the global account we have define earlier:

streamsets12
streamsets12

Not defining a Processor to modify figures between source and target but obviously this is possible:

streamsets13
streamsets13

Set JDBC connection string. I am in a multitenant configuration but here I just connect to the pluggable destination database directly:

streamsets14
streamsets14

Local pluggable database user to connect with:

streamsets15
streamsets15

In StreamSets configuration I change Oracle CDC Client configuration to instruct to start at a particular SCN (the one we have extracted above):

streamsets16
streamsets16

StreamSets Data Collector Oracle to Oracle replication testing

Then on source database I can start inserting new figures:

SQL> alter session set container=pdb1;

Session altered.

SQL> insert into test01.table01 values(3,'Three');

1 row created.

SQL> commit;

Commit complete.

And you should see them on target database as well as having a nice monitoring screen of the pipeline:

streamsets17
streamsets17

Then to generate a bit of traffic I have used below PL/SQL script (number of inserted rows is up to you and I have personally done multiple test):

declare
  max_id number;
  i number;
  inserted_rows number:=10000;
begin
  select max(id) into max_id from test01.table01;
  i:=max_id+1;
  loop
    insert into test01.table01 values(i,dbms_random.string('U', 20));
    commit;
    i:=i+1;
    exit when i>max_id + inserted_rows;
  end loop;
end;
/

And if you capture the monitoring screen while it’s running you should be able to see transfer rate figures:

streamsets18
streamsets18

StreamSets Data Collector Oracle to JSON file generation

In this extra testing I wanted to test the capability on top of JDBC insertion in a target Oracle database the capability to generate a JSON file. I have started by adding a new destination called Local FS and then draw with the mouse in the GUI interface a new line between the Oracle CDC Client and the Local FS. The only parameter I have changed is generated Data Format as classical JSON:

streamsets19
streamsets19

Once I insert a row in source table (I have restarted from an empty table) the record is duplicated in:

  • The same target Oracle database, same as above.
  • A text file, on StreamSets Data Collector server, in JSON format.

We can see the output generated records is equal to two:

streamsets20
streamsets20

The output file (located on server where StreamSets Data Collector is running i.e. server4.domain.com):

[root@server4 ~]# cat /tmp/out/2018-09-11-15/_tmp_sdc-2622d297-ac69-11e8-bf06-e301dabcb2ba_0
{"ID":1,"DESCR":"One"}

StreamSets Data Collector Oracle to MySQL replication

I have obviously created a small MySQL 8 instance. I have used my personal account to connect to it and created a test01 database to map schema name of source Oracle pluggable database:

mysql> create user  'yjaquier'@'%' identified by 'secure_password';
Query OK, 0 rows affected (0.31 sec)

mysql>  grant all privileges on *.* to 'yjaquier'@'%' with grant option;
Query OK, 0 rows affected (0.33 sec)
mysql> create database if not exists test01
    -> CHARACTER SET = utf32
    -> COLLATE = utf32_general_ci;
Query OK, 1 row affected (0.59 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test01             |
+--------------------+
5 rows in set (0.00 sec)

I create the target table same as the one of source Oracle table:

mysql> create table test01.table01 (
    -> id int not null,
    -> descr varchar(50) null,
    -> primary key (id));
Query OK, 0 rows affected (1.30 sec)

Remark:
I have started from an empty table, but if it’s not the case then an export and an import of pre-existing figures should be handle…

The JDBC connect string for MySQL is (3322 is my MySQL listening port):

jdbc:mysql://server2.domain.com:3322

I have also set two below JDBC parameters (see Errors encountered section):

streamsets21
streamsets21

The Schema Name parameter for MySQL must be inserted in lower case so test01 in my case and Table Name must also be in lowercase so use below formula (str:toLower function) to convert uppercase Oracle table name to lower case:

${str:toLower(record:attribute('oracle.cdc.table'))}
streamsets22
streamsets22

Finally records have also been inserted in MySQL target table:

mysql> select * from test01.table01;
+----+-------+
| id | descr |
+----+-------+
|  1 | One   |
|  2 | Two   |
|  3 | Three |
+----+-------+
3 rows in set (0.00 sec)

Errors encountered

JDBC_52 – Error starting LogMiner

In sdc.log or in View Logs of interface file you should see something like:

LOGMINER - CONTINUOUS_MINE  - failed to add logfile /u01/app/oracle/oradata/ORCL/arch/1_4_983097959.dbf because of status 1284
2018-08-07T17:19:02.844615+02:00

It was a mistake from my side where I have deleted archived log file directly on disk. Recovered the situation with:

RMAN> crosscheck archivelog all;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=277 device type=DISK
validation failed for archived log
archived log file name=/u01/app/oracle/oradata/ORCL/arch/1_4_983097959.dbf RECID=1 STAMP=983191889
validation failed for archived log
.
.

RMAN> list archivelog all;

using target database control file instead of recovery catalog
List of Archived Log Copies for database with db_unique_name ORCL
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - ---------
1       1    4       X 02-AUG-18
        Name: /u01/app/oracle/oradata/ORCL/arch/1_4_983097959.dbf
.
.

RMAN> delete noprompt expired archivelog all;

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=271 device type=DISK
List of Archived Log Copies for database with db_unique_name ORCL
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - ---------
1       1    4       X 02-AUG-18
        Name: /u01/app/oracle/oradata/ORCL/arch/1_4_983097959.dbf

I also had the brother of above one with:

JDBC_44 - Error while getting changes due to error: com.streamsets.pipeline.api.StageException: JDBC_52 - Error starting LogMiner

It was simply because no archived log file was containing a dictionary log. This can happen when you purge archived log files. Generate one with:

SQL> execute dbms_logmnr_d.build(options=> dbms_logmnr_d.store_in_redo_logs);

PL/SQL procedure successfully completed.

JDBC_44 – Error while getting changes due to error: java.sql.SQLRecoverableException: Closed Connection: getBigDecimal

When validating pipeline I got:

JDBC_44 - Error while getting changes due to error: java.sql.SQLRecoverableException: Closed Connection: getBigDecimal

And found in sdc.log file below error:

2018-08-31 10:53:35,754 [user:*admin] [pipeline:Oracle-to-Oracle/OracletoOracle7593b814-1185-4829-9fe5-6247106856c0] [runner:] [thread:preview-pool-1-thread-4] WARN  OracleCDCSource - Error while stopping LogMiner
java.sql.SQLRecoverableException: Closed Connection

From what I have seen around it looks like my test server is too slow and if you get this you might need to increase the timeout parameters in Advanced tab of Oracle CDC client…

ORA-01291: missing logfile

This one has kept me busy for a while:

106856c0-Oracle-to-Oracle] INFO  JdbcUtil - Driver class oracle.jdbc.OracleDriver (version 18.3)
2018-09-03 17:50:48,069 [user:*admin] [pipeline:Oracle-to-Oracle/OracletoOracle7593b814-1185-4829-9fe5-6247106856c0] [runner:0] [thread:ProductionPipelineRunnable-OracletoOracle7593b814-1185-4829-9fe5-6247
106856c0-Oracle-to-Oracle] INFO  HikariDataSource - HikariPool-1 - is starting.
2018-09-03 17:50:49,354 [user:*admin] [pipeline:Oracle-to-Oracle/OracletoOracle7593b814-1185-4829-9fe5-6247106856c0] [runner:] [thread:ProductionPipelineRunnable-OracletoOracle7593b814-1185-4829-9fe5-62471
06856c0-Oracle-to-Oracle] INFO  OracleCDCSource - Trying to start LogMiner with start date: 31-08-2018 10:40:33 and end date: 31-08-2018 12:40:33
2018-09-03 17:50:49,908 [user:*admin] [pipeline:Oracle-to-Oracle/OracletoOracle7593b814-1185-4829-9fe5-6247106856c0] [runner:] [thread:ProductionPipelineRunnable-OracletoOracle7593b814-1185-4829-9fe5-62471
06856c0-Oracle-to-Oracle] ERROR OracleCDCSource - SQLException while trying to setup record generator thread
java.sql.SQLException: ORA-01291: missing logfile
ORA-06512: at "SYS.DBMS_LOGMNR", line 58
ORA-06512: at line 1

        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:494)
        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:446)
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1052)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:537)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:255)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:610)
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:249)
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:82)
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:924)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1136)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.

One of the reason I have identified is because StreamSets start LogMiner two hours in the past even when you choose from latest changes:

2018-08-31 12:53:00,846 [user:*admin] [pipeline:Oracle-to-Oracle/OracletoOracle7593b814-1185-4829-9fe5-6247106856c0] [runner:] [thread:ProductionPipelineRunnable-OracletoOracle7593b814-1185-4829-9fe5-62471
06856c0-Oracle-to-Oracle] INFO  OracleCDCSource - Trying to start LogMiner with start date: 31-08-2018 10:40:33 and end date: 31-08-2018 12:40:33

I suspect it occurs because Oracle CDC Client LogMiner Session Window parameter is set to 2 hours you must have the archived log files from last 2 hours available when starting the pipeline. So never ever purge archivelog file with:

RMAN> delete noprompt archivelog all;

But use:

RMAN> delete noprompt archivelog all completed before 'sysdate-3/24';

But even when applying this I also noticed StreamSets was always starting LogMiner from the time where the pipeline has failed or when you have stopped it. This is saved in offset.json file:

[root@server4 0]# pwd
/streamsets/streamsets-datacollector-3.4.1/data/runInfo/OracletoOracle7593b814-1185-4829-9fe5-6247106856c0/0
[root@server4 0]# ll
total 472
-rw-r--r-- 1 streamsets users    100 Sep  3 18:20 offset.json
-rw-r--r-- 1 streamsets users 257637 Sep  3 18:20 pipelineStateHistory.json
-rw-r--r-- 1 streamsets users  20316 Sep  3 18:20 pipelineState.json
[root@server4 0]# cat offset.json
{
  "version" : 2,
  "offsets" : {
    "$com.streamsets.datacollector.pollsource.offset$" : "v3::1535715633::3661001::1"
  }
}

If this is expected and you know what you are doing (first setup, testing, ..) you can reset the pipeline with graphical interface:

streamsets23
streamsets23

Confirm you will not capture what happened in meanwhile:

streamsets24
streamsets24

Which emtpy offset.json file:

[root@server4 0]# cat offset.json
{
  "version" : 2,
  "offsets" : { }
}

JDBC_16 – Table ” does not exist or PDB is incorrect. Make sure the correct PDB was specified

In JDBC Producer replace for Table Name field:

${record:attribute('tablename')}

By

${record:attribute('oracle.cdc.table')}

Then it failed for:

JDBC_16 - Table 'TABLE01' does not exist or PDB is incorrect. Make sure the correct PDB was specified

In JDBC Producer Errors section I noticed:

oracle.cdc.user: SYS

Because I inserted the record on master database with SYS account, tried with TEST01 account but failed for exact same error…

Finally found the solution when setting Schema Name field to TEST01, in uppercase, because as suggested in Oracle CDC Client documentation Oracle uses all caps for schema, table, and column names by default.

JDBC_00 – Cannot connect to specified database: com.zaxxer.hikari.pool.PoolInitializationException: Exception during pool initialization: The server time zone value ‘CEST’ is unrecognized or represents more than one time zone.

The full error message also contains:

You must configure either the server or JDBC driver (via the serverTimezone configuration property) to use a more specific time zone value if you want to utilize time zone support.

I have been obliged to add an additional JDBC property to set server time zone:

serverTimezone = UTC

Establishing SSL connection without server’s identity verification is not recommended

Thu Sep 13 09:53:39 CEST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

To solve this set in JDBC driver parameters:

useSSL = false

JdbcGenericRecordWriter – No parameters found for record with ID

Complete error message is:

2018-09-12 15:26:41,696 [user:*admin] [pipeline:Oracle-to-Oracle/OracletoOracle7593b814-1185-4829-9fe5-6247106856c0] [runner:0] [thread:ProductionPipelineRunnable-OracletoOracle7593b814-1185-4829-9fe5-6247
106856c0-Oracle-to-Oracle] WARN  JdbcGenericRecordWriter - No parameters found for record with ID  0x0000bc.0001bec8.0018 ::0; skipping
2018-09-12 15:41:31,650 [user:*admin] [pipeline:Oracle-to-Oracle/OracletoOracle7593b814-1185-4829-9fe5-6247106856c0] [runner:0] [thread:ProductionPipelineRunnable-OracletoOracle7593b814-1185-4829-9fe5-6247
106856c0-Oracle-to-Oracle] WARN  JdbcGenericRecordWriter - No parameters found for record with ID  0x0000bd.000057e7.0010 ::1; skipping

For Oracle to MySQL replication I had to manually do column mapping explicitly like this:

streamsets22
streamsets22

References

The post StreamSets Data Collector replication with Oracle, MySQL and JSON appeared first on IT World.

]]>
https://blog.yannickjaquier.com/oracle/streamsets-data-collector-oracle-cdc-client.html/feed 2
Active Session History visualization with Matplotlib and Altair https://blog.yannickjaquier.com/python/active-session-history-visualization-with-matplotlib-and-altair.html https://blog.yannickjaquier.com/python/active-session-history-visualization-with-matplotlib-and-altair.html#comments Wed, 13 Mar 2019 15:38:05 +0000 https://blog.yannickjaquier.com/?p=4469 Preamble After a previous post to install Jupyter Lab and displaying charts suggested by Dominic Gilles it’s now time to move a bit further. Let’s be honest what I REALLY wanted to display is a Cloud Control performance like chart. Means an Active Sessions History visualization in python using one of the many available graphical […]

The post Active Session History visualization with Matplotlib and Altair appeared first on IT World.

]]>

Table of contents

Preamble

After a previous post to install Jupyter Lab and displaying charts suggested by Dominic Gilles it’s now time to move a bit further. Let’s be honest what I REALLY wanted to display is a Cloud Control performance like chart. Means an Active Sessions History visualization in python using one of the many available graphical libraries !

Of course any other performance charts are possible and if you have the query then display them in Jupyter Lab should be relatively straightforward…

I have initially decided to continue with Altair with the ultimate goal to do the same with leading Python graphical library called Matplotlib. At a point in time I expected to use Seaborn as an high level wrapper for Matplotlib but area charts have not been implemented at the time of writing this post (!).

Active Session History visualization with Altair

It all start with a Python query like:

%%sql result1 <<
SELECT
TRUNC(sample_time,'MI') AS sample_time,
DECODE(NVL(wait_class,'ON CPU'),'ON CPU',DECODE(session_type,'BACKGROUND','BCPU','CPU'),wait_class) AS wait_class,
COUNT(*)/60 AS nb
FROM v$active_session_history
WHERE sample_time>=TRUNC(sysdate-interval '1' hour,'MI')
AND sample_time

But my first try gave below error:

result1_df = result1.DataFrame()
alt.Chart(result1_df).mark_area().encode(
    x='sample_time:T',
    y='nb:Q',
    color='wait_class'
).properties(width=700,height=400)

ValueError: Can't clean for JSON: Decimal('0.5833333333333333333333333333333333333333')

I have been able to understand why using below commands. The nb column is seen as an object not a decimal:

>>> result1_df.info()


RangeIndex: 444 entries, 0 to 443
Data columns (total 3 columns):
sample_time    444 non-null datetime64[ns]
wait_class     444 non-null object
nb             444 non-null object
dtypes: datetime64[ns](1), object(2)
memory usage: 10.5+ KB

>>> result1_df.dtypes

sample_time    datetime64[ns]
wait_class             object
nb                    float64
dtype: object

So I converted this column using:

result1_df[['nb']]=result1_df[['nb']].astype('float')

I have set Cloud Control colors using:

colors = alt.Scale(domain=['Other','Cluster','Queueing','Network','Administrative','Configuration','Commit',
                           'Application','Concurrency','System I/O','User I/O','Scheduler','CPU Wait','BCPU','CPU'],
                   range=['#FF69B4','#F5DEB3','#D2B48C','#BC8F8F','#708090','#800000','#FF7F50','#DC143C','#B22222',
                          '#1E90FF','#0000FF','#90EE90','#9ACD32','#3CB371','#32CD32'])

And set axis and series title using something like:

alt.Chart(result1_df).mark_area().encode(
    x=alt.X('sample_time:T', axis=alt.Axis(title='Time')),
    y=alt.Y('nb:Q', axis=alt.Axis(title='Average Active Sessions')),
    color=alt.Color('wait_class', legend=alt.Legend(title='Wait Class'), scale=colors)
).properties(width=700,height=400)

Which gives:

ash_python01
ash_python01

Or without time limitation (update the initial query to do so):

ash_python02
ash_python02

This is overall very easy to display interesting figures and I have to say that it is much much less work than doing it with Highchart and JavaScript.

You might have noticed that versus the Visualizing Active Session History (ASH) to produce Grid Control charts article I'm still lacking the CPU Wait figures. We have seen that query is something like (that I put in a different pandas dataframe):

%%sql result2 <<
SELECT
TRUNC(begin_time,'MI') AS sample_time,
'CPU_ORA_CONSUMED' AS wait_class,
value/100 AS nb
FROM v$sysmetric_history
WHERE group_id=2
AND metric_name='CPU Usage Per Sec'
AND begin_time>=TRUNC(sysdate-interval '1' hour,'MI')
AND begin_time

CPU Wait value is CPU plus background CPU subtracted by CPU used by Oracle, and only if value is positive... The idea is to create from scratch the result dataframe and add it to result1_df database we have seen above.

Creation can be done with:

result3_df=pd.DataFrame(pd.date_range(start=result1_df['sample_time'].min(), end=result1_df['sample_time'].max(), freq='T'),
             columns=['sample_time'])
result3_df['wait_class']='CPU Wait'
result3_df['nb']=float(0)

Or:

result3_df=pd.DataFrame({'sample_time': pd.Series(pd.date_range(start=result1_df['sample_time'].min(), 
                                                                end=result1_df['sample_time'].max(), freq='T')),
                         'wait_class': 'CPU Wait',
                         'nb': float(0)})

Then the computation is done with below code. Of course you must handle the fact that sometime there is no value for CPU or BCPU:

for i in range(1,int(result3_df['sample_time'].count())+1):
    time=result3_df['sample_time'][i-1]
    #result=float((result1_df.query('wait_class == "CPU" and sample_time==@time').fillna(0))['nb'])
    cpu=0
    bcpu=0
    cpu_ora_consumed=0
    #result1=result1_df.loc[(result1_df['wait_class']=='CPU') | (result1_df['wait_class']=='BCPU')]['nb']
    result1=(result1_df.query('wait_class == "CPU" and sample_time==@time'))['nb']
    result2=(result1_df.query('wait_class == "BCPU" and sample_time==@time'))['nb']
    result3=(result2_df.query('wait_class == "CPU_ORA_CONSUMED" and sample_time==@time'))['nb']
    if not(result1.empty):
        cpu=float(result1)
    if not(result2.empty):
        bcpu=float(result2)
    if not(result3.empty):
        cpu_ora_consumed=float(result3)
    cpu_wait=cpu+bcpu-cpu_ora_consumed
    #print('{:d} - {:f},{:f},{:f} - {:f}'.format(i,cpu,bcpu,cpu_ora_consumed,cpu_wait))
    if cpu_wait<0:
        cpu_wait=0.0
    #result3_df['nb'][i-1]=cpu_wait
    result3_df.loc[i:i,('nb')]=cpu_wait
print('Done')

For performance reason and to avoid a warning you cannot use what's below to set the value:

result3_df['nb'][i-1]=cpu_wait

Or you get:

/root/python37_venv/lib64/python3.7/site-packages/ipykernel_launcher.py:24: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

Finally I have decided to concatenate result1_df dataframe to resul3_df dataframe and sort it on sample_time (if you do not sort it Altair might produce strange result):

result3_df=pd.concat([result1_df, result3_df])
result3_df=result3_df.sort_values(by='sample_time')

To get final chart with CPU Wait figures. I have also added a tooltip to display wait class and value (but at the end I believe the feature is a bit buggy, sample_time is always set at same value in tooltip, or I don't know how to use it):

colors = alt.Scale(domain=['Other','Cluster','Queueing','Network','Administrative','Configuration','Commit',
                           'Application','Concurrency','System I/O','User I/O','Scheduler','CPU Wait','BCPU','CPU'],
                   range=['#FF69B4','#F5DEB3','#D2B48C','#BC8F8F','#708090','#800000','#FF7F50','#DC143C','#B22222',
                          '#1E90FF','#0000FF','#90EE90','#9ACD32','#3CB371','#32CD32'])
alt.Chart(result3_df).mark_area().encode(
    x=alt.X('sample_time:T', axis=alt.Axis(format='%d-%b-%Y %H:%M', title='Time')),
    y=alt.Y('nb:Q', axis=alt.Axis(title='Average Active Sessions')),
    color=alt.Color('wait_class', legend=alt.Legend(title='Wait Class'), scale=colors),
    tooltip=['wait_class',alt.Tooltip(field='sample_time',title='Time',type='temporal',format="%d-%b-%Y %H:%M"),alt.Tooltip('nb',format='.3')]
).properties(width=700,height=400)
ash_python03
ash_python03

Are we done ? Almost... If you look closely to a Cloud Control Active Session History (ASH) chart you will see that the stack order of the wait class is not random nor ordered by wait class name. First come CPU, then Scheduler then User I/O and so on... So how to do that with Altair ? Well at the time of writing this blog post it is not possible. The feature request has been validated to be added and they alternatively propose a trick using calculate.

You can also use the order property of the stack chart but obviously it will order wait class by name (default behavior by the way). You can simply change the order with something like:

colors = alt.Scale(domain=['Other','Cluster','Queueing','Network','Administrative','Configuration','Commit',
                           'Application','Concurrency','System I/O','User I/O','Scheduler','CPU Wait','BCPU','CPU'],
                   range=['#FF69B4','#F5DEB3','#D2B48C','#BC8F8F','#708090','#800000','#FF7F50','#DC143C','#B22222',
                          '#1E90FF','#0000FF','#90EE90','#9ACD32','#3CB371','#32CD32'])
alt.Chart(result3_df).mark_area().encode(
    x=alt.X('sample_time:T', axis=alt.Axis(format='%d-%b-%Y %H:%M', title='Time')),
    y=alt.Y('nb:Q', axis=alt.Axis(title='Average Active Sessions')),
    color=alt.Color('wait_class', legend=alt.Legend(title='Wait Class'), scale=colors),
    tooltip=['wait_class',alt.Tooltip(field='sample_time',title='Time',type='temporal',format="%d-%b-%Y %H:%M"),'nb'],
    order = {'field': 'wait_class', 'type': 'nominal', 'sort': 'descending'}
).properties(width=700,height=400)

To really sort the wait class by the order you like till they allow it simply in the encode function, as suggested, you have to use the transform_calculate function and to be honest I have fight a lot with Vega expression to make it working:

from altair import datum, expr
colors = alt.Scale(domain=['Other','Cluster','Queueing','Network','Administrative','Configuration','Commit',
                           'Application','Concurrency','System I/O','User I/O','Scheduler','CPU Wait','BCPU','CPU'],
                   range=['#FF69B4','#F5DEB3','#D2B48C','#BC8F8F','#708090','#800000','#FF7F50','#DC143C','#B22222',
                          '#1E90FF','#0000FF','#90EE90','#9ACD32','#3CB371','#32CD32'])
#kwds={'calculate': 'indexof(colors.domain, datum.wait_class)', 'as': 'areaorder' }
kwds={'calculate': "indexof(['Other','Cluster','Queueing','Network','Administrative','Configuration','Commit',\
      'Application','Concurrency','System I/O','User I/O','Scheduler','CPU Wait','BCPU','CPU'], datum.wait_class)",
      'as': "areaorder" }
alt.Chart(result3_df).mark_area().encode(
    x=alt.X('sample_time:T', axis=alt.Axis(format='%d-%b-%Y %H:%M', title='Time')),
    y=alt.Y('nb:Q', axis=alt.Axis(title='Average Active Sessions')),
    color=alt.Color('wait_class', legend=alt.Legend(title='Wait Class'), scale=colors),
    tooltip=['wait_class',alt.Tooltip(field='sample_time',title='Time',type='temporal',format="%d-%b-%Y %H:%M"),'nb']
    #,order='siteOrder:Q'
    ,order = {'field': 'areaorder', 'type': 'ordinal', 'sort': 'descending'}
).properties(width=700,height=400).transform_calculate(**kwds)
ash_python04
ash_python04

Here it is ! I have not been able to use the colors Altair scale variable in the computation (line in comment) so I have just copy/paste it...

Active Session History visualization with Matplotlib

Matplotlib claim to be the leading visualization library available in Python. With the well known drawback of being a bit complex to handle...

I will start from the result3_df we have just created above. Matplotlib is not exactly ingesting the same format as Altair and you are obliged like with Highchart to pivot your result. While I was asking me how to do I have touched the beauty of Pandas as it is already there with pivot function:

result3_df_pivot=result3_df.pivot(index='sample_time', columns='wait_class', values='nb').fillna(0)

I also use fillna to fill non existing value (NaN) as you do not have session waiting for all wait class at a given time.

Then you need to finally import Matplotlib (if not yet installed do a pip install matplotlib) and you need to configure few things. The minimum required is figure size that you have to specify in inches. If like me you live in a metric world you would wonder how do I specify size in pixels, for example. I have found below trick:

>>> import pylab
>>> pylab.gcf().get_dpi()
72.0

Then you can divide size in pixels by 72 and use it in matplotlib.rc procedure:

import matplotlib
# figure size in inches
matplotlib.rc('figure', figsize = (1000/72, 500/72))
# Font size to 14
matplotlib.rc('font', size = 14)
# Do not display top and right frame lines
#matplotlib.rc('axes.spines', top = False, right = False)
# Remove grid lines
#matplotlib.rc('axes', grid = False)

To get all possible parameters as well as their default values you can use:

>>> matplotlib.rc_params()
RcParams({'_internal.classic_mode': False,
          'agg.path.chunksize': 0,
          'animation.avconv_args': [],
          'animation.avconv_path': 'avconv',
          'animation.bitrate': -1,
          'animation.codec': 'h264',
.
.
.

You can also set a value using rcParams procedure and do not work with groups using rc procedure:

matplotlib.rcParams['figure.figsize']= [6.4, 4.8]

As the most simple example you can use the Matplotlib wrapper of Pandas and do a simple:

result3_df_pivot.plot.area()

Or if you want to save the figure in a file (on the machine where is running Jupyter Lab):

import matplotlib.pyplot as plt
plt.figure()
result3_df_pivot.plot.area()
plt.savefig('/tmp/visualization.png')

It gives the below not so bad result for lowest possible effort. Of course we have not chosen the colors and customize nothing but for a quick and dirty display it already gives a lot of information:

ash_python05
ash_python05

Of course we want more and for this we will have to enter in Matplotlib internals. The idea is to define a Pandas series with categories and related colors. Then build the data, colors and labels based of which wait class category we have. Finally a bit of cosmetic with axes label and chart label as well as label in top right corner:

from matplotlib.dates import DayLocator, HourLocator, DateFormatter, drange
colors_ref = pd.Series({
    'Other': '#FF69B4',
    'Cluster': '#F5DEB3',
    'Queueing': '#D2B48C',
    'Network': '#BC8F8F',
    'Administrative': '#708090',
    'Configuration': '#800000',
    'Commit': '#FF7F50',
    'Application': '#DC143C',
    'Concurrency': '#B22222',
    'System I/O': '#1E90FF',
    'User I/O': '#0000FF',
    'Scheduler': '#90EE90',
    'CPU Wait': '#9ACD32',
    'BCPU': '#3CB371',
    'CPU': '#32CD32'
})
data=[]
labels=[]
colors=[]
for key in ('CPU','BCPU','CPU Wait','Scheduler','User I/O','System I/O','Concurrency','Application','Commit',
            'Configuration','Administrative','Network','Queueing','Cluster','Other'):
    if key in result3_df_pivot.keys():
        data.append(result3_df_pivot[key].values)
        labels.append(key)
        colors.append(colors_ref[key])

figure, ax = plt.subplots()
plt.stackplot(result3_df_pivot.index.values,
              data,
              labels=labels,
              colors=colors)
# format the ticks
#ax.xaxis.set_major_locator(years)
#ax.xaxis.set_major_formatter(yearsFmt)
#ax.xaxis.set_minor_locator(months)
#ax.autoscale_view()
#ax.xaxis.set_major_locator(DayLocator())
#ax.xaxis.set_minor_locator(HourLocator(range(0, 25, 6)))
#ax.fmt_xdata = DateFormatter('%Y-%m-%d %H:%M:%S')
ax.xaxis.set_major_formatter(DateFormatter('%Y-%m-%d %H:%M:%S'))
figure.autofmt_xdate()
plt.legend(loc='upper right')
figure.suptitle('Cloud Control', fontsize=20)
plt.xlabel('Time', fontsize=16)
plt.ylabel('Average Active Sessions', fontsize=16)
plt.show()
figure.savefig('/tmp/visualization.png')

Looks really close to a default Cloud Control performance active session history (ASH) chart:

ash_python06
ash_python06

References

The post Active Session History visualization with Matplotlib and Altair appeared first on IT World.

]]> https://blog.yannickjaquier.com/python/active-session-history-visualization-with-matplotlib-and-altair.html/feed 2