A sample text widget

Etiam pulvinar consectetur dolor sed malesuada. Ut convallis euismod dolor nec pretium. Nunc ut tristique massa.

Nam sodales mi vitae dolor ullamcorper et vulputate enim accumsan. Morbi orci magna, tincidunt vitae molestie nec, molestie at mi. Nulla nulla lorem, suscipit in posuere in, interdum non magna.

WARN appmaster.SliderAppMaster – python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory” when starting LLAP while loading shared libraries

Cause

This field indicates the primary cause of the issue solved in this KB article.
This issue occurs because, the Hive user launching the Slider application picks up environment variables from Ambari-agent for newer version of Python configured, while the Slider application looks for the new Python libraries in the location of the older version of Python.
Solution

This field indicates the solution.
To help LLAP find the Libraries for the correct version of Python, do the following to resolve the issue on all Slider client nodes:

  1. Add the path of Python libraries in /etc/ld.so.conf  as in the following example (pick the path of python libraries from /var/lib/ambari-agent/ambari-env.sh):
    slidernode$ cat /etc/ld.so.conf
    include ld.so.conf.d/*.conf
    /opt/rh/python27/root/usr/lib64
  2. Run the following command to create the necessary links and cache to the most recent shared libraries found in the directories specified in /etc/ld.so.conf:
    slidernode$ ldconfig
  3. Restart Ambari agent on all the Slider client nodes.
  4. Refresh configs for Slider clients from Ambari.
  5. Restart all Hive services from Ambari.

Reference : https://stackoverflow.com/questions/20842732/libpython2-7-so-1-0-cannot-open-shared-object-file-no-such-file-or-directory

How to kill a Hive query that does not have a YARN application_ID?

ometimes, when running CREATE and SELECT queries, YARN application ID is not created. These queries may have to be killed manually. To achieve this, do the following: 

  1. Take the query ID from the Beeline / Hive CLI console:
    hive> select * from firsttable;
    Query ID = mapr_20170523134547_2c04d119-5495-401e-b7b7-2ad87f0f5627
    Total jobs = 1
    Launching Job 1 out of 1
  2. Use “kill query” command, which is available from HDP 2.6.3:
    KILL QUERY "<queryid1>" "<queryid2>"

Migrate/Transfer/Copy AWS Data from volumes or S3 to Hadoop

Hadoop distcp s3n://bucketname/directoryname/test.csv /user/myuser/mydirectory/

 

Detail information can be found at following link.

https://hortonworks.github.io/hdp-aws/s3-copy-data/index.html

Placing Namenode in safemode and Perform checkpoint

Normally in real environment secondary node performs checkpoint operation and keep syncing with active namenode to keep latest FSimage.  In HA environment active namenode writes edit log into  journal node and standby node go and get the edit log and create new FSimage.

[hdfs@datanode1 ~]$ hdfs dfsadmin -safemode get
Safe mode is OFF

[hdfs@datanode1 ~]$ hdfs dfsadmin -safemode enter
Safe mode is ON

[hdfs@datanode1 ~]$ hdfs dfsadmin -saveNamespace       (checkpoint command)
Save namespace successful

[hdfs@datanode1 ~]$ hdfs dfsadmin -safemode leave
Safe mode is OFF

 

 

RHEL 5 vs RHEL 7 comparision

Command changes for common tasks

System basics

Task RHEL5 RHEL7
View subscription information /etc/sysconfig/rhn/systemid subscription-manager identity
Configure subscription rhn_register subscription-manager
rhn_register
View system profile sosreport
dmidecode
hwbrowser
sosreport
dmidecode
lstopo
lscpu

Basic configuration

Task RHEL5 RHEL7
Graphical configuration tools system-config-* gnome-control-center
Configure network system-config-network nmcli
nmtui
nm-connection-editor
Configure system language system-config-language localectl
Configure time and date system-config-date
date
timedatectl
date
Synchronize time and date ntpdate
/etc/ntp.conf
timedatectl
/etc/chrony.conf
ntpdate
Configure keyboard system-config-keyboard localectl

Jobs and services

Task RHEL5 RHEL7
List all services chkconfig –list
ls /etc/init.d/
systemctl -at service
ls /etc/systemd/system/*.service
ls /usr/lib/systemd/system/*.service
List running services service –status-all systemctl -t service –state=active
Start/stop service service name start
service name stop
systemctl start name.service
systemctl stop name.service
Enable/disable service chkconfig name on
chkconfig name off
systemctl enable name.service
systemctl disable name.service
View service status service name status systemctl status name.service
Check if service is enabled chkconfig name –list systemctl is-enabled name
Create new service file or modify configuration chkconfig –add systemctl daemon-reload
/etc/systemd/system/*.service
View run level/target runlevel
who -r
systemctl get-default
who -r
Change run level/target /etc/inittab
init run_level
systemctl isolate name.target
systemctl set-default
Configure logging /etc/syslog.conf /etc/rsyslog.conf
/etc/rsyslog.d/*.conf
/var/log/journal
systemd-journald.service
View logs /var/log /var/log
journalctl

Kernel, boot, and hardware

Task RHEL5 RHEL7
Single user/rescue mode append 1 or s or init=/bin/bash to kernel cmdline append rd.break or init=/bin/bash to kernel cmdline
Shut down system shutdown shutdown
systemctl shutdown
Power off system poweroff poweroff
systemctl poweroff
Halt system halt halt
systemctl halt
Reboot system reboot reboot
systemctl reboot
Configure default run level/target /etc/inittab systemctl set-default
Configure GRUB bootloader /boot/grub/grub.conf /etc/default/grub
grub2-mkconfig
grub-set-default
View hardware configured hwbrowser lshw (in EPEL)

Software management

Task RHEL5 RHEL7
Install software yum install
yum groupinstall
yum install
yum group install
View software info yum info
yum groupinfo
yum info
yum group info

File systems, volumes, and disks

Task RHEL5 RHEL7
Default file system ext3 xfs
Create/modify disk partitions fdisk
parted
fdisk
gdisk
parted
ssm create
Format disk partition mkfs.filesystem_type (ext4, xfs)
mkswap
mkfs.filesystem_type (ext4, xfs)
mkswap
ssm create
Defragment disk space copy data to new file system
fsck (look for ‘non-contiguous inodes’)
copy data to new file system
fsck (look for ‘non-contiguous inodes’)
xfs_fsr
Mount storage mount
/etc/fstab
mount
/etc/fstab
ssm mount
Create physical volume pvcreate pvcreate
ssm create (if backend is lvm)
Create volume group vgcreate vgcreate
ssm create (if backend is lvm)
Create logical volume lvcreate lvcreate
ssm create (if backend is lvm)
Enlarge volumes formatted with default file system vgextend
lvextend
resize2fs
vgextend
lvextend
xfs_growfs
ssm resize
Shrink volumes formatted with default file system resize2fs
lvreduce
vgreduce
XFS cannot currently be shrunk; copy desired data to a smaller file system.
Check/repair file system fsck fsck
ssm check
Configure NFS share /etc/exports
service nfs reload
/etc/exports
systemctl reload nfs.service

Networking

Task RHEL5 RHEL7
Configure name resolution /etc/hosts
/etc/resolv.conf
/etc/hosts
/etc/resolv.conf
nmcli con mod
Configure hostname /etc/sysconfig/network hostnamectl
/etc/hostname
nmtui
View network interface info ip addr
ifconfig
brctl
ip addr
nmcli dev show
teamdctl
brctl
bridge
Configure network interface /etc/sysconfig/network-scripts/ifcfg-* /etc/sysconfig/network-scripts/ifcfg-*
nmcli con [add|mod|edit]
nmtui
nm-connection-editor
Configure routes ip route add
system-config-network
/etc/sysconfig/route-iface
ip route add
nmcli
nmtui
nm-connection-editor
/etc/sysconfig/route-iface
Configure firewall iptables and ip6tables
/etc/sysconfig/ip*tables
firewall-cmd
firewall-config
View ports/sockets ss
lsof
netstat
ss
lsof

Resource management

Task RHEL5 RHEL7
View system usage top
ps
sar
iostat
netstat
vmstat
mpstat
numastat
top
ps
sar
iostat
ss
vmstat
mpstat
numastat
tuna
View disk usage df df
iostat

RHEL 7 – Changed mount behavior at boot – Nofail

Earlier versions of Red Hat Enterprise Linux booted regardless of whether all partitions specified in /etc/fstab could be mounted. This could result in a system appearing “up” and healthy, while booting without required partitions.
To prevent this situation, in Red Hat Enterprise Linux 7, if a partition defined in /etc/fstab cannot be mounted at boot, boot fails. If a partition should not cause boot to fail in the event that it cannot be mounted, use the new nofail parameter in /etc/fstab.
/dev/critical     /critical   xfs   defaults          1  2
/dev/optional     /optional   xfs   defaults,nofail   1  2
In this example, the device mounted at /optional would not cause boot to fail if it could not be mounted successfully.