Python操作hadoop、hive、hbase方法

2018/05/04 大数据 阅读次数:

摘要:基于业务需要,需要对hadoop、hive、hbase操作,平时都是在命令行或图形界面操作,现在需要在代码块中连接操作,第一次装遇到不少坑,记录下正确的方法

系统:deepin15.7 python版本:python3.5;

连接hive方法,采用pyhive连接 需要安装的依赖,好多实在安装依赖时卡主了 按照以下步骤安装,顺利完成

sudo apt-get install sasl2-bin
sudo apt-get install libsasl2-dev
pip install pyhs2
pip install pyhive
pip install thrift_sasl

from pyhive import hive
conn = hive.Connection(host="xxxxx", port=10000, username='xxxx', database='default')
正常操作就ok
  1. 连接hbase方法,当你hive连接没问题,那连接hbase没什么问题

需要安装依赖关系,按照一下步骤安装

pip install thrift   #这一步,就是上面 pip install thrift_sasl,如果装了,就不用再装
pip install happybase  #happybase 连接跟我们正常连接数据库一样

from happybase import Connection
connection = Connection(host="192.168.0.156",port=9090,timeout=None,autoconnect=True,table_prefix=None,table_prefix_separator=b'_',compat='0.98', transport='buffered',protocol='binary')

host:主机名
port:端口
timeout:超时时间
autoconnect:连接是否直接打开
table_prefix:用于构造表名的前缀
table_prefix_separator:用于table_prefix的分隔符
compat:兼容模式
transport:运输模式
protocol:协议
  1. 连接hdfs方法,

安装 pip install hdfs   

from hdfs import Client
Client("url",root="/",proxy=None,timeout=None,session=None)
urlip:端口
root:制定的hdfs根目录
proxy:制定登陆的用户身份
timeout:设置的超时时间
session:连接标识
  1. python 操作hadoop,进行分词统计  
mapper.py  
    
        import sys
        for line in sys.stdin:      # 接收屏幕的值
            words = line.split()    # 以空格切割
            for word in words:      # 输出信息
                print('{}:{}'.format(word, 1))  # 用:号隔开
    reducer.py  
        
        import sys
        curr_word = None
        curr_count = 0
        
        for line in sys.stdin:
            word, count = line.split(':')   # 因为mapper中用:隔开,这里用:分割
            count = int(count)              # 获取数量是字符串,需要转换成int
            if word == curr_word:           # 统计个数
                curr_count += count
            else:
                if curr_word:
                    print('{}\t{}'.format(curr_word, curr_count))
                curr_word = word
                curr_count = count
        
        if curr_word == word:
            print('{}\t{}'.format(curr_word, curr_count))

    同一个目录下,创建123.txt文件,执行一下命令  
    
        cat 123.txt | python mapper.py | sort -t 1 | python reducer.py 
        这条命令的意思,将123.txt文件内容输出到屏幕,将结果作为参数,给mapper.py,输出的结果进行排序,间隔符1;将输出的结果再出作为参数给reducer.py
        

最新python3.5+,python3.6+;连接hive

之前的连接方法死活不好使  

sudo apt-get install sasl2-bin
sudo apt-get install libsasl2-dev
PyHive
pip install --upgrade pip
pip install sasl
pip install --upgrade thrift
pip install thrift-sasl
pip install PyHive
impyla
pip install --upgrade pip
pip install pure-sasl
pip install thrift_sasl==0.2.1
pip install thrift
pip install impyla
效果图

报错

thriftpy.transport.TTransportException: TTransportException(message=”Could not start SASL: b’Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2’”, type=1)

hive-site.xml,默认是NONE,改成NOSASL

<property>
    <name>hive.server2.authentication</name>
    <value>NOSASL</value>
    <description>
      Expects one of [nosasl, none, ldap, kerberos, pam, custom].
      Client authentication types.
        NONE: no authentication check
        LDAP: LDAP/AD based authentication
        KERBEROS: Kerberos/GSSAPI authentication
        CUSTOM: Custom authentication provider
                (Use with property hive.server2.custom.authentication.class)
        PAM: Pluggable authentication module
        NOSASL:  Raw transport
    </description>
  </property>
安装sasl报错 CentOS:

 sasl/saslwrapper.h: In member function bool saslwrapper::ClientImpl::getSSF(int*):
    sasl/saslwrapper.h:390: 错误:‘conn’在此作用域中尚未声明
    sasl/saslwrapper.h:390: 错误:‘SASL_SSF’在此作用域中尚未声明
    sasl/saslwrapper.h:390: 错误:‘sasl_getprop’在此作用域中尚未声明
    sasl/saslwrapper.h:391: 错误:‘SASL_OK’在此作用域中尚未声明
    sasl/saslwrapper.h: In member function void saslwrapper::ClientImpl::addCallback(long unsigned int, void*):
    sasl/saslwrapper.h:407: 错误:‘callbacks’在此作用域中尚未声明
    sasl/saslwrapper.h: In member function void saslwrapper::ClientImpl::setError(const std::string&, int, const std::string&, const std::string&):
    sasl/saslwrapper.h:419: 错误:‘conn’在此作用域中尚未声明
    sasl/saslwrapper.h:420: 错误:‘sasl_errdetail’在此作用域中尚未声明
    sasl/saslwrapper.h:422: 错误:‘sasl_errstring’在此作用域中尚未声明
    sasl/saslwrapper.h: At global scope:
    sasl/saslwrapper.h:434: 错误:变量或字段‘interact’声明为 void
    sasl/saslwrapper.h:434: 错误:‘sasl_interact_t’在此作用域中尚未声明
    sasl/saslwrapper.h:434: 错误:‘prompt’在此作用域中尚未声明
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/root/.virtualenvs/django/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-i87ths29/sasl/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-5e0ytsoh/install-record.txt --single-version-externally-managed --compile --install-headers /root/.virtualenvs/django/include/site/python3.5/sasl" failed with error code 1 in /tmp/pip-install-i87ths29/sasl/

解决方法:  

Debian/Ubuntu: apt-get install python-dev libsasl2-dev gcc 
CentOS/RHEL: yum install gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64

Search

    Table of Contents