Ceph对象存储网关安装配置



  • 于已部署好的Ceph集群,部署一个网关服务器,进行对象存储服务。全文分别对依赖包安装、配置、验证做了详细说明。该过程经过实际验证。

    引言

    基于已部署好的Ceph集群,部署一个网关服务器,进行对象存储服务。操作系统CentOS6.5 CEPH0.94.3其实基于librados可以直接进行访问,但是我看了百度,UCLOUD的对象存储,用户在网页上进行文件的上传、下载时,都通过web服务器间接和存储集群打交道,进行了一层隔离,而不是直接和集群进行通信操作。我得理解是便于访问控制以及隔离。

    1.依赖包安装

    Ceph rados-gateway依赖Apache和FastCGI, 用户的请求先到web服务器,再走rados-gateway进入集群之中。

    1.1 安装Apache服务

    sudo yum install httpd
    Package httpd-2.2.15-47.el6.centos.x86_64 already installed and latest version
    

    1.2 配置http服务器

    sudo vim /etc/httpd/conf/httpd.conf
    

    将ServerName的注释号去掉,添加上自己网关服务器的IP地址

    272 # If your host doesn't have a registered DNS name, enter its IP address here.
    273 # You will have to access it by its address anyway, and this will make 
    274 # redirections work in a sensible way.
    275 #
    276 ServerName 101.67.163.34:80
    

    在配置中增加如下信息,加载mod_proxy_fcgi

    <IfModule !proxy_fcgi_module>
    LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
    </IfModule>
    

    此处需注意,需要将该段内容加载LoadModule系列的后面,否则会报如下错误:

    sudo service httpd start
    Starting httpd: httpd: Syntax error on line 129 of /etc/httpd/conf/httpd.conf: Cannot load /etc/httpd/modules/mod_proxy_fcgi.so into server: /etc/httpd/modules/mod_proxy_fcgi.so: undefined symbol: ap_proxy_release_connection
    

    修改配置中的LISTEN字段,将网关所在主机的IP地址添加进去

    # Listen: Allows you to bind Apache to specific IP addresses and/or
    # ports, in addition to the default. See also the <VirtualHost>
    # directive.
    #
    # Change this to Listen on specific IP addresses as shown below to 
    # prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
    #
    Listen 101.67.163.34:80
    #Listen 80
    

    1.3 SSL支持 (此处是否必须不是很清楚,只是按照官方文档走)

    秘钥文件生成

    sudo yum install mod_ssl openssl
    openssl genrsa -out ca.key 2048
    openssl req -new -key ca.key -out ca.csr
    openssl x509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt
    

    文件目录放置sudo cp ca.crt /etc/pki/tls/certs

    sudo cp ca.key /etc/pki/tls/private/ca.key
    sudo cp ca.csr /etc/pki/tls/private/ca.csr
    

    配置文件修改/etc/httpd/conf.d/ssl.conf.

    SSLCertificateFile  /etc/pki/tls/certs/ca.crt
    SSLCertificateKeyFile /etc/pki/tls/private/ca.key
    

    重启httpd服务sudo service httpd restart

    1.4 网关服务安装

    sudo yum install ceph-radosgw
    

    至此,相关依赖包安装完毕

    2. CEPH网关服务配置

    ceph网关其实是ceph集群的一个客户端,用户通过这个网关间接访问ceph集群,作为客户端,它需要准备如下内容:
    网关名称,此处用gateway称呼
    一个可以访问存储集群的用户以及对应的KEYRING
    数据资源池,这个由ceph集群提供
    为网关服务示例准备一个数据存放空间
    在ceph.conf配置文件中设置gateway信息

    2.1 创建访问用户及权限设置

    创建gateway keyring,一开始该文件为空

    sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
    sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
    

    创建网关用户名以及key 此处名字为 client.radosgw.gateway

    sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key
    

    为KEYRING添加权限

    sudo ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
    

    将key添加到集群中

    sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring
    

    将相关的KEYRING文件拷贝到rados-gateway所在的主机 /etc/ceph/目录下

    2.2 数据资源池创建

    .rgw.root
    .rgw.control
    .rgw.gc
    .rgw.buckets
    .rgw.buckets.index
    .rgw.buckets.extra
    .log
    .intent-log
    .usage
    .users
    .users.email
    .users.swift
    .users.uid
    
    [root@gnop029-ct-zhejiang_wenzhou-16-34 conf]# ceph osd lspools
    4 rbd,6 pool-1,7 pool-2,8 .rgw,9 .rgw.root,10 .rgw.control,11 .rgw.gc,12 .rgw.buckets,13 .rgw.buckets.index,14 .log,15 .intent-log,16 .usage,17 .users,18 .users.email,19 .users.swift,20 .users.uid
    

    2.3 将网关配置信息添加到集群配置中

    [client.radosgw.gateway]
    host=ceph-24
    keyring=/etc/ceph/ceph.client.radosgw.keyring
    rgw socket path=/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
    log file=/var/log/radosgw/client.radosgw.gateway.log
    rgw frontends=fastcgi socket_port=9000 socket_host=0.0.0.0
    rgw print continue=false
    

    2.4 目录及权限调整

    创建数据目录

    sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway   
    调整apache运行权限
    sudo chown apache:apache /var/run/ceph
    

    调整日志权限

    sudo chown apache:apache /var/log/radosgw/client.radosgw.gateway.log
    

    启动网关服务sudo /etc/init.d/ceph-radosgw start

    2.5 网关配置文件

    一个配置文件,用于web server和FastCGI之间的交互
    sudo vi /etc/httpd/conf.d/rgw.conf

    <VirtualHost *:80>
    ServerName 101.67.163.34
    DocumentRoot /var/www/html
     
    ErrorLog /var/log/httpd/rgw_error.log
    CustomLog /var/log/httpd/rgw_access.log combined
     
    RewriteEngine On
    RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
     
    SetEnv proxy-nokeepalive 1
     
    ProxyPass / fcgi://101.67.163.34:9000/
    </VirtualHost>
    

    其中部分地方是要根据实际情况填写

    2.6 用户创建

    radosgw-admin user create --uid=xuwenping --display-name="ceph xuwenping" --email=xuwenping@dnion.com
    {
        "user_id": "xuwenping",
        "display_name": "ceph xuwenping",
        "email": "xuwenping@dnion.com",
        "suspended": 0,
        "max_buckets": 1000,
        "auid": 0,
        "subusers": [],
        "keys": [
            {
                "user": "xuwenping",
                "access_key": "4J3GD7GJIJKSDCVS1I9T",
                "secret_key": "yfmxvzQdWT4EmVDijOFp6oNt4kZ25y9wRVARas4I"
            }
        ],
        "swift_keys": [],
        "caps": [],
        "op_mask": "read, write, delete",
        "default_placement": "",
        "placement_tags": [],
        "bucket_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        },
        "temp_url_keys": []
    }
    

    创建SWIFT类型USER

    sudo radosgw-admin subuser create --uid=xuwenping --subuser=xuwenping :swift --access=full
     
    2015-10-10 14:19:19.854951 7f402eadc8a0  0 max_buckets=1000 specified=0
    {
        "user_id": "xuwenping",
        "display_name": "ceph xuwenping",
        "email": "xuwenping@dnion.com",
        "suspended": 0,
        "max_buckets": 1000,
        "auid": 0,
        "subusers": [
            {
                "id": "xuwenping:swift",
                "permissions": "full-control"
            }
        ],
        "keys": [
            {
                "user": "xuwenping",
                "access_key": "4J3GD7GJIJKSDCVS1I9T",
                "secret_key": "yfmxvzQdWT4EmVDijOFp6oNt4kZ25y9wRVARas4I"
            },
            {
                "user": "xuwenping:swift",
                "access_key": "PEIT99BBWMZP31BD6S3I",
                "secret_key": ""
            }
        ],
        "swift_keys": [
            {
                "user": "xuwenping:swift",
                "secret_key": "qWHPhvUy4md1XSa2PSbcxUyMU5YXodlqxt0ZC2hn"
            }
        ],
        "caps": [],
        "op_mask": "read, write, delete",
        "default_placement": "",
        "placement_tags": [],
        "bucket_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        },
        "temp_url_keys": []
    }
    

    2.7 实际验证

    编写了一段python代码,用于访问网关,并创建bucket,并通过list方法罗列出当前所有的bucket (官方示例)
    依赖库安装

    sudo yum install python-boto
    
    import boto
    import boto.s3.connection
    access_key = '4J3GD7GJIJKSDCVS1I9T'
    secret_key = 'yfmxvzQdWT4EmVDijOFp6oNt4kZ25y9wRVARas4I'
    conn = boto.connect_s3(
    aws_access_key_id = access_key,
    aws_secret_access_key = secret_key,
    host = '101.67.163.34',
    is_secure=False,
    calling_format = boto.s3.connection.OrdinaryCallingFormat(),
    )
    bucket = conn.create_bucket('my-new-bucket')
    for bucket in conn.get_all_buckets():
            print "{name}\t{created}".format(
                    name = bucket.name,
                    created = bucket.creation_date,
    )
    

    运行结果

    [root@gnop029-ct-zhejiang_wenzhou-16-34 ceph-rados]# python s3test.py 
    my-new-bucket   2015-10-10T06:23:48.000Z
    

    至此,Ceph集群的对象存储网关安装设置完毕

    附:

    有时候通过yum安装软件是报如下错误:

    Downloading Packages:
    warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
    Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
     
     
    GPG key retrieval failed: [Errno 14] Could not open/read file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
    

    在执行 安装命令时带上如下参数即可:

    yum install mod_proxy_fcgi   --nogpgcheck
    

    原文链接: http://my.oschina.net/myspaceNUAA/blog/515261


登录后回复
 

与 青云QingCloud 社区 的连接断开,我们正在尝试重连,请耐心等待