StorNext FS 配置
(Red Hat Enterprise Linux/x86 AS 4 +win2K)
一、环境描述:
硬件描述:
1. Brocade 3950
2. S2A 8000(本例中3个LUN,分别为MetaData(30GB)、Journal(100MB)、数据)
3. DELL 2650 (内存 2G,建议4G)
软件描述:
1. Red Hat Enterprise Linux/x86 AS 4 update 1 (2.6.9-11)
2. StorNext FS Server/Client 软件
二、StorNext FS系统配置
1.准备工作
1.1配置网络
# vi /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.0.5.25 mds1 //这里为两台MDS
10.0.5.26 mds2
1.2 安装Emulex LP8000/982驱动和应用程序
安装驱动程序 tar xzf lpfc-<driver version>.tar.gz //解压安装包
如:#tar zxvf lpfc_2.6_driver_kit-8.0.16.17-2.tar.gz
#cd lpfc_2.6_driver_kit-8.0.16.17-2
#./lpfc-install //执行安装脚本
#reboot //重起服务器
注意:Red Hat Enterprise Linux/x86 4 updata 1 内核中已经加载Emulex LP8000/982驱动,可以不做这一步。
如果在安装驱动程序中,报IO和IRQ参数错,检查Firmware版本,升级为最新的版本。
安装emulex hbanyware应用程序
#tar xvf elxlinuxapps-2.1a25-8.0.16.17-1-1.tar
#./install //安装emulex hnanyware
#/usr/sbin/hbanyware/hbanyware //运行emulex hnanyware,查看WWN,硬件信息。
注意:需要使用emulex hnanyware时需要安装lpfc_2.6_driver_kit-8.0.16.17-2.tar.gz程序,而linux 系统自带的LPFC的驱动不能被emulex hnanyware所以识别。
1.3 效验HBA卡加载。
通过cat /proc/scsi/scsi 命令查看HBA卡检测到的磁盘信息
如:# cat /proc/scsi/scsi
Host: scsi0 Channel: 00 Id: 06 Lun: 00
Vendor: PE/PV Model: 1×5 SCSI BP Rev: 1.1
Type: Processor ANSI SCSI revision: 02
Host: scsi2 Channel: 00 Id: 00 Lun: 03
Vendor: DDN Model: S2A 8000 Rev: 5.02
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi2 Channel: 00 Id: 00 Lun: 08
Vendor: DDN Model: S2A 8000 Rev: 5.02
Type: Direct-Access ANSI SCSI revision: 03
1.4 效验SCSI、RAID、HBA卡加载顺序 ——–此步骤在redhat AS 4以后的版本中,不需要做。
下面的例子是redhat as 3 update4
如果SCSI、RAID卡的加载在HBA卡后面。不然在本地SCSI系统盘的SWAP分区,不可用。如下
#cat /etc/module.conf
alias scsi_hostadapter lpfc // Emulex HBA
alias scsi_hostadapter1 aic7xxx //scsi卡
alias usb-controller usb-ohci
options scsi_mod max_scsi_luns=256 scsi_allow_ghost_devices=1
需要将加载顺序改变,如
alias scsi_hostadapter1 aic7xxx //scsi卡
alias scsi_hostadapter2 lpfc // Emulex HBA
alias usb-controller usb-ohci
options scsi_mod max_scsi_luns=256 scsi_allow_ghost_devices=1
重建内核
# cd /boot
# mv initrd-2.4.21-27.ELsmp.img initrd-2.4.21-27.ELsmp.img.ori
# mkinitrd initrd-2.4.21-27.ELsmp.img 2.4.21-27.ELsmp
# sync
# reboot
2.安装StorNext FS
2.1 以root用户登录Linux,插入StorNext FS CD
#cd /mnt/cdrom
如果CDROM没有被mounted,那么执行#mount /dev/cdrom /mnt
# cd /mnt/cdrom/StorNextFS/linux //此路径根据实际情况而定
#uname -a 查看当前系统内核的版本。
如:uname -a
Linux mds01 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 i686 i686 i386 GNU/Linux
# rpm -ivh snfs-server_RH_AS_4_269-2.6.3-32.i386.rpm
# rpm –ivh snfs-client_RH_AS_4_269-2.6.3-32.i386.rpm
注意,在linux上服务器上必须安装Server和client两个软件。
2.2 修改环境变量
#vi /etc/profile
# Path manipulation
if [ `id -u` = 0 ]; then
pathmunge /usr/cvfs/bin
fi
ulimit -S -c 0 > /dev/null 2 > &1
ulimit -c unlimited > /dev/null 2 >&1
注销root用户,重新以root用户登陆系统,才能使环境变量起作用
2.3 磁盘Label
2.3.1 查看盘阵信息,执行fdisk -l
如: # fdisk –l
Disk /dev/sdc (Sun disk label): 0 heads, 0 sectors, 52221 cylinders
Units = cylinders of 1 * 512 bytes
Device Flag Start End Blocks Id System
/dev/sdc3 u 0 -872571904 -436285952 5 Whole disk
Note: sector size is 2048 (not 512)
Disk /dev/sdg (Sun disk label): 0 heads, 0 sectors, 959 cylinders
Units = cylinders of 1 * 512 bytes
Device Flag Start End Blocks Id System
/dev/sdg3 u 0 62902272 31451136 5 Whole disk
Disk /dev/sdl (Sun disk label): 0 heads, 0 sectors, 2 cylinders
Units = cylinders of 1 * 512 bytes
Device Flag Start End Blocks Id System
/dev/sdl3 u 0 192512 96256 5 Whole disk
2.1查看disk Label
#/usr/cvfs/bin/cvlabel –l
/dev/sdc [DDN S2A 8000 5.02] CVFS ” unknown ” Sectors: 3422394368.
SectorSize: 1024. Maximum sectors: 3422394368.
/dev/sdg [DDN S2A 8000 5.02] CVFS ” unknown ” Sectors: 62900224.
SectorSize: 512. Maximum sectors: 62900224.
/dev/sdl [DDN S2A 8000 5.02] CVFS ” unknown ” Sectors: 190464. S
ectorSize: 512. Maximum sectors: 190464.
2.2 编辑disk Label配置文件
# /usr/cvfs/bin/cvlabel -c > /usr/cvfs/config/cvlabels
//将cvlabel输出到cvlabels配置文件中
# vi /usr/cvfs/config/cvlabels
将unkown更改为H_Upload,MD_H_Upload, JR_H_Upload。。。等,如本例中:
H_Upload /dev/sdc # host 0 lun 0 sectors 3422394368 sector_size 1024 inquiry [DDN S2A 8000 5.02] serial 60001FF020021101030197FECD180000
MD_H_Upload /dev/sdg # host 0 lun 4 sectors 62900224 sector_size 512 inquiry [DDN S2A 8000 5.02] serial 60001FF0200211010301932703C00400
JR_H_Upload /dev/sdl # host 0 lun 9 sectors 190464 sector_size 512 inquiry [DDN S2A 8000 5.02] serial 60001FF020021101030194A700030900
2.3 disk Label
# /usr/cvfs/bin/cvlabel /usr/cvfs/config/cvlabels
*WARNING* This program will over-write volume labels on the
devices specified in the file “/usr/cvfs/config/cvlabels”.
After execution, the devices will only be usable by the
CentraVision File System. You will have to re-partition the
devices to use them on a different file system.
Do you want to proceed? (Y / N ) -> y
/dev/sdc [DN S2A 8000 5.02] CVFS ” unknown ” Sectors: 3422394368.
Do you want to label it SNFS – Name: H_Upload Sectors: 3422394368? (Y / N ) ->y
New Volume Label -Device: /dev/sda SNFS Label: H_Upload Sectors: 3422394368.
/dev/sdg [DN S2A 8000 5.02] CVFS ” unknown ” Sectors: 62900224.
Do you want to label it SNFS – Name: MD_H_Upload Sectors: 62900224? (Y / N ) ->y
New Volume Label -Device: /dev/sdb SNFS Label: MD_H_Upload Sectors: 62900224.
/dev/sdl [DN S2A 8000 5.02] CVFS ” unknown ” Sectors: 190464.
Do you want to label it SNFS – Name: JR_H_Upload Sectors: 190464? (Y / N ) ->y
New Volume Label -Device: /dev/sdb SNFS Label: JR_H_Upload Sectors: 190464.
Done. 3 source lines. 3 labels.
3. 配置StorNext FS
3.1创建StorNext FS文件系统的配置文件,
若查看参数信息,键入#man -M /usr/cvfs/man cvfs_config
# cd /usr/cvfs/config
#cp .. /examples/example.cfg ./H_Upload.cfg //例程中拷贝出文件系统的配置文件
# vi /usr/cvfs/config/H_Upload.cfg
GlobalSuperUser Yes ## Must be set to Yes for SNMS Managed File Systems ##
WindowsSecurity No
Quotas No
FileLocks No
DataMigration No ## SNMS Managed File Systems Only ##
InodeExpandMin 32K
InodeExpandInc 128K
InodeExpandMax 8M
FsBlockSize 4K
JournalSize 4M
AllocationStrategy Round
MaxConnections 78 //该数为一般设为客户数加+10
ForceStripeAlignment Yes
Debug 0x0
MaxLogSize 4M
MaxLogs 4
## Globals Defaulted
ThreadPoolSize 32 # default 16, 512 KB memory per thread
InodeCacheSize 32K # 800-1000 bytes each, default 8K
# BufferCacheSize 64M # default 32MB
# StripeAlignSize 2M # auto alignment, default MAX(StripeBreadth)
# MaxMBPerClientReserve 50 # in MBs, default 100 MB reserved per client
# OpHangLimitSecs 300 # default 180 secs
# DataMigrationThreadPoolSize 128 # Managed only, default 8
#************************************************************
# A disktype section for defining disk hardware parameters.
#************************************************************
[DiskType MD_H_Upload] ##1+1 Raid 1 Mirrored Pair##
Sectors 62900224 ## Sectors Per Disk From Command “cvlabel -l”
SectorSize 512
[DiskType JR_H_Upload] ##1+1 Raid 1 Mirrored Pair##
Sectors 190464
SectorSize 512
[DiskType H_Upload] ##4+1 Raid 5 Lun for Regular Data##
Sectors 3422394368
SectorSize 1024 //这里的1024与盘塔上该LUN的Block Size 相一致。
#************************************************************
# A disk section for defining disks in the hardware configuration.
#************************************************************
[Disk MD_H_Upload]
Status UP
Type MD_H_Upload
[Disk JR_H_Upload]
Status UP
Type JR_H_Upload
[Disk H_Upload]
Status UP
Type H_Upload
#************************************************************
# A stripe section for defining stripe groups.
*************************************************************
[StripeGroup MetaFiles]
Status UP
MetaData Yes
Journal No
Exclusive Yes
Read Enabled
Write Enabled
StripeBreadth 256K
MultiPathMethod Rotate
Node MD_H_Upload 0
[StripeGroup JournFiles]
Status UP
Journal Yes
MetaData No
Exclusive Yes
Read Enabled
Write Enabled
StripeBreadth 256K
MultiPathMethod Rotate
Node JR_H_Upload 0
[StripeGroup H_Upload_Files]
Status UP
Exclusive No ##Non-Exclusive StripeGroup for all Files##
Read Enabled
Write Enabled
StripeBreadth 1M //该参数根据应用可以调整
MultiPathMethod Rotate
Node H_Upload 0
若有多个文件系统依次创建文件系统配置文件
#vi /usr/cvfs/config/ file_system_name.cfg
# /etc/init.d/cvfs restart //重起cvfs服务。
Unmounting SNFS filesystems
Stopping SNFS Daemons
Stopping SNFS PortMapper
Waiting for FSMs to finish
SNFS Stop [ 确定 ]
Initializing StorNext Filesystem (SNFS)
No disk subsystem config for SNFS, disks not started
Loading SNFS modules
Starting /usr/cvfs/bin/fsmpm..
Starting /usr/cvfs/bin/cvfsd…
Mounting SNFS filesystems
SNFS Initialized [ 确定 ]
# /usr/cvfs/bin/cvmkfs H_Upload //初始化文件系统
StorNext File System File System Initializer.
** WARNING ** This will destroy all contents and data of the
— [H_Upload] —
file system. Re-initialization cannot be un-done!
Do you want to proceed? (Y/[N]) -> y
Re-initializing file system ‘ H_Upload ‘.
Shared Meta Data File System.
Meta Data Root is on “ MetaFiles “.
Resetting Arbitration Block.
Writing Configuration Information Block.
Writing Root Directory Block.
Writing Disk Information Block.
Writing Stripe Group Index “ JournFiles “.
Resetting File System Journal.
Creating Free List.
Writing Initial 1024 Inodes.
Writing Initialization Control Block.
Resetting Super Block.
Writing Reserved Allocations.
File system ‘ H_Upload ‘ successfully created.
同样方法依次初始化文件系统 # /usr/cvfs/bin/cvmkfs file_system_name
3.2 创建文件系统配置文件列表的文件
# cd /usr/cvfs/config
# cp ../examples/fsmlist.example ./fsmlist
//例程中拷贝出文件系统配置文件列表的配置文件
#vi /usr/cvfs/config/fsmlist
//将当前的StorNext FS文件系统的配置文件H_Upload.cfg的文件名写入fsmlist
file_system_name . 0
如: H_Upload . 0
// 其中的 . 表示当前机器的ip地址,0是优先,可为一个0~7. 但当前默认的为0是activate
3.3 创建文件系统配置服务器列表的文件
# cd /usr/cvfs/config #cp ../examples/fsnameservers.example ./fsnameservers
//将当前的StorNext FS文件系统的服务器的ip写入fsnameservers
如: 10.5.0.25
10.5.0.26
在做群集时,可以将每台服务器ip加入进来.并保持每一台的fsnameservers文件一致
3.2 手动启动文件系统配置
# /usr/cvfs/bin/cvadmin
StorNext File System Administrator
Enter command(s)
For command help, enter “help” or “?”.
List FSS
File System Services (* indicates service is in control of FS):
1>* H_Upload [0] located on mds1:32790 (pid 2949)
//我们可以看到系统已经自动将H_Upload启动并激活了.
* 表示激活
如果没有显示文件系统名称, 一般需要重新启动cvfs服务或检查fsmlist配置
或者手动启动,如下
Select FSM “none”
snadmin> start H_Upload
Start FSS ” H_Upload ”
FSS ‘ H_Upload ‘ start initiated.
FSS ‘ H_Upload ‘ started.
snadmin> activate H_Upload
Activate FSM ” H_Upload ”
Created : Fri Aug 19 10:02:50 2005
Active Connections: 24
Fs Block Size : 4K
Msg Buffer Size : 4K
Disk Devices : 3
Stripe Groups : 3
Mirror Groups : 0
Fs Blocks : 855598592 (3.19 TB)
Fs Blocks Free : 838919232 (3.13 TB) (98%)
snadmin (H_Upload) > show long
Show stripe groups (File System “H_Upload”)
Stripe Group 0 [MetaFiles] Status:Up,MetaData,Exclusive
Total Blocks:7862528 (29.99 GB) Free:7827253 (29.86 GB) (99%)
MultiPath Method:Rotate
Stripe Depth:1 Stripe Breadth:64 blocks (256.00 KB)
Realtime limit IO/sec:0 (~0 mb/sec) Non-Realtime reserve IO/sec:0
Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0
Disk stripes:
Primary Stripe 0 [MetaFiles] Read:Enabled Write:Enabled
Node 0 [MD_H_Upload]
Stripe Group 1 [JournFiles] Status:Up,Journal,Exclusive
Total Blocks:23808 (93.00 MB) Free:22784 (89.00 MB) (95%)
MultiPath Method:Rotate
Stripe Depth:1 Stripe Breadth:64 blocks (256.00 KB)
Realtime limit IO/sec:0 (~0 mb/sec) Non-Realtime reserve IO/sec:0
Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0
Disk stripes:
Primary Stripe 0 [JournFiles] Read:Enabled Write:Enabled
Node 0 [JR_H_Upload]
Stripe Group 2 [H_Upload_Files] Status:Up
Total Blocks:855598592 (3.19 TB) Free:838919232 (3.13 TB) (98%)
MultiPath Method:Rotate
Stripe Depth:1 Stripe Breadth:256 blocks (1024.00 KB)
Realtime limit IO/sec:0 (~0 mb/sec) Non-Realtime reserve IO/sec:0
Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0
Disk stripes:
Primary Stripe 0 [H_Upload_Files] Read:Enabled Write:Enabled
Node 0 [H_Upload]
snadmin (H_Upload) > who //查看系统用户信息
Who (File System “H_Upload”)
# SNFS I.D. Type Location Up Time License Expires
— ———- —- ————— ——— —————
0> FSM 1d 3h 55m N/A
1> F1F6B61DB CLI 10.5.0.12 0d 3h 6m Permanent
37> ADM mds1 0d 0h 0m N/A
snadmin (H_Upload) >exit
4. Mount StorNext FS //当linux作为client时,需要mount cvfs文件系统
# mkdir /H_Upload //创建mount点的目录
# chmod 777 / H_Upload //设置/usr/snfs目录的权限,当前权限为可读可写可执行
# vi /etc/fstab //当系统启动时,自动装载StorNext FS文件系统H_Upload
H_Upload /H_Upload cvfs verbose=yes 0 0
//手动装载StorNext FS文件系统,
# mount -t cvfs H_Upload /H_Upload
mount.cvfs: Filesystem H_Upload mounted on / H_Upload
2 条回复
真的很是不错的说。。。在这个的指导下,俺也装完了SNFS,但属于很简单的那种。赞一个。偶也是西安理工大的说,水院的师兄,谢谢:P
Hey, thanks for the post.Really thank you! Keep writing.