Bacula 不会同时使用 2 个磁带设备。(搜索#-#-#(TL;DR)
也许,需要一点背景知识。
在尝试获取一个可行的备份解决方案(备份 >20TB 并不便宜,也不容易)的过程中$dayjob
,我们购买了一堆东西来使其发挥作用。
首先,有一个 Spectra Logic T50e 自动转换器、40 个 LTO5 插槽,并且该机器人有一对 IBM HH5 Ultrium LTO5 驱动器,通过光纤通道仲裁环路连接到我们的备份服务器。
这是备份服务器。一台戴尔 R715,配备 2 个 16 核 AMD 62xx CPU 和 32GB RAM。太棒了。该服务器有 2 个 Emulex FCe-12000E 卡和一个 Intel X520-SR 双端口 10GE NIC。
我们还销售了 Commvault Backup(非 NDMP)。
这就是事情变得真正复杂的地方。
Spectra Logic 和 Commvault 都派出了各自的工程师,他们设置了库和软件。只要控制器工作正常,Commvault 就运行良好。戴尔服务器有 Ubuntu 12.04 服务器,运行 CommVault 的 MediaAgent,并将我们的 BlueArc NAS 作为 NFS 安装到几个安装点,例如/home
,以及中的一些东西/mnt
。
从 NFS 挂载点备份时,我们看到290GB/hr
吞吐量 ~=。考虑到我们要在不到 48 小时的备份时间内处理 20 多 TB 的数据,这简直是垃圾。BlueArc 的额定最大值是700MB/s
( 2460GB/hr
),磁带设备的额定最大写入速度是每驱动器 140MB/s,所以这就是492GB/hr
(或两倍,总吞吐量)。
因此,下一步是使用 IOzone 对 NFS 性能进行基准测试,结果发现我们获得了史诗般的写入性能(跨 20 多个线程),写入速度约为 1.5-2.5 TB/小时,但读取性能却令人绝望。我永远无法获得高于 343GB/小时的最高速度。因此,让我们假设这343GB/hr
是 NAS 读取性能的理论最大值,那么理论上我们应该能够从中获得该性能A)CommVault 和b)任何其他备份代理。
事实并非如此。Commvault 似乎只为我提供200-250GB/hr
吞吐量,出于实验,我安装了 Bacula 来查看其状态。例如,如果 Bacula 的性能和速度始终优于 Commvault,那么我们可以说"**$.$ Refunds Plz $.$**"
#-#-#
唉,我发现 Bacula 存在另一个问题。Commvault 似乎很乐意用一个线程从挂载点的一部分读取数据,并将其传输到磁带设备,同时用另一个线程从其他目录读取数据,并写入自动转换器中的第二个驱动器。
我无论如何都无法让 Bacula 安装并写入两个磁带驱动器同时地。
我尝试过的事情:
Maximum Concurrent Jobs = 20
Director、文件和存储守护进程中的设置Prefer Mounted Volumes = no
作业定义中的设置- 在自动转换器资源中设置多个设备。
文档似乎非常以单驱动器为中心,我们觉得这有点像把火箭绑在仓鼠身上。大多数示例 Bacula 配置适用于 DDS4 驱动器、手动磁带交换和 FreeBSD 或 IRIX 系统。
我可能应该补充一点,我不太在意这个不是有可能,但我会感到惊讶。我基本上想用 Bacula 作为证据,让软件供应商知道他们的价格过高 ;)
我从某处读到@KyleBrandt 使用现代磁带解决方案做了类似的事情。
配置文件: bacula-dir.conf
#
# Default Bacula Director Configuration file
Director { # define myself
Name = backuphost-1-dir
DIRport = 9101 # where we listen for UA connections
QueryFile = "/etc/bacula/scripts/query.sql"
WorkingDirectory = "/var/lib/bacula"
PidDirectory = "/var/run/bacula"
Maximum Concurrent Jobs = 20
Password = "yourekiddingright" # Console password
Messages = Daemon
DirAddress = 0.0.0.0
#DirAddress = 127.0.0.1
}
JobDefs {
Name = "DefaultFileJob"
Type = Backup
Level = Incremental
Client = backuphost-1-fd
FileSet = "Full Set"
Schedule = "WeeklyCycle"
Storage = File
Messages = Standard
Pool = File
Priority = 10
Write Bootstrap = "/var/lib/bacula/%c.bsr"
}
JobDefs {
Name = "DefaultTapeJob"
Type = Backup
Level = Incremental
Client = backuphost-1-fd
FileSet = "Full Set"
Schedule = "WeeklyCycle"
Storage = "SpectraLogic"
Messages = Standard
Pool = AllTapes
Priority = 10
Write Bootstrap = "/var/lib/bacula/%c.bsr"
Prefer Mounted Volumes = no
}
#
# Define the main nightly save backup job
# By default, this job will back up to disk in /nonexistant/path/to/file/archive/dir
Job {
Name = "BackupClient1"
JobDefs = "DefaultFileJob"
}
Job {
Name = "BackupThisVolume"
JobDefs = "DefaultTapeJob"
FileSet = "SpecialVolume"
}
#Job {
# Name = "BackupClient2"
# Client = backuphost-12-fd
# JobDefs = "DefaultJob"
#}
# Backup the catalog database (after the nightly save)
Job {
Name = "BackupCatalog"
JobDefs = "DefaultFileJob"
Level = Full
FileSet="Catalog"
Schedule = "WeeklyCycleAfterBackup"
# This creates an ASCII copy of the catalog
# Arguments to make_catalog_backup.pl are:
# make_catalog_backup.pl <catalog-name>
RunBeforeJob = "/etc/bacula/scripts/make_catalog_backup.pl MyCatalog"
# This deletes the copy of the catalog
RunAfterJob = "/etc/bacula/scripts/delete_catalog_backup"
Write Bootstrap = "/var/lib/bacula/%n.bsr"
Priority = 11 # run after main backup
}
#
# Standard Restore template, to be changed by Console program
# Only one such job is needed for all Jobs/Clients/Storage ...
#
Job {
Name = "RestoreFiles"
Type = Restore
Client=backuphost-1-fd
FileSet="Full Set"
Storage = File
Pool = Default
Messages = Standard
Where = /srv/bacula/restore
}
FileSet {
Name = "SpecialVolume"
Include {
Options {
signature = MD5
}
File = /mnt/SpecialVolume
}
Exclude {
File = /var/lib/bacula
File = /nonexistant/path/to/file/archive/dir
File = /proc
File = /tmp
File = /.journal
File = /.fsck
}
}
# List of files to be backed up
FileSet {
Name = "Full Set"
Include {
Options {
signature = MD5
}
File = /usr/sbin
}
Exclude {
File = /var/lib/bacula
File = /nonexistant/path/to/file/archive/dir
File = /proc
File = /tmp
File = /.journal
File = /.fsck
}
}
Schedule {
Name = "WeeklyCycle"
Run = Full 1st sun at 23:05
Run = Differential 2nd-5th sun at 23:05
Run = Incremental mon-sat at 23:05
}
# This schedule does the catalog. It starts after the WeeklyCycle
Schedule {
Name = "WeeklyCycleAfterBackup"
Run = Full sun-sat at 23:10
}
# This is the backup of the catalog
FileSet {
Name = "Catalog"
Include {
Options {
signature = MD5
}
File = "/var/lib/bacula/bacula.sql"
}
}
# Client (File Services) to backup
Client {
Name = backuphost-1-fd
Address = localhost
FDPort = 9102
Catalog = MyCatalog
Password = "surelyyourejoking" # password for FileDaemon
File Retention = 30 days # 30 days
Job Retention = 6 months # six months
AutoPrune = yes # Prune expired Jobs/Files
}
#
# Second Client (File Services) to backup
# You should change Name, Address, and Password before using
#
#Client {
# Name = backuphost-12-fd
# Address = localhost2
# FDPort = 9102
# Catalog = MyCatalog
# Password = "i'mnotjokinganddontcallmeshirley" # password for FileDaemon 2
# File Retention = 30 days # 30 days
# Job Retention = 6 months # six months
# AutoPrune = yes # Prune expired Jobs/Files
#}
# Definition of file storage device
Storage {
Name = File
# Do not use "localhost" here
Address = localhost # N.B. Use a fully qualified name here
SDPort = 9103
Password = "lalalalala"
Device = FileStorage
Media Type = File
}
Storage {
Name = "SpectraLogic"
Address = localhost
SDPort = 9103
Password = "linkedinmakethebestpasswords"
Device = Drive-1
Device = Drive-2
Media Type = LTO5
Autochanger = yes
}
# Generic catalog service
Catalog {
Name = MyCatalog
# Uncomment the following line if you want the dbi driver
# dbdriver = "dbi:sqlite3"; dbaddress = 127.0.0.1; dbport =
dbname = "bacula"; DB Address = ""; dbuser = "bacula"; dbpassword = ""
}
# Reasonable message delivery -- send most everything to email address
# and to the console
Messages {
Name = Standard
mailcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: %t %e of %c %l\" %r"
operatorcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: Intervention needed for %j\" %r"
mail = root@localhost = all, !skipped
operator = root@localhost = mount
console = all, !skipped, !saved
#
# WARNING! the following will create a file that you must cycle from
# time to time as it will grow indefinitely. However, it will
# also keep all your messages if they scroll off the console.
#
append = "/var/lib/bacula/log" = all, !skipped
catalog = all
}
#
# Message delivery for daemon messages (no job).
Messages {
Name = Daemon
mailcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula daemon message\" %r"
mail = root@localhost = all, !skipped
console = all, !skipped, !saved
append = "/var/lib/bacula/log" = all, !skipped
}
# Default pool definition
Pool {
Name = Default
Pool Type = Backup
Recycle = yes # Bacula can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 365 days # one year
}
# File Pool definition
Pool {
Name = File
Pool Type = Backup
Recycle = yes # Bacula can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 365 days # one year
Maximum Volume Bytes = 50G # Limit Volume size to something reasonable
Maximum Volumes = 100 # Limit number of Volumes in Pool
}
Pool {
Name = AllTapes
Pool Type = Backup
Recycle = yes
AutoPrune = yes # Prune expired volumes
Volume Retention = 31 days # one Moth
}
# Scratch pool definition
Pool {
Name = Scratch
Pool Type = Backup
}
#
# Restricted console used by tray-monitor to get the status of the director
#
Console {
Name = backuphost-1-mon
Password = "LastFMalsostorePasswordsLikeThis"
CommandACL = status, .status
}
bacula-sd.conf
#
# Default Bacula Storage Daemon Configuration file
#
Storage { # definition of myself
Name = backuphost-1-sd
SDPort = 9103 # Director's port
WorkingDirectory = "/var/lib/bacula"
Pid Directory = "/var/run/bacula"
Maximum Concurrent Jobs = 20
SDAddress = 0.0.0.0
# SDAddress = 127.0.0.1
}
#
# List Directors who are permitted to contact Storage daemon
#
Director {
Name = backuphost-1-dir
Password = "passwordslinplaintext"
}
#
# Restricted Director, used by tray-monitor to get the
# status of the storage daemon
#
Director {
Name = backuphost-1-mon
Password = "totalinsecurityabound"
Monitor = yes
}
Device {
Name = FileStorage
Media Type = File
Archive Device = /srv/bacula/archive
LabelMedia = yes; # lets Bacula label unlabeled media
Random Access = Yes;
AutomaticMount = yes; # when device opened, read it
RemovableMedia = no;
AlwaysOpen = no;
}
Autochanger {
Name = SpectraLogic
Device = Drive-1
Device = Drive-2
Changer Command = "/etc/bacula/scripts/mtx-changer %c %o %S %a %d"
Changer Device = /dev/sg4
}
Device {
Name = Drive-1
Drive Index = 0
Archive Device = /dev/nst0
Changer Device = /dev/sg4
Media Type = LTO5
AutoChanger = yes
RemovableMedia = yes;
AutomaticMount = yes;
AlwaysOpen = yes;
RandomAccess = no;
LabelMedia = yes
}
Device {
Name = Drive-2
Drive Index = 1
Archive Device = /dev/nst1
Changer Device = /dev/sg4
Media Type = LTO5
AutoChanger = yes
RemovableMedia = yes;
AutomaticMount = yes;
AlwaysOpen = yes;
RandomAccess = no;
LabelMedia = yes
}
#
# Send all messages to the Director,
# mount messages also are sent to the email address
#
Messages {
Name = Standard
director = backuphost-1-dir = all
}
bacula-fd.conf
#
# Default Bacula File Daemon Configuration file
#
#
# List Directors who are permitted to contact this File daemon
#
Director {
Name = backuphost-1-dir
Password = "hahahahahaha"
}
#
# Restricted Director, used by tray-monitor to get the
# status of the file daemon
#
Director {
Name = backuphost-1-mon
Password = "hohohohohho"
Monitor = yes
}
#
# "Global" File daemon configuration specifications
#
FileDaemon { # this is me
Name = backuphost-1-fd
FDport = 9102 # where we listen for the director
WorkingDirectory = /var/lib/bacula
Pid Directory = /var/run/bacula
Maximum Concurrent Jobs = 20
#FDAddress = 127.0.0.1
FDAddress = 0.0.0.0
}
# Send all messages except skipped files back to Director
Messages {
Name = Standard
director = backuphost-1-dir = all, !skipped, !restored
}
答案1
当你在 bacula 中设置文件集时,它将逐行读取路径规范并像这样备份。
它不会创建两个线程来读取代理中的不同文件路径。
正如@SpacemanSpiff 所说,如果您想这样做,那么前进的方向就是设置不同的作业,为每个您想要备份的文件规范设置一个作业。
答案2
我有三点建议给你:
- 使用多个存储守护进程。您可以在同一台机器的不同端口上运行多个存储守护进程。
- 使用基础作业进行重复数据删除。节省时间和空间。
- 使用压缩 - 如果您的磁带驱动器支持压缩,那当然好,但您可能需要权衡利弊,并尝试使用 bacula-fd 压缩。这发生在客户端,因此也节省了带宽,但只牺牲了一点 CPU 时间。