☰

文件越大，Linux 重定向“>>”速度是否会变慢？

tag-icon • tag-icon io-redirection
文件越大，Linux 重定向“>>”速度是否会变慢？

我有一个 shell 脚本，它进行一些分析并准备（即：写入）一些命令以在单独的文件中运行。

所以我有类似的东西：

echo my_command_to_run >> /tmp/file_command_to_run.txt

我感觉程序越来越慢。 当文件较大（约 3M 行）时，程序是否可能需要更长的时间？

我还在内存中存储了一些内容，所以这也可能是我的问题的根源，但我只想知道是否需要在不同的文件中重定向输出。 （例如：写几个2000行的文件）

编辑：我的脚本正在准备将约 64M（数百万）文件移动到更好的架构中。因此，我浏览了所有不同的“结构化”文件夹，并准备移动。

我内存中有这样的数组：

topic1 -> /path/to/my/folder
topic1_number_of_files -> nb

所以我的数组也变得更大，因为我有几个条目（最大〜4'000）。

否则，运行的操作始终相同。只有我的数组和文件在变大。

EDIT2：下面是我的脚本

笔记：

  • 我有几个文件夹，里面最多包含 100'000 个文件。
  • 我可以有：folder1 -> (source1__description1, source1__description2, source2__description3)

目标：有类似的东西：

  • 源 1/文件夹 1 ->（源 1__ 描述 1，源 1 __ 描述 2）
  • 源 2/文件夹 1 -> (源 2__description3, ...)

目前演出：

14 小时内插入约 900'000 行 <=> 这将需要大约 40 天来准备所有移动命令

#!/bin/bash

argument=$1

if [[ -n "$argument" ]] && [[ -e $argument ]]; then
    html_folder=$argument
    echo "We will move [folder]/files from your parameters: '$html_folder'"
else
    html_folder="/var/files/html_files/"
    echo "NO PARAMETER (or folder does not exist) - We will move [folder]/files from $html_folder"
fi

######################## create the list ########################
filename="/var/files/html_files/list_folder.txt" # list generated with  ls -1 -f (this doesn't take everything in memory)
ls -1 -fp $html_folder |  grep '/$' |  grep 'folder'> "$filename"
#################### END create the list ########################


echo " "


# --------------------------------------------------------------
# -------------- Global variables for moving part --------------
# --------------------------------------------------------------
    # Variables for storing the folder/files tree
    declare -A folder_array         # array of folder '/files/publisher_html/10.3390' => 4 (i.e.: 4 folders for mdpi)
    declare -A folder_files_array   # array of files in last folder '/files/publisher_html/10.3390' => 51 (i.e.: 51 files in the 4th folders for mdpi)
    storageFolder="/files/publisher_html/"
    nb_limit=100000 # max number of file per folder
    file_nb=0
    current_folder=""
# --------------------------------------------------------------
# --------------------------------------------------------------
# --------------------------------------------------------------


# --------------------------------------------------------------
# -------------- Global functions for moving part --------------
# --------------------------------------------------------------

    countNumberOfFilesPerFolder () {
        nb=0
        if [[ -e $1 ]]; then        
            nb=$(ls -1fp $1 |  grep -v '/$' | wc -l )        
        fi
        echo $nb
    }

    createFolderIfNeeded () {
        # $1  # first arg (/path/to/htmlfiles/10.3390)
        tmp_folder=""
        nb_folder=1
        nb_files=0

        if [[ ! -e $1 ]]; then # if folder doesn't exist
            sudo mkdir -p "$1/folder$nb_folder" ; # create the folders if don't exist
        else
            #echo "THE FOLDER $tmp_folder ALREADY EXISTED...BE AWARE!!!"
            if [[ -e ${folder_array[$1]} ]]; then
                nb_folder=${folder_array[$1]} # take the value from memory if available
            else
                nb_folder=$(ls -1f $1 | grep folder | wc -l )
            fi

            if (($nb_folder==0)); then # if no subfolder for the publisher folder
                nb_folder=1
                nb_files=0
                sudo mkdir -p "$1/folder$nb_folder" # simply create the first folder
            else
                # if [[ -e ${folder_files_array[$1]} ]]; then
                if [[ ${folder_files_array[$1]} ]]; then
                    nb_files=${folder_files_array[$1]}  # value from memory
                    #echo "value from MEEEEEM: $1 => $nb_files"
                else
                    nb_files=`countNumberOfFilesPerFolder "$1/folder$nb_folder"`
                    #echo "value from COOOOOOUNT: $1 => $nb_files"
                fi

                if (($nb_files >= $nb_limit)); then # create a new folder + reset memory value
                    ((nb_folder++))
                    nb_files=0
                    sudo mkdir -p "$1/folder$nb_folder"
                    #`createFolderIfNeeded "$1/folder$nb_folder"` # NO CORRECT -> will create a subfolder
                fi            
            fi       
        fi

        #((nb_files++))
        folder_files_array[$1]=$nb_files
        folder_array[$1]=$nb_folder

        current_folder="$1/folder$nb_folder" # change the global variable
    }

    extractPrefix() {
        whotest[0]='test' || (echo 'Failure: arrays not supported in this version of
    bash.' && exit 2)
        array=(${1//__/ })
        prefix=${array[0]}
        echo $prefix
    }
# --------------------------------------------------------------
# --------------------------------------------------------------
# --------------------------------------------------------------


toMoveFolder=$html_folder"toMove/"
toMoveFileIndex=1
toMoveCmdNumber=0
maxCmdInFile=2000
if [[ ! -e $toMoveFolder ]]; then # if folder doesn't exist
    sudo mkdir -p $toMoveFolder ; # create the folders
fi

cd $html_folder

while read -r folder # for each folder
do
    if [[ -e $folder ]]; then        
        echo "Will manage folder: $folder"


# ---------------------------------------------------------------------------------------------------
# -------------------------------------- MOVE INDIVIDUAL FILES --------------------------------------
# ---------------------------------------------------------------------------------------------------

    argument=$html_folder$folder
    cpt=0
    #argument=$1

    if [[ -n "$argument" ]] && [[ -e $argument ]]; then
        html_files_folder=$argument
    else
        html_files_folder="/var/files/html_files/html_files/"
    fi

    ######################## create the list ########################

    htmlList="/var/files/html_files/list_html.txt" # list generated with  ls -1 -f (this doesn't take everything in memory)
    ls -1f $html_files_folder > "$htmlList" # no need to exclude the "." and ".." (we exclude from the foreach)
    #################### END create the list ########################

    echo " "


    current_folder=$storageFolder # probably useless

    while read -r line
    do
        name=$line

        if [[ $name != "." ]] && [[ $name != ".." ]]; then # don't take the folder itself
            prefix=`extractPrefix $name`
            if [ -n $prefix ]; then
                # change the global $current_folder
                # + create new subfolder if needed
                # + increment nb of files in folder
                createFolderIfNeeded $storageFolder$prefix
                ((cpt++))

                if(( $toMoveCmdNumber >= $maxCmdInFile )); then
                    toMoveCmdNumber=0
                    ((toMoveFileIndex++))
                fi

        echo "sudo mv $html_files_folder$name $current_folder/$name" | sed -r 's/[\(\)]+/\\&/g' >> $toMoveFolder"command_"$toMoveFileIndex".txt"
                ((toMoveCmdNumber++))

                ((folder_files_array[$storageFolder$prefix]++))

                if (( $cpt % 50 == 0 ));then
                    echo ""
                    echo "Remind: folder -> $current_folder/"
                    echo "${#folder_array[@]} publishers in memory!"
                fi

                echo "#$cpt - $name (${folder_files_array[$storageFolder$prefix]} files)"
            else
                echo "ERROR -> $name has not been moved as expected"
            fi        
       fi

    # >> $toMoveFolder"file"$toMoveFileIndex".txt" # <== does not take the toMoveFileIndex variation in consideration
    done < "$htmlList" # useful if we use the while

    echo "Folder $html_files_folder has been processed"
    echo " "


# ---------------------------------------------------------------------------------------------------
# ---------------------------------------------------------------------------------------------------
# ---------------------------------------------------------------------------------------------------


    else  # END  if [[ -e $folder ]]; then
        echo " "; echo "ERROR -> folder $folder does NOT exist!"; echo " "
        continue
    fi

done < "$filename" # useful if we use the while

echo "The script to prepare the move of the html files FROM FOLDER in other folders finished!"
echo " "
echo " "
echo "FOLDER ARRAY AT THE END: "
    for i in "${!folder_array[@]}"; do echo "folder  : $i => nb_folder: ${folder_array[$i]} / nb__file in last folder: ${folder_files_array[$i]}"; done

echo " "
echo " "
echo "This is the end of the script"

和分区：

$df -h
/dev/sdb1         2.0T  370G  1.7T  19% /var/files
X.X.X.X:/files     11T  2.8T  7.2T  28% /files

最后编辑：

经过进一步分析，我发现 /var/files/html_files/ 是 /files/html_files/ 的符号链接，因此源和目标实际上是同一（远程）服务器。我将脚本放在“远程”服务器上运行，它似乎要快得多。

感谢您的帮助和有趣的评论！

答案1

我只是想知道是否需要重定向不同文件中的输出。 （例如：写几个2000行的文件）

拆分成更多数量的文件并不一定等于执行速度更快。三个简单的测试用例说明了这一点。这三个案例各打印 300 万行。这些按执行速度从最快到最慢的顺序列出。

  1. 循环外一次重定向

    for i in $(seq $((3000000/2000))); do seq 2000; done > file

  2. 在循环内附加到同一文件

    for i in $(seq $((3000000/2000))); do seq 2000 >> file; done

  3. 将输出拆分为多个文件

    for i in $(seq $((3000000/2000))); do seq 2000 > file$i; done

后面的命令始终比前面的命令花费更多的用户和系统时间。

由此我们可以得出结论，在这种简单的情况下，分割成更多的文件并不能保证性能的提高。事实恰恰相反。

I/O 操作数

性能不仅取决于文件的大小，还取决于IO操作的数量。当追加 ( >>) 时，会发生更多的 I/O 调用以查找文件末尾。

第一个脚本执行 i/o 操作 ( >>)外部循环for：

$ cat outloop.sh
#!/bin/sh
>file
for i in $(seq 1 ${1:?})
do
    echo $i
done >> file

另一方面，该脚本>>在每次迭代中执行 i/o 操作 ( )，里面循环for：

$ cat inloop.sh
#!/bin/sh
>file
for i in $(seq 1 ${1:?})
do
    echo $i >> file
done

运行并比较，看看算子的位置如何>>影响性能：

$ x=500000; time sh outloop.sh $x; time sh inloop.sh $x; 

real    0m1.227s
user    0m0.389s
sys     0m0.859s

real    0m2.996s
user    0m0.809s
sys     0m2.197s

放置重定向运算符外部当写入 500000 行时（在我的系统上），循环使性能翻倍。

相关内容