☰

用于计算和汇总每行中的数字和唯一数字的脚本

tag-icon • tag-icon linux shell-script centos awk sort
用于计算和汇总每行中的数字和唯一数字的脚本 
timestamp : IDs returned
20160420084726:-
20160420085418:[111783178, 111557953, 111646835, 111413356, 111412662, 105618372, 111413557]
20160420085418:[111413432, 111633904, 111783198, 111792767, 111557948, 111413225, 111413281]
20160420085418:[111413432, 111633904, 111783198, 111792767, 111557948, 111413225, 111413281]
20160420085522:[111344871, 111394583, 111295547, 111379566, 111352520]
20160420090022:[111344871, 111394583, 111295547, 111379566, 111352520]

时间戳的格式为 YYYYMMDDhhmmss

· ads 是用方括号括起来的以逗号分隔的广告资产 ID 列表，或者 - 如果没有返回广告

我需要编写一个脚本，在一天中的每十分钟时间内输出：

  1. 返回的 ID 数

  2. 返回的唯一 ID 的数量

  3. 该脚本应支持命令行参数来选择是否应给出唯一 ID 或总 ID。

使用上述日志摘录的示例输出（在总计模式下）：

20160420084:0
20160420085:26
20160420090:5

在独特的计数模式下，它会给出：

20160420084:0
20160420085:19
20160420090:5

这是我到目前为止的情况：

#!/usr/bin/bash
awk -F":" ' { print $1":" $2 } ' file.log | sort -r | uniq -c

结果：

1 20160420090022:[111344871, 111394583, 111295547, 111379566, 111352520] 
1 20160420085522:[111344871, 111394583, 111295547, 111379566, 111352520] 
1 20160420085418:[111783178, 111557953, 111646835, 111413356, 111412662, 105618372, 111413557] 
2 20160420085418:[111413432, 111633904, 111783198, 111792767, 111557948, 111413225, 111413281] 
1 20160420084726:- 
7: –

答案1

使用 GNU awk 处理数组数组（length(array)但现在大多数 awk 都这样做）：

$ cat tst.awk
BEGIN { FS=OFS=":" }
NR>1 {
    time = substr($1,1,11)
    totIds[time] += 0
    if ( gsub(/[][ ]/,"",$2) ) {
        totIds[time] += split($2,ids,/,/)
        for ( i in ids ) {
            unqIds[time][ids[i]]
        }
    }
}
END {
    for ( time in totIds ) {
        print time, ( type ~ /^tot/ ? totIds[time] : length(unqIds[time]) )
    }
}


$ awk -v type='tot' -f tst.awk file
20160420084:0
20160420085:26
20160420090:5


$ awk -v type='unq' -f tst.awk file
20160420084:0
20160420085:19
20160420090:5

如果您没有 GNU awk，您可以使用任何 awk 执行相同的操作，只是代码和内存使用量稍多一些：

$ cat tst.awk
BEGIN { FS=OFS=":" }
NR>1 {
    time = substr($1,1,11)
    totIds[time] += 0
    if ( gsub(/[][ ]/,"",$2) ) {
        totIds[time] += split($2,ids,/,/)
        for ( i in ids ) {
            if ( !seen[time,ids[i]]++ ) {
                numUnq[time]++
            }
        }
    }
}
END {
    for ( time in totIds ) {
        print time, ( type ~ /^tot/ ? totIds[time] : numUnq[time]+0 )
    }
}


$ awk -v type='tot' -f tst.awk file
20160420084:0
20160420085:26
20160420090:5


$ awk -v type='unq' -f tst.awk file
20160420084:0
20160420085:19
20160420090:5

答案2

$ cat summarise.pl
#!/usr/bin/perl

use strict;
use List::Util qw(sum0); # useful function to sum elements of array

# handle command-line options
use Getopt::Long qw(VersionMessage :config gnu_getopt);
use Pod::Usage;
$main::VERSION='0.0-alpha';
my $help = 0;
my $type;
GetOptions('total|sum|s|t' => sub { $type .=  't' },
           'unique|u'      => sub { $type .=  'u' },
           'both|b'        => sub { $type  = 'tu' },
           'help|h|?'      => \$help,
           'version|V'     => sub { VersionMessage() },
          ) or pod2usage(2);
pod2usage(-exitval => 0, -verbose => 2) if $help or (! @ARGV && -t);
$type = 'b' if length($type) > 1;
$type = 'u' if length($type) == 0; # default is unique

# Hash-of-Hashes (HoH) to hold timestamps, ids, and id counts.
# Primary key will be timestamp, secondary keys are the IDs,
# and values are the counts of each secondary key.
my %timestamps;

# read and process input file
while(<<>>) {
  next if ($. == 1); # skip first line of input file(s)
  chomp;

  # remove square brackets and spaces from input line
  s/[][]|\s+//g;

  # split current line on colon and commas
  my($ts,@ids) = split /[:,]/;
  $ts = substr($ts,0,11);
  $timestamps{$ts} = () unless (defined $timestamps{$ts});

  # remove non-numeric elements of @ids array
  @ids = grep { /^\d+$/ } @ids;

  # Convert to the HoH structure.
  map { $timestamps{$ts}{$_}++ } @ids;

  close(ARGV) if eof; # reset line counter $. at end of each input file
}

# all input has been processed, so print the results.
foreach my $ts (sort keys %timestamps) {
  my $u = scalar keys %{ $timestamps{$ts} };
  my $s = sum0 values %{ $timestamps{$ts} };

  if ($type eq 'u') {
    printf "%s: %i\n", $ts, $u;
  } elsif ($type eq 't') {
    printf "%s: %i\n", $ts, $s;
  } elsif ($type eq 'b') {
    printf "%s: %i, %i\n", $ts, $u, $s;
  };
}

__END__

=head1 summarise.pl

Script for counting and summarising Numbers & Unique numbers in each line.

=head1 SYNOPSIS

summarise.pl [options] [file ...]

=head1 OPTIONS

=over

=item B<--total>

print total number of IDs per timestamp

=item B<--unique>

print number of unique IDs per timestamp

=item B<--both>

print both unique and total number of IDs

=item B<--version>

Print script's version and exit

=item B<--help>

Print this help message and exit

=back
=cut

笔记：列表::实用程序,Pod::用法， 和Getopt::长都是核心 Perl 模块并且包含在 Perl 中。

perldoc pod有关 perl 的“Plain Old Documentation”内置代码文档如何工作的详细信息，请参阅参考资料。我仅仅触及了它的功能的表面。

示例输出：

$ ./summarise.pl input.txt 
20160420084: 0
20160420085: 19
20160420090: 5

$ ./summarise.pl input.txt -t
20160420084: 0
20160420085: 26
20160420090: 5

$ ./summarise.pl input.txt -b
20160420084: 0, 0
20160420085: 19, 26
20160420090: 5, 5

$ ./summarise.pl  --version
./summarise.pl version 0.0-alpha
(Getopt::Long::GetOptions version 2.51; Perl version 5.32.1)

$ ./summarise.pl  --help
summarise.pl
    Script for counting and summarising Numbers & Unique numbers in each
    line.

SYNOPSIS
    summarise.pl [options] [file ...]

OPTIONS
    --total, --sum, -t, -s
        print total number of IDs per timestamp

    --unique, -u
        print number of unique IDs per timestamp

    --both, -b
        print both unique and total number of IDs

    --version, -V
        Print script's version and exit

    --help, -h, -?
        Print this help message and exit

getopt_long()顺便说一句，就像许多 C 程序使用的GNU函数一样，perlGetopt::Long模块理解选项的缩写 - 例如--uni、--uniq、等将被视为与或-un相同。只要缩写足够长，能够明确地对应于选项名称，它就可以工作。-u--unique

而且，如果您想知道%timestamps散列数据结构“是什么样子”：

{
  "20160420084" => undef,
  "20160420085" => { 
     105618372 => 1, 111295547 => 1, 111344871 => 1, 111352520 => 1, 111379566 => 1,
     111394583 => 1, 111412662 => 1, 111413225 => 2, 111413281 => 2, 111413356 => 1,
     111413432 => 2, 111413557 => 1, 111557948 => 2, 111557953 => 1, 111633904 => 2,
     111646835 => 1, 111783178 => 1, 111783198 => 2, 111792767 => 2,
  },
  "20160420090" => {
     111295547 => 1, 111344871 => 1, 111352520 => 1, 111379566 => 1, 111394583 => 1
  },
}

哈希的哈希是一个关联数组（“哈希”），其中每个元素也可以是哈希。

perldoc perldsc有关 perl 数据结构的详细信息，请参阅参考资料。

相关内容