☰

如何仅过滤文件中的特定行？

tag-icon • tag-icon awk grep cut json
如何仅过滤文件中的特定行？

我有一个包含以下几行的文件。我只是想看到第一行、第二行国家/地区的 IP 地址只有一次，得分线也只有一次，但得分线应该是最高值，在本例中为 7.1。

{
  "ip": "86.75.227.72",
  "history": [
    {
     "created": "2012-03-22T07:26:00.000Z",
     "reason": "Regional Internet Registry",
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.64.0.0/12",
     "categoryDescriptions": {},
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1,
     "cats": {}
   },
   {
     "created": "2012-04-13T13:34:00.000Z",
     "reason": "DNS heuristics",
     "cats": {
       "Dynamic IPs": 100
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.64.0.0/12",
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "Based on statistical DNS analysis.",
     "score": 1
   },
   {
     "created": "2014-01-22T19:08:00.000Z",
     "reason": "DNS heuristics",
     "cats": {
       "Dynamic IPs": 86
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.72.0.0/14",
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "Based on statistical DNS analysis.",
     "score": 1
   },
   {
     "created": "2014-03-09T13:11:00.000Z",
     "reason": "DNS heuristics",
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.224.0/21",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "Based on statistical DNS analysis.",
     "score": 1
   },
   {
     "created": "2017-07-26T06:24:00.000Z",
     "reason": "Regional Internet Registry",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.224.0/21",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1
   },
   {
     "created": "2017-10-10T06:23:00.000Z",
     "reason": "Regional Internet Registry",
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.224.0/21",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1
   },
   {
     "created": "2017-10-18T06:23:00.000Z",
     "reason": "Regional Internet Registry",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.224.0/21",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1
   },
   {
     "created": "2017-11-20T18:16:00.000Z",
     "reason": "Third party feed",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.227.72/32",
     "cats": {
       "Dynamic IPs": 71,
       "Bots": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines.",
       "Bots": "IPs known for botnet-member activity. Devices using these IPs are obviously infected and take part in DDoS-attacks, port-scanning, spam-sending etc."
     },
     "reasonDescription": "This data was imported from a third party feed.",
     "score": 7.1
   },
   {
     "created": "2017-11-25T21:46:00.000Z",
     "reason": "Third party feed",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.227.72/32",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "This data was imported from a third party feed.",
     "score": 1
   }
 ],
 "subnets": [
   {
     "created": "2017-10-18T06:23:00.000Z",
     "reason": "Regional Internet Registry",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.64.0.0",
     "categoryDescriptions": {},
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1,
     "cats": {},
     "subnet": "86.64.0.0/12"
   },
   {
     "created": "2014-03-09T13:11:00.000Z",
     "reason": "DNS heuristics",
     "cats": {
       "Dynamic IPs": 71
     },
     "ip": "86.75.224.0",
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "Based on statistical DNS analysis.",
     "score": 1,
     "subnet": "86.75.224.0/21"
   }
 ],
 "cats": {
   "Dynamic IPs": 71
 },
 "geo": {
   "country": "France",
   "countrycode": "FR"
 },
 "score": 1,
 "reason": "Third party feed",
 "reasonDescription": "This data was imported from a third party feed.",
 "categoryDescriptions": {
   "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
 },
 "tags": []
}

"Bots":"已知僵尸网络成员活动的 IP。使用这些 IP 的设备显然已被感染并参与 DDoS 攻击

"score":7.1}
"geo":{"country":"France"
"score":1}]
"geo":{"country":"France"
"score":1
"score":1
"geo":{"country":"France"
"score":1

答案1

$ jq -r '.history | max_by(.score) | .ip' file.json
86.75.227.72/32

这用于查找数组中具有最大值的jq条目。一旦找到，就会从找到的条目中提取该值。.history.score.ip

输出格式略有不同，此处使用 CSV 输出 IP 地址、国家/地区、公司名称（如果有）以及数组中的分数.history：

$ jq -r '.history[] | [.ip, .geo.country, .asns."15557".Company, .score] | @csv' file.json
"86.64.0.0/12","France",,1
"86.64.0.0/12","France",,1
"86.72.0.0/14","France",,1
"86.75.224.0/21","France",,1
"86.75.224.0/21","France","LDCOMNET, FR",1
"86.75.224.0/21","France",,1
"86.75.224.0/21","France","LDCOMNET, FR",1
"86.75.227.72/32","France","LDCOMNET, FR",7.1
"86.75.227.72/32","France","LDCOMNET, FR",1

与此相同，但只取最高分条目：

$ jq -r '.history | max_by(.score) | [.ip, .geo.country, .asns."15557".Company, .score] | @csv' file.json
"86.75.227.72/32","France","LDCOMNET, FR",7.1

答案2

一种awk方法：

$ awk '$1=="\"ip\":"{
        ip=$2
       }
       $1=="\"country\":"{
        c[ip]=$2
       }
       $1=="\"score\":" && s[ip]<$2{ 
         s[ip]=$2
       }
       END{
           for(ip in c){
            print ip,c[ip],s[ip]
           }
       }' file 
"86.72.0.0/14", "France", 1
"86.64.0.0/12", "France", 1,
"86.75.224.0/21", "France", 1
"86.75.227.72/32", "France", 7.1
"86.75.227.72", "France", 
"86.75.224.0", "France", 1,

或者，如果您只想要得分最高的 IP 而没有其他：

$ awk '$1=="\"ip\":"{ip=$2}$1=="\"score\":" && score<$2{score=$2;sip=ip}END{print sip} ' file 
"86.75.227.72/32"

相关内容