☰

awk/sed/perl 一个衬垫来编辑 json 文件

tag-icon • tag-icon command-line awk sed perl json
awk/sed/perl 一个衬垫来编辑 json 文件

我们要添加以下几行：

export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g"

和：

export KAFKA_JVM_PERFORMANCE_OPTS=" -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80"

在“内容”行

我们可以得到如何使用 awk、sed、perl oneliner 等编辑文件的建议吗

更新前的 json 示例

{
  "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
  "items" : [
    {
      "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
      "tag" : "version1527250007610",
      "type" : "kafka-env",
      "version" : 8,
      "Config" : {
        "cluster_name" : "HDP",
        "stack_id" : "HDP-2.6"
      },
      "properties" : {
        "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",
        "is_supported_kafka_ranger" : "true",
        "kafka_log_dir" : "/var/log/kafka",
        "kafka_pid_dir" : "/var/run/kafka",
        "kafka_user" : "kafka",
        "kafka_user_nofile_limit" : "128000",
        "kafka_user_nproc_limit" : "65536"
      }
    }
  ]

预期输出（更新后的 json 示例）

{
  "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
  "items" : [
    {
      "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
      "tag" : "version1527250007610",
      "type" : "kafka-env",
      "version" : 8,
      "Config" : {
        "cluster_name" : "HDP",
        "stack_id" : "HDP-2.6"
      },
      "properties" : {
        "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi\nexport KAFKA_HEAP_OPTS=\"-Xmx8g -Xms8g\"\nKAFKA_JVM_PERFORMANCE_OPTS=\"-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80\n"",
        "is_supported_kafka_ranger" : "true",
        "kafka_log_dir" : "/var/log/kafka",
        "kafka_pid_dir" : "/var/run/kafka",
        "kafka_user" : "kafka",
        "kafka_user_nofile_limit" : "128000",
        "kafka_user_nproc_limit" : "65536"
      }
    }
  ]

更新后内容行的其他示例

 "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi\nexport KAFKA_HEAP_OPTS=\"-Xmx8g -Xms8g\"\nKAFKA_JVM_PERFORMANCE_OPTS=\"-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80\n"",

答案1

Perl 解决方案使用JSON模块：

perl -MJSON -we '
    $json = JSON->new->pretty->canonical;
    open $fh, "<", shift or die $!;
    $j = $json->decode(do { undef $/; <$fh> });
    $j->{items}[0]{properties}{content} .= qq(\nexport KAFKA_HEAP_OPTS="-Xmx8g -Xms8g"\nexport KAFKA_JVM_PERFORMANCE_OPTS=" -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80");
    print $json->encode($j);
' -- file.json > new.json

我必须修复您预期的文件（export第二个变量丢失，加上一些空格问题）并将其规范化以获得完全相同的输出。

答案2

不是一句单行话，而是...

$ new_lines='\\nexport KAFKA_HEAP_OPTS=\\"-Xmx8g -Xms8g\\"\\nexport KAFKA_JVM_PERFORMANCE_OPTS=\\" -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80\\"'
$ new_content=$( jq '.items[0].properties.content' file.json | sed 's/"$/'"$new_lines"'"/')
$ jq '.items[0].properties.content = '"$new_content" file.json
{
  "href": "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
  "items": [
    {
      "href": "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
      "tag": "version1527250007610",
      "type": "kafka-env",
      "version": 8,
      "Config": {
        "cluster_name": "HDP",
        "stack_id": "HDP-2.6"
      },
      "properties": {
        "content": "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi\nexport KAFKA_HEAP_OPTS=\"-Xmx8g -Xms8g\"\nexport KAFKA_JVM_PERFORMANCE_OPTS=\" -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80\"",
        "is_supported_kafka_ranger": "true",
        "kafka_log_dir": "/var/log/kafka",
        "kafka_pid_dir": "/var/run/kafka",
        "kafka_user": "kafka",
        "kafka_user_nofile_limit": "128000",
        "kafka_user_nproc_limit": "65536"
      }
    }
  ]
}

要验证新内容的可读性：

$ printf "$new_content\n"
"
#!/bin/bash

# Set KAFKA specific environment variables here.

# The java implementation to use.
export JAVA_HOME={{java64_home}}
export PATH=$PATH:$JAVA_HOME/bin
export PID_DIR={{kafka_pid_dir}}
export LOG_DIR={{kafka_log_dir}}
export KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}
export JMX_PORT=9997
# Add kafka sink to classpath and related depenencies
if [ -e "/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar" ]; then
  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar
  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*
fi

if [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then
. /etc/kafka/conf/kafka-ranger-env.sh
fi
export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g"
export KAFKA_JVM_PERFORMANCE_OPTS=" -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80""

答案3

这是一个简单的修复，使用 Perl 将行附加到行（每行），而不content解析 JSON 文档。这是一行代码（复制并粘贴到命令行中）。我猜你想要这个：

<in perl -nple'$_.="\\nexport KAFKA_HEAP_OPTS=\"-Xmx8g -Xms8g\"\\nexport KAFKA_JVM_PERFORMANCE_OPTS=\" -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80\"\\n\"" 
if/^\s+"content"/ '

这里的in开头是输入文件。您将在屏幕上看到输出，您可以轻松调整。您还可以让 Perl 使用该开关修改文件-i。 （先了解一下。）

输出是这样的：

{
  "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
  "items" : [
    {
      "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
      "tag" : "version1527250007610",
      "type" : "kafka-env",
      "version" : 8,
      "Config" : {
        "cluster_name" : "HDP",
        "stack_id" : "HDP-2.6"
      },
      "properties" : {
        "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",\nexport KAFKA_HEAP_OPTS="-Xmx8g -Xms8g"\n export KAFKA_JVM_PERFORMANCE_OPTS=" -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"\n"
        "is_supported_kafka_ranger" : "true",
        "kafka_log_dir" : "/var/log/kafka",
        "kafka_pid_dir" : "/var/run/kafka",
        "kafka_user" : "kafka",
        "kafka_user_nofile_limit" : "128000",
        "kafka_user_nproc_limit" : "65536"
      }
    }
  ]

正如你所看到的，我在这里做了更正：

export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g"\nexport KAFKA_JVM_PERFORMANCE_OPTS=" -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"\n

我更改了-XX选项布局，结尾是-XX:MaxMetaspaceFreeRatio=80"\n，而不是- XX:MaxMetaspaceFreeRatio=80\n"您所要求的。因此，如果这是您真正想要的，请更改此设置。另外，shebang 之前的换行符也是我要删除的内容。但我把这个留给了你。

答案4

和jq：

v1='export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g"'
v2='export KAFKA_JVM_PERFORMANCE_OPTS=" -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80"'

jq --arg v1 "$v1" --arg v2 "$v2" '.items[].properties.content += $v1 + "\n" + $v2 + "\n"' infile.json

相关内容