☰

过滤 opentelemetry-agent 发送到 Coralogix 的日志

tag-icon • tag-icon kubernetes metrics
过滤 opentelemetry-agent 发送到 Coralogix 的日志

我正在尝试弄清楚如何配置它过滤处理器过滤某些日志。

我在 minikube 中为自己创建了一个游乐场，其中的应用程序可以随机生成日志，并安装了 otel 集成到珊瑚虫。

日志示例：

{
    "level": "error",
    "ts": 1709652353.609003,
    "caller": "logs-generator/logs-generator.go:60",
    "msg": "Each Set Am.",
    "error": "expected header is undefined",
    "stacktrace": "anatoly.dev/dummy-logs/internal/logs-generator.(*LogsGenerator).GenerateRandomErrorLog\n\t/app/internal/logs-generator/logs-generator.go:60\nanatoly.dev/dummy-logs/internal/logs-generator.(*LogsGenerator).GenerateRandomLog\n\t/app/internal/logs-generator/logs-generator.go:71\nanatoly.dev/dummy-logs/cmd.NewRootCommand.NewStartCommand.func1\n\t/app/cmd/start.go:38\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:983\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039\ngithub.com/spf13/cobra.(*Command).ExecuteContext\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032\nmain.main\n\t/app/main.go:14\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"
  },
  {
    "level": "warn",
    "ts": 1709652354.6102042,
    "caller": "logs-generator/logs-generator.go:47",
    "msg": "The SMS circuit is down, construct the redundant monitor so we can synthesize the EXE sensor!",
    "name": "Silas Anderson",
    "email": "[email protected]",
    "phone": "8381490753",
    "color": "White"
  },
{
    "level": "info",
    "ts": 1709652350.6055088,
    "caller": "logs-generator/logs-generator.go:47",
    "msg": "The SSL array is down, deconstruct the online protocol so we can load the AI interface!",
    "name": "George Rice",
    "email": "[email protected]",
    "phone": "3695382906",
    "color": "Blue"
  },
  {
    "level": "debug",
    "ts": 1709652351.6065476,
    "caller": "logs-generator/logs-generator.go:47",
    "msg": "If we compress the protocol, we can get to the HDD circuit through the 1080p JSON port!",
    "name": "Kamron Marvin",
    "email": "[email protected]",
    "phone": "4465694677",
    "color": "Green"
  },

你可以看到，它有不同的日志级别，不同的颜色等等...现在我想配置这个过滤处理器因此某些类型的日志将被丢弃。例如具有以下级别的日志error：debug

我尝试了以下操作：

  opentelemetry-agent:
    config:
      exporters:
        debug:
          verbosity: detailed
      processors:
        filter/ottl:
          error_mode: ignore
          logs:
            log_record:
              - 'IsMatch(body["level"], "debug")'
              - 'IsMatch(body["level"], "error")'
      service:
        pipelines:
          logs:
            exporters:
              - debug
            processors:
              - filter/ottl
              - k8sattributes
              - resourcedetection/env
              - resourcedetection/region
              - batch

没用，我仍然可以看到这些日志传到 Coralogix。我尝试了多种方法，但都不起作用。

你可以看到我在这里使用了调试导出器试图弄清楚需要过滤什么，但这对我没有帮助，您可以在此处看到调试的示例输出：

{
  "level": "info",
  "ts": 1709665690.10114,
  "msg": {
    "ResourceLog #0": {
      "Resource SchemaURL": "https://opentelemetry.io/schemas/1.6.1",
      "Resource attributes": {
        "-> k8s.container.name": "Str(dummylogs)",
        "-> k8s.namespace.name": "Str(default)",
        "-> k8s.pod.name": "Str(dummylogs-58f59cf6c8-mtn7q)",
        "-> k8s.container.restart_count": "Str(0)",
        "-> k8s.pod.uid": "Str(2b42cdfc-05bf-4004-bca5-a5f74cb899e3)",
        "-> k8s.cluster.name": "Str(coralogix-dummylogs-2)",
        "-> cx.otel_integration.name": "Str(coralogix-integration-helm)",
        "-> k8s.deployment.name": "Str(dummylogs)",
        "-> k8s.node.name": "Str(coralogix-dummylogs-2)",
        "-> host.name": "Str(coralogix-dummylogs-2)",
        "-> os.type": "Str(linux)",
        "-> host.id": "Str(61419f744ec9452499a59356fc030992)"
      }
    },
    "ScopeLogs #0": {
      "ScopeLogs SchemaURL": "InstrumentationScope",
      "LogRecord #0": {
        "ObservedTimestamp": "2024-03-05 19:08:09.996222879 +0000 UTC",
        "Timestamp": "2024-03-05 19:08:09.845494796 +0000 UTC",
        "SeverityText": "",
        "SeverityNumber": "Unspecified(0)",
        "Body": "Str({\"level\":\"error\",\"ts\":1709665689.8447518,\"caller\":\"logs-generator/logs-generator.go:60\",\"msg\":\"Those Meeting Dig.\",\"error\":\"error\",\"stacktrace\":\"anatoly.dev/dummy-logs/internal/logs-generator.(*LogsGenerator).GenerateRandomErrorLog\\n\\t/app/internal/logs-generator/logs-generator.go:60\\nanatoly.dev/dummy-logs/internal/logs-generator.(*LogsGenerator).GenerateRandomLog\\n\\t/app/internal/logs-generator/logs-generator.go:71\\nanatoly.dev/dummy-logs/cmd.NewRootCommand.NewStartCommand.func1\\n\\t/app/cmd/start.go:38\\ngithub.com/spf13/cobra.(*Command).execute\\n\\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:983\\ngithub.com/spf13/cobra.(*Command).ExecuteC\\n\\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115\\ngithub.com/spf13/cobra.(*Command).Execute\\n\\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039\\ngithub.com/spf13/cobra.(*Command).ExecuteContext\\n\\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032\\nmain.main\\n\\t/app/main.go:14\\nruntime.main\\n\\t/usr/local/go/src/runtime/proc.go:271\"})",
        "Attributes": {
          "-> log.file.path": "Str(/var/log/pods/default_dummylogs-58f59cf6c8-mtn7q_2b42cdfc-05bf-4004-bca5-a5f74cb899e3/dummylogs/0.log)",
          "-> time": "Str(2024-03-05T19:08:09.845494796Z)",
          "-> log.iostream": "Str(stderr)",
          "Trace ID": "",
          "Span ID": "",
          "Flags": "0"
        },
        "kind": "exporter",
        "data_type": "logs",
        "name": "debug"
      }
    }
  }
}

或者

{
  "level": "info",
  "ts": 1709665689.2971187,
  "msg": {
    "ResourceLog #0": {
      "Resource SchemaURL": "https://opentelemetry.io/schemas/1.6.1",
      "Resource attributes": {
        "-> k8s.container.restart_count": "Str(0)",
        "-> k8s.pod.uid": "Str(2b42cdfc-05bf-4004-bca5-a5f74cb899e3)",
        "-> k8s.container.name": "Str(dummylogs)",
        "-> k8s.namespace.name": "Str(default)",
        "-> k8s.pod.name": "Str(dummylogs-58f59cf6c8-mtn7q)",
        "-> k8s.cluster.name": "Str(coralogix-dummylogs-2)",
        "-> cx.otel_integration.name": "Str(coralogix-integration-helm)",
        "-> k8s.deployment.name": "Str(dummylogs)",
        "-> k8s.node.name": "Str(coralogix-dummylogs-2)",
        "-> host.name": "Str(coralogix-dummylogs-2)",
        "-> os.type": "Str(linux)",
        "-> host.id": "Str(61419f744ec9452499a59356fc030992)"
      }
    },
    "ScopeLogs #0": {
      "ScopeLogs SchemaURL": "InstrumentationScope",
      "LogRecord #0": {
        "ObservedTimestamp": "2024-03-05 19:08:08.995622545 +0000 UTC",
        "Timestamp": "2024-03-05 19:08:08.84215517 +0000 UTC",
        "SeverityText": "",
        "SeverityNumber": "Unspecified(0)",
        "Body": "Str({\"level\":\"debug\",\"ts\":1709665688.8411458,\"caller\":\"logs-generator/logs-generator.go:47\",\"msg\":\"I'll bundle the haptic ADP bandwidth, that should construct the PCI circuit!\",\"name\":\"Laurianne Aufderhar\",\"email\":\"[email protected]\",\"phone\":\"2654965339\",\"color\":\"Green\"})",
        "Attributes": {
          "-> log.file.path": "Str(/var/log/pods/default_dummylogs-58f59cf6c8-mtn7q_2b42cdfc-05bf-4004-bca5-a5f74cb899e3/dummylogs/0.log)",
          "-> log.iostream": "Str(stderr)",
          "-> time": "Str(2024-03-05T19:08:08.84215517Z)",
          "Trace ID": "",
          "Span ID": "",
          "Flags": "0"
        },
        "kind": "exporter",
        "data_type": "logs",
        "name": "debug"
      }
    }
  }
}

因此，我的问题是如何使用定义 filterprocessorOTTL 语言根据以下内容过滤日志：

  1. 级别，即错误、调试信息
  2. 基于k8s.node.name和水平一起

答案1

我不是 OTEL 专家，也从未从头编写过 OTEL 代理配置，但似乎您正在尝试过滤字符串字段，但您的过滤表达式将其视为 JSON 对象

我猜这个表达式应该大致如此

'IsMatch(ParseJSON(body)["level"], "debug")'

相关内容