我使用名为 airbyte 的 ETL 工具将数据从各种数据源移动到我们的主要数据仓库。我也在 airbyte slack 频道中问了这个问题,但也想在这里问。直到星期四(2023 年 7 月 6 日),从 mysql 数据库到 snowflake 的连接通过 airbyte 工作正常。但是,它停止工作了,我不确定这是什么原因。我在下面添加了日志文件的输出。有人知道问题可能来自哪里吗?感谢您的支持
编辑:我尝试从其他 ETL 工具(Airbyte、Fivetran)连接到该数据库,但收到此错误:
Unable to validate certificate
com.fivetran.lambda.Retrier$RetrierException: The action failed after 1 attempt(s), the failures were: 1) java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
尽管提供商提到他们那边(mysql 数据库)没有任何变化,但这似乎与他们有关,特别是因为我在使用不同的工具和客户端时收到了相同的错误。
编辑:我可以通过 ubuntu CLI 和 mysql workbench 访问 mysql 数据库。我仍然不确定为什么第三方提供商无法使用此功能
来自 airbyte 的日志文件:
2023-07-08 07:04:31 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-08 07:04:31 [32mINFO[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.auth_type: must be a constant value OAuth2.0, $.auth_type: does not have a value in the enumeration [OAuth2.0], $.access_token: is missing but it is required, $.refresh_token: is missing but it is required
2023-07-08 07:04:31 [32mINFO[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.auth_type: must be a constant value Key Pair Authentication, $.auth_type: does not have a value in the enumeration [Key Pair Authentication], $.private_key: is missing but it is required
2023-07-08 07:04:31 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-08 07:04:31 [32mINFO[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.password: object found, string expected
2023-07-08 07:04:31 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/63142/0/logs.log
2023-07-08 07:04:31 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.23
2023-07-08 07:04:31 [32mINFO[m i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to save workflow id for cancellation
2023-07-08 07:04:32 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
2023-07-08 07:04:32 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK -----
2023-07-08 07:04:32 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
2023-07-08 07:04:32 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/source-mysql:2.0.11 exists...
2023-07-08 07:04:43 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - airbyte/source-mysql:2.0.11 was found locally.
2023-07-08 07:04:44 [32mINFO[m i.a.w.p.DockerProcessFactory(create):119 - Creating docker container = source-mysql-check-63142-0-izimd with resources io.airbyte.config.ResourceRequirements@525765e7[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2023-07-08 07:04:44 [32mINFO[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/63142/0 --log-driver none --name source-mysql-check-63142-0-izimd --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=airbyte/source-mysql:2.0.11 -e WORKER_JOB_ATTEMPT=0 -e AUTO_DETECT_SCHEMA=false -e AIRBYTE_VERSION=0.40.23 -e WORKER_JOB_ID=63142 airbyte/source-mysql:2.0.11 check --config source_config.json
2023-07-08 07:04:44 [32mINFO[m i.a.w.i.VersionedAirbyteStreamFactory(create):97 - Reading messages from protocol version 0.2.0
2023-07-08 07:05:30 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.s.m.MySqlSource(main):407 starting source: class io.airbyte.integrations.source.mysql.MySqlSource
2023-07-08 07:05:30 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {check=null, config=source_config.json}
2023-07-08 07:05:30 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationRunner(runInternal):108 Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2023-07-08 07:05:30 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationRunner(runInternal):109 Command: CHECK
2023-07-08 07:05:30 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationRunner(runInternal):110 Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2023-07-08 07:05:31 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-08 07:05:31 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-08 07:05:32 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.s.SshTunnel(getInstance):204 Starting connection with method: NO_TUNNEL
2023-07-08 07:05:33 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO c.z.h.HikariDataSource(<init>):80 HikariPool-1 - Starting...
2023-07-08 07:05:33 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO c.z.h.HikariDataSource(<init>):82 HikariPool-1 - Start completed.
2023-07-08 07:06:33 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO c.z.h.HikariDataSource(close):350 HikariPool-1 - Shutdown initiated...
2023-07-08 07:06:36 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO c.z.h.HikariDataSource(close):352 HikariPool-1 - Shutdown completed.
2023-07-08 07:06:36 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationRunner(runInternal):186 Completed integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2023-07-08 07:06:36 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.s.m.MySqlSource(main):409 completed source: class io.airbyte.integrations.source.mysql.MySqlSource
2023-07-08 07:06:36 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
2023-07-08 07:06:36 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-07-08 07:06:36 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
2023-07-08 07:06:36 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
2023-07-08 07:06:36 [33mWARN[m i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - Failure during reporting of activity result to the server. ActivityId = cc93e74a-db81-37bc-a6b6-3895b739fa1f, ActivityType = RunWithJobOutput, WorkflowId=connection_manager_c68a63fd-ec58-45a3-b8ee-8d787ddf52e7, WorkflowType=ConnectionManagerWorkflow, RunId=61607809-4637-4161-b0f7-ef83a37e2d9d
io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.50.2.jar:1.50.2]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.50.2.jar:1.50.2]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.50.2.jar:1.50.2]
at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskCompleted(WorkflowServiceGrpc.java:3840) ~[temporal-serviceclient-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$0(ActivityWorker.java:303) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52) ~[temporal-serviceclient-1.17.0.jar:?]
at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
at io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50) ~[temporal-serviceclient-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:298) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-07-08 07:07:02 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/63142/0/logs.log
2023-07-08 07:07:02 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.23
2023-07-08 07:07:02 [32mINFO[m i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to save workflow id for cancellation
2023-07-08 07:07:02 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
2023-07-08 07:07:02 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK -----
2023-07-08 07:07:02 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
2023-07-08 07:07:02 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/source-mysql:2.0.11 exists...
2023-07-08 07:07:02 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - airbyte/source-mysql:2.0.11 was found locally.
2023-07-08 07:07:02 [32mINFO[m i.a.w.p.DockerProcessFactory(create):119 - Creating docker container = source-mysql-check-63142-0-otjun with resources io.airbyte.config.ResourceRequirements@525765e7[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2023-07-08 07:07:02 [32mINFO[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/63142/0 --log-driver none --name source-mysql-check-63142-0-otjun --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=airbyte/source-mysql:2.0.11 -e WORKER_JOB_ATTEMPT=0 -e AUTO_DETECT_SCHEMA=false -e AIRBYTE_VERSION=0.40.23 -e WORKER_JOB_ID=63142 airbyte/source-mysql:2.0.11 check --config source_config.json
2023-07-08 07:07:02 [32mINFO[m i.a.w.i.VersionedAirbyteStreamFactory(create):97 - Reading messages from protocol version 0.2.0
2023-07-08 07:07:03 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.s.m.MySqlSource(main):407 starting source: class io.airbyte.integrations.source.mysql.MySqlSource
2023-07-08 07:07:03 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {check=null, config=source_config.json}
2023-07-08 07:07:03 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationRunner(runInternal):108 Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2023-07-08 07:07:03 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationRunner(runInternal):109 Command: CHECK
2023-07-08 07:07:03 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationRunner(runInternal):110 Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2023-07-08 07:07:03 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-08 07:07:04 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-07-08 07:07:04 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.s.SshTunnel(getInstance):204 Starting connection with method: NO_TUNNEL
2023-07-08 07:07:04 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO c.z.h.HikariDataSource(<init>):80 HikariPool-1 - Starting...
2023-07-08 07:07:04 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO c.z.h.HikariDataSource(<init>):82 HikariPool-1 - Start completed.
2023-07-08 07:08:04 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO c.z.h.HikariDataSource(close):350 HikariPool-1 - Shutdown initiated...
2023-07-08 07:08:05 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO c.z.h.HikariDataSource(close):352 HikariPool-1 - Shutdown completed.
2023-07-08 07:08:05 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.b.IntegrationRunner(runInternal):186 Completed integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2023-07-08 07:08:05 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - INFO i.a.i.s.m.MySqlSource(main):409 completed source: class io.airbyte.integrations.source.mysql.MySqlSource
2023-07-08 07:08:05 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-07-08 07:08:05 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
2023-07-08 07:08:05 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
2023-07-08 07:08:05 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
2023-07-08 07:08:06 [32mINFO[m i.a.n.SlackNotificationClient(notify):149 - Successful notification (200): ok
2023-07-08 07:08:06 [32mINFO[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.auth_type: must be a constant value OAuth2.0, $.auth_type: does not have a value in the enumeration [OAuth2.0], $.access_token: is missing but it is required, $.refresh_token: is missing but it is required
2023-07-08 07:08:06 [32mINFO[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.auth_type: must be a constant value Key Pair Authentication, $.auth_type: does not have a value in the enumeration [Key Pair Authentication], $.private_key: is missing but it is required
2023-07-08 07:08:06 [32mINFO[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.password: object found, string expected
2023-07-08 07:08:06 [32mINFO[m i.a.c.f.EnvVariableFeatureFlags(autoDisablesFailingConnections):20 - Auto Disable Failing Connections: false
答案1
虽然我不确定为什么会发生这种情况,但现在我能够再次通过第三方客户端连接到数据仓库。今天增加了连接超时时间,所以这似乎起了作用。然而,我仍然不确定为什么这有帮助,因为连接本身在过去几个月里没有更新过