Iceberg Writer

Iceberg Writer plugin implements writing data to Apache Iceberg.

Configuration Example

This plugin is used to write data to Iceberg tables. For detailed configuration and parameters, please refer to the original Iceberg Writer documentation.

{
  "job": {
    "setting": {
      "speed": {
        "channel": 3
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0
      }
    },
    "content": [
      {
        "reader": {
          "name": "streamreader",
          "parameter": {
            "column": [
              {
                "value": "1",
                "type": "long"
              },
              {
                "value": "1989-06-04 00:00:00",
                "type": "timestamp"
              },
              {
                "value": "test1",
                "type": "string"
              }
            ],
            "sliceRecordCount": 1000
          }
        },
        "writer": {
          "name": "icebergwriter",
          "parameter": {
            "tableName": "test.test1",
            "writeMode": "truncate",
            "catalogType": "hadoop",
            "warehouse": "s3a://pvc-91d1e2cd-4d25-45c9-8613-6c4f7bf0a4cc/iceberg",
            "hadoopConfig": {
              "fs.s3a.endpoint": "http://localhost:9000",
              "fs.s3a.access.key": "gy0dX5lALP176g6c9fYf",
              "fs.s3a.secret.key": "ReuUrCzzu5wKWAegtswoHIWV389BYl9AB1ZQbiKr",
              "fs.s3a.connection.ssl.enabled": "false",
              "fs.s3a.path.style.access": "true",
              "fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem"
            }
          }
        }
      }
    ]
  }
}

Parameters

This plugin supports writing data to Iceberg with configurable catalog, table, and partition options.