JSON File Reader

JSON File Reader provides the ability to read data from local file system storage.

Configuration Example

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1,
        "bytes": -1
      }
    },
    "content": {
      "writer": {
        "name": "streamwriter",
        "parameter": {
          "print": "true"
        }
      },
      "reader": {
        "name": "jsonfilereader",
        "parameter": {
          "path": [
            "/tmp/test*.json"
          ],
          "column": [
            {
              "index": "$.id",
              "type": "long"
            },
            {
              "index": "$.name",
              "type": "string"
            },
            {
              "index": "$.age",
              "type": "long"
            },
            {
              "index": "$.score.math",
              "type": "double"
            },
            {
              "index": "$.score.english",
              "type": "double"
            },
            {
              "index": "$.pubdate",
              "type": "date"
            },
            {
              "type": "string",
              "value": "constant string"
            }
          ]
        }
      }
    }
  }
}

Where /tmp/test*.json are multiple copies of the same JSON file, with content as follows:

{"name": "zhangshan","id": 19890604,"age": 12,"score": {"math": 92.5,"english": 97.5,"chinese": 95},"pubdate": "2020-09-05"}
{"name": "lisi","id": 19890605,"age": 12,"score": {"math": 90.5,"english": 77.5,"chinese": 90},"pubdate": "2020-09-05"}
{"name": "wangwu","id": 19890606,"age": 12,"score": {"math": 89,"english": 100,"chinese": 92},"pubdate": "2020-09-05"}

Parameters

Configuration	Required	Data Type	Default Value	Description
path	Yes	list	None	Local file system path information, note that multiple paths can be supported
column	Yes	list	None	List of fields to read, type specifies the type of source data
compress	No	string	None	Text compression type, detected from the content and the suffix when empty
encoding	No	string	utf-8	Encoding configuration for reading files
singleLine	No	boolean	true	Whether each data record is on one line

path

Local file system path information, note that multiple paths can be supported, for example:

{
  "path": [
    "/var/ftp/test.json", // Read test.json file in /var/ftp directory
    "/var/tmp/*.json", // Read all json files in /var/tmp directory
    "/public/ftp", // Read all files in /public/ftp directory, if ftp is a file, read directly
    "/public/a??.json" // Read all files in /public directory starting with 'a', followed by two characters, ending with json
  ]
}

It is particularly important to note that if there are no matching files for extraction under the path specified by Path, Addax will report an error.

column

List of fields to read, type specifies the type of source data, index specifies the current column from json specification using Jayway JsonPath syntax, value specifies that the current type is constant, not reading data from source file, but automatically generating corresponding columns based on value. Users must specify Column field information.

For user-specified Column information, type must be filled, and index/value must choose one. An unsupported type is rejected while the job is set up.

When an index matches no value, the column holds NULL and one warning is logged per column; the task keeps running. A string column whose index points at an object or an array holds its json text instead of dropping the column.

compress

The compression type: zip, lzo, or any type commons-compress supports (such as gzip and bzip2, the spellings gz and bz2 are accepted as well).

Without this option the type is detected from the content. zip and lzo are not covered by that detection and fall back to the file suffix (.zip, .lzo), which means a xxx.zip archive is read without the option as well.

An unsupported compression type fails while the job is set up.

singleLine

There are two ways to store data in JSON format in the industry: one is one JSON object per line, which is Single Line JSON (aka. JSONL or JSON Lines); the other is that the entire file is a JSON array, and each element is a JSON object, which is Multiline JSON.

Addax supports one JSON object per line by default, i.e., singleLine = true. In this case, note that:

1. There should be no comma at the end of each line's JSON object, otherwise parsing will fail.
2. A JSON object cannot span multiple lines, otherwise parsing will fail.
3. Blank lines are skipped and do not affect the lines that follow.

If the data is an entire file as a JSON array with each element being a JSON object, you need to set singleLine to false. Suppose the data in the above example is represented in the following format:

{
  "result": [
    {
      "name": "zhangshan",
      "id": 19890604,
      "age": 12,
      "score": {
        "math": 92.5,
        "english": 97.5,
        "chinese": 95
      },
      "pubdate": "2020-09-05"
    },
    {
      "name": "lisi",
      "id": 19890605,
      "age": 12,
      "score": {
        "math": 90.5,
        "english": 77.5,
        "chinese": 90
      },
      "pubdate": "2020-09-05"
    },
    {
      "name": "wangwu",
      "id": 19890606,
      "age": 12,
      "score": {
        "math": 89,
        "english": 100,
        "chinese": 92
      },
      "pubdate": "2020-09-05"
    }
  ]
}

Because this format is valid JSON format, each JSON object can span multiple lines. Correspondingly, when reading such data, its column configuration should be filled as follows:

{
  "singleLine": false,
  "column": [
    {
      "index": "$.result[*].id",
      "type": "long"
    },
    {
      "index": "$.result[*].name",
      "type": "string"
    },
    {
      "index": "$.result[*].age",
      "type": "long"
    },
    {
      "index": "$.result[*].score.math",
      "type": "double"
    },
    {
      "index": "$.result[*].score.english",
      "type": "double"
    },
    {
      "index": "$.result[*].pubdate",
      "type": "date"
    },
    {
      "type": "string",
      "value": "constant string"
    }
  ]
}

For more detailed usage instructions, please refer to Jayway JsonPath syntax.

Every record of such a document is built from the columns by position, which requires the following of the configuration:

1. Every index has to be a multi-value path that matches a JSON array, otherwise the task fails.
2. Every index has to match the same number of elements, otherwise the task fails, so that values of different elements are not written into the same record.
3. Every index should point at a leaf field. When an intermediate level of the JSONPath is missing from some elements, json-path skips those elements, and the columns can still hold the same number of values while no longer lining up, which the reader cannot detect. For example, with $.data[*].a.v and $.data[*].b.w, a document whose elements carry a or b gives both columns 2 values, and the a.v of the third element ends up on the b.w of the second.

Note: When this type of data is in a JSON array, the program can only read the entire file into memory and then parse it, so it is not suitable for reading large files. For reading large files, it is recommended to use the format of one JSON object per line, which is the Single Line JSON format. This format can be read line by line without taking up too much memory.

Type Conversion

Addax Internal Type	Local File Data Type
Long	Long
Double	Double
String	String
Boolean	Boolean
Date	Date