TxtFile Writer

TxtFile Writer provides writing CSV-like format to one or more table files in local file system.

Configuration Example

{
  "job": {
    "setting": {
      "speed": {
        "channel": 2,
        "bytes": -1
      }
    },
    "content": {
      "reader": {
        "name": "txtfilereader",
        "parameter": {
          "path": [
            "/tmp/data"
          ],
          "encoding": "UTF-8",
          "column": [
            {
              "index": 0,
              "type": "long"
            },
            {
              "index": 1,
              "type": "boolean"
            },
            {
              "index": 2,
              "type": "double"
            },
            {
              "index": 3,
              "type": "string"
            },
            {
              "index": 4,
              "type": "date",
              "format": "yyyy.MM.dd"
            }
          ],
          "fieldDelimiter": ","
        }
      },
      "writer": {
        "name": "txtfilewriter",
        "parameter": {
          "path": "/tmp/result",
          "fileName": "luohw",
          "writeMode": "truncate",
          "dateFormat": "yyyy-MM-dd"
        }
      }
    }
  }
}

Parameters

Configuration	Required	Data Type	Default Value	Description
path	Yes	string	None	Path information of local file system, write multiple files under Path directory
fileName	Yes	string	None	Name of file to write, this filename will have random suffix added as actual filename for each thread
writeMode	Yes	string	None	Data cleanup processing mode before writing, see below
fieldDelimiter	Yes	string	,	Field delimiter for reading
compress	No	string	None	Text compression type, supports zip, lzo, lzop, tgz, bzip2
encoding	No	string	utf-8	Encoding configuration for reading files
nullFormat	No	string	\N	Define which strings can represent null
dateFormat	No	string	None	Format when date type data is serialized to file, e.g. "yyyy-MM-dd"
fileFormat	No	string	text	Format of file output, see below
table	Yes	string	None	Table name to specify in SQL mode
column	No	list	None	Optional column names to specify in SQL mode
extendedInsert	No	boolean	true	Whether to use batch insert syntax in SQL mode, see below
batchSize	No	int	2048	Batch size for batch insert syntax in SQL mode, see below
header	No	list	None	Table header for text output, example ['id', 'name', 'age']

writeMode

Data cleanup processing mode before writing:

• truncate: Clean all files with fileName prefix under directory before writing.
• append: No processing before writing, write directly using filename and ensure no filename conflicts.
• nonConflict: If there are files with fileName prefix under directory, report error directly.

fileFormat

Format of file output, including csv, text, and sql (introduced in version 4.1.3).

csv and text are handled by the same CSV writer, so both escape a field that contains the column delimiter, a double quote or a line break, using double quotes " as the escape symbol. When fieldDelimiter is set to \n or \r (the historical "one field per line" form), fields are joined with that delimiter unescaped instead.

SQL format means writing data to file in SQL statement (INSERT INTO ... VALUES) format.

Binary columns

A Bytes column (Oracle BLOB/RAW/LONG RAW, MySQL BLOB/VARBINARY, and so on) is written as a base64 string rather than decoded with encoding: decoding it as text replaces every byte above 0x7F with U+FFFD and writes 0x0A through untouched, which breaks the record line. A SQL NULL still renders as nullFormat (default \N). The sql file format emits the same base64 string literal.

table

Only required in sql file format, used to specify the table name to write to.

column

In sql file format, you can specify column names to write. If specified, the sql statement is like INSERT INTO table (col1, col2, col3) VALUES (val1, val2, val3), otherwise it's INSERT INTO table VALUES (val1, val2, val3).

extendedInsert

Whether to enable batch insert syntax. If enabled, batchSize number of data will be written to file at once, otherwise each data is one line. This parameter borrows from the extended-insert parameter syntax of mysqldump tool.

batchSize

Batch size for batch insert syntax. If extendedInsert is true, every batchSize data will be written to file at once, otherwise each data is one line.

Type Conversion

Addax Internal Type	Local File Data Type
Long	Long
Double	Double
string	string
Boolean	Boolean
Date	Date
Bytes	Base64 string