Stream Reader
Stream Reader is a plugin that reads data from memory, mainly used to quickly generate expected data and test write plugins.
A complete StreamReader configuration file is as follows:
{
"reader": {
"name": "streamreader",
"parameter": {
"column": [
{
"value": "unique_id",
"type": "string"
},
{
"value": "1989-06-04 08:12:13",
"type": "date",
"dateFormat": "yyyy-MM-dd HH:mm:ss"
},
{
"value": 1984,
"type": "long"
},
{
"value": 1989.64,
"type": "double"
},
{
"value": true,
"type": "bool"
},
{
"value": "a long text",
"type": "bytes"
}
],
"sliceRecordCount": 10
}
}
}The above configuration file will generate 10 records (assuming channel is 1), with each record containing:
unique_id,'1989-06-04 08:12:13',1984,1989.64,true,'a long text'
Currently StreamReader supports all output data types listed above:
stringString typedateDate typelongAll integer typesdoubleAll floating point numbersboolBoolean typebytesByte type
The date type also supports dateFormat configuration to specify the format of input dates, default is yyyy-MM-dd HH:mm:ss. For example, your input can be like this:
{
"value": "1989/06/04 12:13:14",
"type": "date",
"dateFormat": "yyyy/MM/dd HH:mm:ss"
}Note that regardless of the input format for date type, it is internally converted to yyyy-MM-dd HH:mm:ss format.
Use HH (the hour of a 24 hour clock) for the hour in a date format. hh is the hour of a 12 hour clock and needs an AM/PM marker to be read, so a format with hh but without the marker fails the job at startup with a message that asks for HH.
StreamReader also supports random input functionality. For example, to randomly get any integer between 0-10, we can configure the column like this:
{
"random": "0,10",
"type": "long"
}To get a random floating point number between 0 and 100, configure like this:
{
"random": "0,100",
"type": "double"
}To specify decimal places for floating point numbers, e.g., 2 decimal places, configure like this:
{
"random": "0,100,2",
"type": "double"
}Note: It cannot guarantee that the generated decimal always has exactly 2 places. If the decimal part is 0, the decimal places will be fewer than specified.
Here we use the random keyword to indicate its value is random, with the range being a closed interval.
Other random type configurations are as follows:
long: random 0, 10 - random number between 0 and 10string: random 0, 10 - random string of length between 0 and 10bool: random 0, 10 - ratio of false and true occurrencesdouble: random 0, 10 - random floating point between 0 and 10double: random 0, 10, 2 - random floating point between 0 and 10 with 2 decimal placesdate: random '2014-07-07 00:00:00', '2016-07-07 00:00:00' - random time between start time and end time, default date format (commas not supported) yyyy-MM-dd HH:mm:ssBYTES: random 0, 10 - random string of length between 0 and 10, get its UTF-8 encoded binary string
StreamReader also supports increment functions. For example, to get an arithmetic sequence starting from 1 with increment of 5, configure like this:
{
"incr": "1,5",
"type": "long"
}To get a decreasing sequence, change the step size (5 in the above example) to negative. Default step size is 1.
Increment also supports date type (introduced in version 4.0.1), for example:
{
"incr": "1989-06-04 09:01:02,2,d",
"type": "date"
}incr consists of three parts: start date, step size, and step unit, separated by English commas (,).
- Start date: Correct date string, default format is
yyyy-MM-dd HH:mm:ss. If time format is different, need to configuredateFormatto specify date format. This is mandatory. - Step size: Length to increase each time, default is 1. For decreasing, fill in negative number. This is optional.
- Step unit: What time unit to increment/decrement by, default is by day. This is optional. Available units:
- d/day
- M/month
- y/year
- h/hour
- m/minute
- s/second
- w/week
Configuration item sliceRecordCount specifies the number of data records to generate. If channel is specified, actual generated records = sliceRecordCount * channel
Built-in data rules
Besides generating constants, random values and increment sequences by type, StreamReader can generate data that looks like the data of a real system - an ID card number, a bank card number, an address, a company name. Such a column names its rule with the rule item, needs no value:
{
"rule": "idCard"
}A complete job that uses every rule:
{
"job": {
"setting": {
"speed": {
"byte": -1,
"channel": 1
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": {
"reader": {
"name": "streamreader",
"parameter": {
"column": [
{
"value": "1,100,2",
"rule": "random",
"type": "double"
},
{
"value": "Addax",
"type": "string"
},
{
"value": "1",
"rule": "incr",
"type": "long"
},
{
"value": "1989/06/04 00:00:01,-1",
"rule": "incr",
"type": "date",
"dateFormat": "yyyy/MM/dd HH:mm:ss"
},
{
"value": "test",
"type": "bytes"
},
{
"rule": "address"
},
{
"rule": "bank"
},
{
"rule": "company"
},
{
"rule": "creditCard"
},
{
"rule": "debitCard"
},
{
"rule": "idCard"
},
{
"rule": "lat"
},
{
"rule": "lng"
},
{
"rule": "name"
},
{
"rule": "job"
},
{
"rule": "phone"
},
{
"rule": "stockCode"
},
{
"rule": "stockAccount"
}
],
"sliceRecordCount": 10
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"print": true,
"encoding": "UTF-8"
}
}
}
}
}The output of that job looks like this:
$ bin/addax.sh job/streamreader-rules.json
___ _ _
/ _ \ | | | |
/ /_\ \ __| | __| | __ ___ __
| _ |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |> <
\_| |_/\__,_|\__,_|\__,_/_/\_\
:: Addax version :: (v6.0.14-SNAPSHOT)
2026-09-13 14:17:39.640 [ main] INFO Engine -
{
"setting":{
"speed":{
"byte":-1,
"channel":1
},
"errorLimit":{
"record":0,
"percentage":0.02
}
},
"content":{
"reader":{
"name":"streamreader",
"parameter":{
"column":[
{
"value":"1,100,2",
"rule":"random",
"type":"double"
},
{
"value":"Addax",
"type":"string"
},
{
"value":"1",
"rule":"incr",
"type":"long"
},
{
"value":"1989/06/04 00:00:01,-1",
"rule":"incr",
"type":"date",
"dateFormat":"yyyy/MM/dd HH:mm:ss"
},
{
"value":"test",
"type":"bytes"
},
{
"rule":"address"
},
{
"rule":"bank"
},
{
"rule":"company"
},
{
"rule":"creditCard"
},
{
"rule":"debitCard"
},
{
"rule":"idCard"
},
{
"rule":"lat"
},
{
"rule":"lng"
},
{
"rule":"name"
},
{
"rule":"job"
},
{
"rule":"phone"
},
{
"rule":"stockCode"
},
{
"rule":"stockAccount"
}
],
"sliceRecordCount":10
}
},
"writer":{
"name":"streamwriter",
"parameter":{
"print":true,
"encoding":"UTF-8"
}
}
}
}
2026-09-13 14:17:39.650 [ main] INFO JobContainer - The jobContainer begins to process the job.
2026-09-13 14:17:39.657 [ job-0] INFO JobContainer - The Reader.Job [streamreader] perform prepare work .
2026-09-13 14:17:39.657 [ job-0] INFO JobContainer - The Writer.Job [streamwriter] perform prepare work .
2026-09-13 14:17:39.657 [ job-0] INFO JobContainer - Job set Channel-Number to 1 channel(s).
2026-09-13 14:17:39.657 [ job-0] INFO JobContainer - The Reader.Job [streamreader] is divided into [1] task(s).
2026-09-13 14:17:39.657 [ job-0] INFO JobContainer - The Writer.Job [streamwriter] is divided into [1] task(s).
2026-09-13 14:17:39.666 [ job-0] INFO JobContainer - The Scheduler launches [1] taskGroup(s).
2026-09-13 14:17:39.668 [ taskGroup-0] INFO TaskGroupContainer - The taskGroupId=[0] started [1] channels for [1] tasks.
2026-09-13 14:17:39.670 [ taskGroup-0] INFO Channel - The Channel set byte_speed_limit to -1, No bps activated.
2026-09-13 14:17:39.670 [ taskGroup-0] INFO Channel - The Channel set record_speed_limit to -1, No tps activated.
98.27 Addax 1 1989-06-04 00:00:01 0x74657374 宁夏回族自治区邯郸市龙潭区新华路249号 阳泉商行 九方房地产集团有限公司 4096678316310982 6217252221144286715 421126196801114274 -3.3966765 -151.6848752 狄红 安防主管 13538523906 008224 8680205898
81.33 Addax 2 1989-06-03 00:00:01 0x74657374 西藏自治区昆明县山亭区韶山街102号 盛京银行 联软办公有限合伙企业 6223473408795921 6213428525497728769 420117193803291935 45.4005918 126.2302139 芦萍 西点师 13420192359 836128 3569253894
28.62 Addax 3 1989-06-02 00:00:01 0x74657374 河南省兴安盟市海港区永联街265号 乌海银行 兰金电子餐饮有限公司 6227593032063763 6212939436209713059 320115200704155336 -37.3717424 83.8123369 孙林 列车/地铁车长 13077891026 883280 5750169522
7.38 Addax 4 1989-06-01 00:00:01 0x74657374 北京市淮安县六枝特区白山路5号 张家口商行 昊嘉文化有限合伙企业 6283121239374080 6235866437693714523 530802196306243564 -81.0485543 -176.9651246 勾玉 锅炉工程师/技师 15088018035 164106 2892469333
27.8 Addax 5 1989-05-31 00:00:01 0x74657374 香港特别行政区太原县平山区黄河街214号 承德银行 精芯机械中外合作有限公司 6259058913047415 6235868387408215337 320305197301266078 -30.3777802 73.0269088 巫凯 印刷排版/制版 19140432084 400570 1119544796
71.41 Addax 6 1989-05-30 00:00:01 0x74657374 河北省辛集市怀柔区碧海路50号 晋城商行 恩悌网络股份有限公司 6259075791345102 6213330694668956477 350582200507135392 54.4136463 62.497184 沙海燕 化验员 15899573264 405982 1551956384
42.79 Addax 7 1989-05-29 00:00:01 0x74657374 湖北省拉萨市梁平区白云街80号 中国建设银行 兰金电子餐饮集团 3568353959366817 6230407263361590072 520121193407021739 -36.5783856 -23.1595399 唐磊 CNC工程师 15762242326 031212 4222388439
89.21 Addax 8 1989-05-28 00:00:01 0x74657374 山东省长春县华龙区解放路239号 民生银行 MBP软件物流股份有限公司 6227539162714957 6213423889811345807 450107193805289620 59.4473891 126.722241 荆秀芳 瘦身顾问 14533489851 685121 2386678456
77.44 Addax 9 1989-05-27 00:00:01 0x74657374 广西壮族自治区合肥市普陀区光明路137号 天津银行 天开信息中外合作有限公司 6259068931268261 6200618145417973849 441600199110099977 70.7287609 143.8947897 滕雪梅 公共卫生/疾病控制 15030051423 682375 4332730599
1.67 Addax 10 1989-05-26 00:00:01 0x74657374 江西省惠州市双滦区七星路155号 中国银行 襄樊地球村信息技术有限公司 5477660282898100 6212939142939212015 130522195402117955 -12.8224904 175.018515 强杨 财务分析经理/主管 13046686503 029874 5578465632
2026-09-13 14:17:42.674 [ job-0] INFO AbstractScheduler - The scheduler has completed all tasks.
2026-09-13 14:17:42.676 [ job-0] INFO JobContainer - The Writer.Job [streamwriter] perform post work.
2026-09-13 14:17:42.677 [ job-0] INFO JobContainer - The Reader.Job [streamreader] perform post work.
2026-09-13 14:17:42.689 [ job-0] INFO StandAloneJobContainerCommunicator - Total 10 records, 1715 bytes | Speed 571B/s, 3 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.010s | Percentage 100.00%
2026-09-13 14:17:42.690 [ job-0] INFO JobContainer -
Job start at : 2026-09-13 14:17:39
Job end at : 2026-09-13 14:17:42
Job took secs : 3s
Total bytes : 1715
Average bps : 571B/s
Average rps : 3rec/s
Number of rec : 10
Failed record : 0The built-in rules are:
| Rule | Description | Example | Type | Note |
|---|---|---|---|---|
address | A domestic address | 辽宁省兰州市徐汇区东山街176号 | string | |
bank | A domestic bank name | 华夏银行 | string | |
company | A domestic company name | 万迅电脑科技有限公司 | string | |
creditCard | A credit card number | 6227544006180760 | string | 16 digits |
debitCard | A debit card number | 6216695638260308313 | string | 19 digits |
email | An email address | [email protected] | string | |
idCard | A domestic ID card number | 350600198508222018 | string | 18 digits with a valid checksum and area code |
job | A job title | 系统工程师 | string | |
lat | A latitude | 48.6648764 | double | 7 decimal places, also known as latitude |
lng | A longitude | 120.6018163 | double | 7 decimal places, also known as longitude |
name | A domestic name | 池浩 | string | |
phone | A domestic mobile phone number | 15292600492 | string | |
stockAccount | A 10 digits stock trading account | 0692522928 | string | |
stockCode | A 6 digits stock symbol | 687461 | string | |
uuid | A random UUID | bc1cf125-929b-43b7-b324-d7c4cc5a75d2 | string | |
zipCode | A 6 digits postal code | 411105 | long |
The name of a rule ignores case, underscores and dashes, so idCard, id_card and ID_CARD are the same rule.
Two notes about the rules:
- the type of a rule is fixed and can not be changed with the
typeitem; atypethat does not match the type of the rule fails the job at startup - a rule builds its value itself, so a configured
valueis ignored with a warning
The rule item
Besides the built-in rules above, rule also accepts the generic rules constant, random and incr. Their parameter is written in the value item and means exactly the same as the items described earlier in this page:
{ "rule": "constant", "value": "addax", "type": "string" }
{ "rule": "random", "value": "1,10", "type": "long" }
{ "rule": "incr", "value": "1,5", "type": "long" }
{ "rule": "incr", "value": "1989-06-04 09:01:02,2,d", "type": "date" }In other words, {"random": "1,10"} is the same as {"rule": "random", "value": "1,10"} and {"incr": "1,5"} is the same as {"rule": "incr", "value": "1,5"}. Both forms are supported; when a column configures rule and random/incr at the same time, the rule wins and the ignored item is reported with a warning.
Migrating from datareader
The datareader plugin has been merged into StreamReader: its built-in rules (ID card, bank card, address, ...) are provided by StreamReader now, and the datareader plugin is no longer released separately.
To migrate a job, change the name of the plugin - the columns need no change:
{
"reader": {
"name": "streamreader",
"parameter": {
"column": [
{ "rule": "idCard" }
],
"sliceRecordCount": 10
}
}
}A date format that uses the 12 hour field hh has to be changed to HH as well.