21.5.14 ndb_import — Import CSV Data Into NDB (original) (raw)

21.5.14 ndb_import — Import CSV Data Into NDB

ndb_import imports CSV-formatted data, such as that produced by mysqldump --tab, directly intoNDB using the NDB API.ndb_import requires a connection to an NDB management server (ndb_mgmd) to function; it does not require a connection to a MySQL Server.

Usage

ndb_import db_name file_name options

ndb_import requires two arguments.dbname is the name of the database where the table into which to import the data is found;filename is the name of the CSV file from which to read the data; this must include the path to this file if it is not in the current directory. The name of the file must match that of the table; the file's extension, if any, is not taken into consideration. Options supported byndb_import include those for specifying field separators, escapes, and line terminators, and are described later in this section.

ndb_import rejects any empty lines read from the CSV file.

ndb_import must be able to connect to an NDB Cluster management server; for this reason, there must be an unused [api] slot in the clusterconfig.ini file.

To duplicate an existing table that uses a different storage engine, such as InnoDB, as anNDB table, use the mysql client to perform aSELECT INTO OUTFILE statement to export the existing table to a CSV file, then to execute aCREATE TABLE LIKE statement to create a new table having the same structure as the existing table, then performALTER TABLE ... ENGINE=NDB on the new table; after this, from the system shell, invoke ndb_import to load the data into the new NDB table. For example, an existing InnoDB table namedmyinnodb_table in a database namedmyinnodb can be exported into anNDB table namedmyndb_table in a database namedmyndb as shown here, assuming that you are already logged in as a MySQL user with the appropriate privileges:

1. In the mysql client:

mysql> USE myinnodb;  
mysql> SELECT * INTO OUTFILE '/tmp/myndb_table.csv'  
     >  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'  
     >  LINES TERMINATED BY '\n'  
     >  FROM myinnodbtable;  
mysql> CREATE DATABASE myndb;  
mysql> USE myndb;  
mysql> CREATE TABLE myndb_table LIKE myinnodb.myinnodb_table;  
mysql> ALTER TABLE myndb_table ENGINE=NDB;  
mysql> EXIT;  
Bye  
$>  

Once the target database and table have been created, a running mysqld is no longer required. You can stop it using mysqladmin shutdown or another method before proceeding, if you wish. 2. In the system shell:

# if you are not already in the MySQL bin directory:  
$> cd path-to-mysql-bin-dir  
$> ndb_import myndb /tmp/myndb_table.csv --fields-optionally-enclosed-by='"' \  
    --fields-terminated-by="," --fields-escaped-by='\\'  

The output should resemble what is shown here:

job-1 import myndb.myndb_table from /tmp/myndb_table.csv  
job-1 [running] import myndb.myndb_table from /tmp/myndb_table.csv  
job-1 [success] import myndb.myndb_table from /tmp/myndb_table.csv  
job-1 imported 19984 rows in 0h0m9s at 2277 rows/s  
jobs summary: defined: 1 run: 1 with success: 1 with failure: 0  
$>  

Options that can be used with ndb_import are shown in the following table. Additional descriptions follow the table.

Table 21.33 Command-line options used with the program ndb_import

• --abort-on-error

  Command-Line Format	--abort-on-error
  Introduced	5.7.18-ndb-7.6.2
  Dump core on any fatal error; used for debugging only.

• --ai-increment=#

  Command-Line Format	--ai-increment=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	1
  Minimum Value	1
  Maximum Value	4294967295
  For a table with a hidden primary key, specify the autoincrement increment, like theauto_increment_increment system variable does in the MySQL Server.

• --ai-offset=#

  Command-Line Format	--ai-offset=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	1
  Minimum Value	1
  Maximum Value	4294967295
  For a table with hidden primary key, specify the autoincrement offset. Similar to theauto_increment_offset system variable.

• --ai-prefetch-sz=#

  Command-Line Format	--ai-prefetch-sz=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	1024
  Minimum Value	1
  Maximum Value	4294967295
  For a table with a hidden primary key, specify the number of autoincrement values that are prefetched. Behaves like thendb_autoincrement_prefetch_sz system variable does in the MySQL Server.

• --character-sets-dir

  Command-Line Format	--character-sets-dir=path
  Introduced	5.7.18-ndb-7.6.2
  Directory containing character sets.

• --connect-retries

  Command-Line Format	--connect-retries=#
  Type	Integer
  Default Value	12
  Minimum Value	0
  Maximum Value	12
  Number of times to retry connection before giving up.

• --connect-retry-delay

  Command-Line Format	--connect-retry-delay=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	5
  Minimum Value	0
  Maximum Value	5
  Number of seconds to wait between attempts to contact management server.

• --connections=#

  Command-Line Format	--connections=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	1
  Minimum Value	1
  Maximum Value	4294967295
  Number of cluster connections to create.

• --connect-string

  Command-Line Format	--connect-string=connection_string
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  Same as--ndb-connectstring.

• --continue

  Command-Line Format	--continue
  Introduced	5.7.18-ndb-7.6.2
  When a job fails, continue to the next job.

• --core-file

  Command-Line Format	--core-file
  Introduced	5.7.18-ndb-7.6.2
  Write core file on error; used in debugging.

• --csvopt=string

  Command-Line Format	--csvopt=opts
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  Provides a shortcut method for setting typical CSV import options. The argument to this option is a string consisting of one or more of the following parameters:

  • c: Fields terminated by comma
  • d: Use defaults, except where overridden by another parameter
  • n: Lines terminated by\n
  • q: Fields optionally enclosed by double quote characters (")
  • r: Line terminated by\r
    The order of the parameters makes no difference, except that if both n and r are specified, the one occurring last is the parameter which takes effect.
    This option is intended for use in testing under conditions in which it is difficult to transmit escapes or quotation marks.
• --db-workers=#

  Command-Line Format	--db-workers=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value (≥ 5.7.20-ndb-7.6.4)	4
  Default Value (≥ 5.7.18-ndb-7.6.2, ≤ 5.7.18-ndb-7.6.3)	1
  Minimum Value	1
  Maximum Value	4294967295
  Number of threads, per data node, executing database operations.

• --defaults-extra-file

  Command-Line Format	--defaults-extra-file=path
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  Read given file after global files are read.

• --defaults-file

  Command-Line Format	--defaults-file=path
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  Read default options from given file only.

• --defaults-group-suffix

  Command-Line Format	--defaults-group-suffix=string
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  Also read groups with concat(group, suffix).

• --errins-type=name

  Command-Line Format	--errins-type=name
  Introduced	5.7.18-ndb-7.6.2
  Type	Enumeration
  Default Value	[none]
  Valid Values	stopjobstopallsighupsigintlist
  Error insert type; use list as the_name_ value to obtain all possible values. This option is used for testing purposes only.

• --errins-delay=#

  Command-Line Format	--errins-delay=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	1000
  Minimum Value	0
  Maximum Value	4294967295
  Unit	ms
  Error insert delay in milliseconds; random variation is added. This option is used for testing purposes only.

• --fields-enclosed-by=char

  Command-Line Format	--fields-enclosed-by=char
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  This works in the same way as the FIELDS ENCLOSED BY option does for the LOAD DATA statement, specifying a character to be interpeted as quoting field values. For CSV input, this is the same as--fields-optionally-enclosed-by.

• --fields-escaped-by=name

  Command-Line Format	--fields-escaped-by=char
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	\
  Specify an escape character in the same way as theFIELDS ESCAPED BY option does for the SQLLOAD DATA statement.

• --fields-optionally-enclosed-by=char

  Command-Line Format	--fields-optionally-enclosed-by=char
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  This works in the same way as the FIELDS OPTIONALLY ENCLOSED BY option does for theLOAD DATA statement, specifying a character to be interpeted as optionally quoting field values. For CSV input, this is the same as--fields-enclosed-by.

• --fields-terminated-by=char

  Command-Line Format	--fields-terminated-by=char
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	\t
  This works in the same way as the FIELDS TERMINATED BY option does for the LOAD DATA statement, specifying a character to be interpeted as the field separator.

• --help

  Command-Line Format	--help
  Introduced	5.7.18-ndb-7.6.2
  Display help text and exit.

• --idlesleep=#

  Command-Line Format	--idlesleep=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	1
  Minimum Value	1
  Maximum Value	4294967295
  Unit	ms
  Number of milliseconds to sleep waiting for more work to perform.

• --idlespin=#

  Command-Line Format	--idlespin=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	0
  Minimum Value	0
  Maximum Value	4294967295
  Number of times to retry before sleeping.

• --ignore-lines=#

  Command-Line Format	--ignore-lines=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	0
  Minimum Value	0
  Maximum Value	4294967295
  Cause ndb_import to ignore the first_#_ lines of the input file. This can be employed to skip a file header that does not contain any data.

• --input-type=name

  Command-Line Format	--input-type=name
  Introduced	5.7.18-ndb-7.6.2
  Type	Enumeration
  Default Value	csv
  Valid Values	randomcsv
  Set the type of input type. The default iscsv; random is intended for testing purposes only. .

• --input-workers=#

  Command-Line Format	--input-workers=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value (≥ 5.7.20-ndb-7.6.4)	4
  Default Value (≥ 5.7.18-ndb-7.6.2, ≤ 5.7.18-ndb-7.6.3)	2
  Minimum Value	1
  Maximum Value	4294967295
  Set the number of threads processing input.

• --keep-state

  Command-Line Format	--keep-state
  Introduced	5.7.20-ndb-7.6.4
  By default, ndb_import removes all state files (except non-empty *.rej files) when it completes a job. Specify this option (nor argument is required) to force the program to retain all state files instead.

• --lines-terminated-by=name

  Command-Line Format	--lines-terminated-by=char
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	\n
  This works in the same way as the LINES TERMINATED BY option does for the LOAD DATA statement, specifying a character to be interpeted as end-of-line.

• --login-path

  Command-Line Format	--login-path=path
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  Read given path from login file.

• --log-level=#

  Command-Line Format	--log-level=#
  Type	Integer
  Default Value	0
  Minimum Value	0
  Maximum Value	2
  Performs internal logging at the given level. This option is intended primarily for internal and development use.
  In debug builds of NDB only, the logging level can be set using this option to a maximum of 4.

• --max-rows=#

  Command-Line Format	--max-rows=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	0
  Minimum Value	0
  Maximum Value	4294967295
  Unit	bytes
  Import only this number of input data rows; the default is 0, which imports all rows.

• --monitor=#

  Command-Line Format	--monitor=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	2
  Minimum Value	0
  Maximum Value	4294967295
  Unit	bytes
  Periodically print the status of a running job if something has changed (status, rejected rows, temporary errors). Set to 0 to disable this reporting. Setting to 1 prints any change that is seen. Higher values reduce the frequency of this status reporting.

• --ndb-connectstring

  Command-Line Format	--ndb-connectstring=connection_string
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  Set connect string for connecting to ndb_mgmd. Syntax: "[nodeid=id;][host=]hostname[:port]". Overrides entries in NDB_CONNECTSTRING and my.cnf.

• --ndb-mgmd-host

  Command-Line Format	--ndb-mgmd-host=connection_string
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	[none]
  Same as--ndb-connectstring.

• --ndb-nodeid

  Command-Line Format	--ndb-nodeid=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	[none]
  Set node ID for this node, overriding any ID set by--ndb-connectstring.

• --ndb-optimized-node-selection

  Command-Line Format	--ndb-optimized-node-selection
  Enable optimizations for selection of nodes for transactions. Enabled by default; use--skip-ndb-optimized-node-selection to disable.

• --no-asynch

  Command-Line Format	--no-asynch
  Introduced	5.7.18-ndb-7.6.2
  Run database operations as batches, in single transactions.

• --no-defaults

  Command-Line Format	--no-defaults
  Introduced	5.7.18-ndb-7.6.2
  Do not read default options from any option file other than login file.

• --no-hint

  Command-Line Format	--no-hint
  Introduced	5.7.18-ndb-7.6.2
  Do not use distribution key hinting to select a data node.

• --opbatch=#

  Command-Line Format	--opbatch=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	256
  Minimum Value	1
  Maximum Value	4294967295
  Unit	bytes
  Set a limit on the number of operations (including blob operations), and thus the number of asynchronous transactions, per execution batch.

• --opbytes=#

  Command-Line Format	--opbytes=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	0
  Minimum Value	0
  Maximum Value	4294967295
  Unit	bytes
  Set a limit on the number of bytes per execution batch. Use 0 for no limit.

• --output-type=name

  Command-Line Format	--output-type=name
  Introduced	5.7.18-ndb-7.6.2
  Type	Enumeration
  Default Value	ndb
  Valid Values	null
  Set the output type. ndb is the default.null is used only for testing.

• --output-workers=#

  Command-Line Format	--output-workers=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	2
  Minimum Value	1
  Maximum Value	4294967295
  Set the number of threads processing output or relaying database operations.

• --pagesize=#

  Command-Line Format	--pagesize=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	4096
  Minimum Value	1
  Maximum Value	4294967295
  Unit	bytes
  Align I/O buffers to the given size.

• --pagecnt=#

  Command-Line Format	--pagecnt=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	64
  Minimum Value	1
  Maximum Value	4294967295
  Set the size of I/O buffers as multiple of page size. The CSV input worker allocates buffer that is doubled in size.

• --polltimeout=#

  Command-Line Format	--polltimeout=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	1000
  Minimum Value	1
  Maximum Value	4294967295
  Unit	ms
  Set a timeout per poll for completed asynchonous transactions; polling continues until all polls are completed, or until an error occurs.

• --print-defaults

  Command-Line Format	--print-defaults
  Introduced	5.7.18-ndb-7.6.2
  Print program argument list and exit.

• --rejects=#

  Command-Line Format	--rejects=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	0
  Minimum Value	0
  Maximum Value	4294967295
  Limit the number of rejected rows (rows with permanent errors) in the data load. The default is 0, which means that any rejected row causes a fatal error. Any rows causing the limit to be exceeded are added to the.rej file.
  The limit imposed by this option is effective for the duration of the current run. A run restarted using--resume is considered a“new” run for this purpose.

• --resume

  Command-Line Format	--resume
  Introduced	5.7.18-ndb-7.6.2
  If a job is aborted (due to a temporary db error or when interrupted by the user), resume with any rows not yet processed.

• --rowbatch=#

  Command-Line Format	--rowbatch=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	0
  Minimum Value	0
  Maximum Value	4294967295
  Unit	rows
  Set a limit on the number of rows per row queue. Use 0 for no limit.

• --rowbytes=#

  Command-Line Format	--rowbytes=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	262144
  Minimum Value	0
  Maximum Value	4294967295
  Unit	bytes
  Set a limit on the number of bytes per row queue. Use 0 for no limit.

• --stats

  Command-Line Format	--stats
  Introduced	5.7.20-ndb-7.6.4
  Save information about options related to performance and other internal statistics in files named*.sto and *.stt. These files are always kept on successful completion (even if --keep-state is not also specified).

• --state-dir=name

  Command-Line Format	--state-dir=path
  Introduced	5.7.18-ndb-7.6.2
  Type	String
  Default Value	.
  Where to write the state files (_`tblname`_.map,_`tblname`_.rej,_`tblname`_.res, and_`tblname`_.stt) produced by a run of the program; the default is the current directory.

• --tempdelay=#

  Command-Line Format	--tempdelay=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	10
  Minimum Value	0
  Maximum Value	4294967295
  Unit	ms
  Number of milliseconds to sleep between temporary errors.

• --temperrors=#

  Command-Line Format	--temperrors=#
  Introduced	5.7.18-ndb-7.6.2
  Type	Integer
  Default Value	0
  Minimum Value	0
  Maximum Value	4294967295
  Number of times a transaction can fail due to a temporary error, per execution batch. The default is 0, which means that any temporary error is fatal. Temporary errors do not cause any rows to be added to the .rej file.

• --usage

  Command-Line Format	--usage
  Introduced	5.7.18-ndb-7.6.2
  Display help text and exit; same as--help.

• --verbose, -v

  Command-Line Format	--verbose[=#]
  Introduced	5.7.18-ndb-7.6.2
  Type (≥ 5.7.20-ndb-7.6.4)	Boolean
  Type (≥ 5.7.18-ndb-7.6.2, ≤ 5.7.18-ndb-7.6.3)	Integer
  Default Value (≥ 5.7.20-ndb-7.6.4)	false
  Default Value (≥ 5.7.18-ndb-7.6.2, ≤ 5.7.18-ndb-7.6.3)	0
  Minimum Value	0
  Maximum Value	2
  Enable verbose output.
  Note
  Previously, this option controlled the internal logging level for debugging messages. In NDB 7.6, use the--log-level option for this purpose instead.

• --version

  Command-Line Format	--version
  Introduced	5.7.18-ndb-7.6.2
  Display version information and exit.

As with LOAD DATA, options for field and line formatting much match those used to create the CSV file, whether this was done usingSELECT INTO ... OUTFILE, or by some other means. There is no equivalent to the LOAD DATA statement STARTING WITH option.

ndb_import was added in NDB 7.6.