Conceptually, Readers convert string 'work units' into records (key, value pairs). Typically the 'work units' are filenames and the records are extracted from the contents of those files. We want a single record produced per step, but a work unit can correspond to many records.
Therefore we introduce some decoupling using a queue. The queue contains the work units and the Reader dequeues from the queue when it is asked to produce a record (via Read()) but it has finished the last work unit.
Args
reader_ref
The operation that implements the reader.
supports_serialize
True if the reader implementation can serialize its state.
Raises
RuntimeError
If eager execution is enabled.
Attributes
reader_ref
Op that implements the reader.
supports_serialize
Whether the Reader implementation can serialize its state.
Returns the next record (key, value) pair produced by a reader.
Will dequeue a work unit from queue if necessary (e.g. when the Reader needs to start reading from a new file since it has finished with the previous file).
Args
queue
A Queue or a mutable string Tensor representing a handle to a Queue, with string work items.
Returns up to num_records (key, value) pairs produced by a reader.
Will dequeue a work unit from queue if necessary (e.g., when the Reader needs to start reading from a new file since it has finished with the previous file). It may return less than num_records even before the last batch.
Args
queue
A Queue or a mutable string Tensor representing a handle to a Queue, with string work items.