Modest performance, address #12647 by kshedden · Pull Request #12656 · pandas-dev/pandas (original) (raw)

@jreback I'm having trouble working with object dtype arrays in cython, would appreciate your help.

I have the following code, where source is a memory slice directly copied from the SAS file, which I need to slice into smaller chunks corresponding to individual data values. I am able to do something similar without trouble for float64 types (not shown below), but I can't get it to work for variable-length byte arrays. These byte arrays will eventually become strings but I don't want to do the conversion here because we give the user the option to retain the raw bytes. The error message I am getting is copied below the code.

I have also tried typing string_chunk as object[:, ::1] but that didn't work either.

cdef void process_byte_array_with_data(object parser, int offset, int length,
                                       uint8_t[:, ::1] byte_chunk,
                                       np.ndarray string_chunk):
    # ...
    bvec = bytearray(source[start:start+lngt])
    string_chunk[js, parser._current_row_in_chunk_index] = bvec

BufferError: Object is not writable.
Exception ignored in: 'pandas.io.sas.saslib.process_byte_array_with_data'
Traceback (most recent call last):
  File "stringsource", line 616, in View.MemoryView.memoryview_cwrapper (pandas/io/sas/saslib.c:14511)
  File "stringsource", line 323, in View.MemoryView.memoryview.__cinit__ (pandas/io/sas/saslib.c:10880)