StataWriter for version 117 fails on None in a string column long enough to be a Stata StrL. · Issue #23633 · pandas-dev/pandas (original) (raw)

The version 114 writer seems to handle columns of strings containing None just fine, but the 117 writer produces the AttributeError below.

Code to reproduce

import pandas as pd

sample_data1 = [ {'str1': 'string' * 30, 'number': 0}, {'str1': None, 'number': 1} ]

sample_data2 = [ {'str1': 'string' * 500, 'number': 0}, {'str1': None, 'number': 1} ]

data1 = pd.DataFrame(sample_data1) data2 = pd.DataFrame(sample_data2)

data1.to_stata('./sample1_114.dta') data1.to_stata('./sample1_117.dta', version=117)

Will produce the ValueError for over 244 characters.

data2.to_stata('./sample1_114.dta')

data2.to_stata('./sample2_117.dta', version=117)

Error:

AttributeError Traceback (most recent call last) in 24 # data2.to_stata('./sample1_114.dta') 25 ---> 26 data2.to_stata('./sample2_117.dta', version=117)

~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in to_stata(self, fname, convert_dates, write_index, encoding, byteorder, time_stamp, data_label, variable_labels, version, convert_strl) 1875 write_index=write_index, 1876 variable_labels=variable_labels, **kwargs) -> 1877 writer.write_file() 1878 1879 def to_feather(self, fname):

~/anaconda3/lib/python3.6/site-packages/pandas/io/stata.py in write_file(self) 2214 self._write_expansion_fields() 2215 self._write_characteristics() -> 2216 self._prepare_data() 2217 self._write_data() 2218 self._write_strls()

~/anaconda3/lib/python3.6/site-packages/pandas/io/stata.py in _prepare_data(self) 2375 self.fmtlist[i]) 2376 # 2. Convert strls -> 2377 data = self._convert_strls(data) 2378 2379 # 3. Convert bad string data to '' and pad to correct length

~/anaconda3/lib/python3.6/site-packages/pandas/io/stata.py in _convert_strls(self, data) 2963 tab, new_data = ssw.generate_table() 2964 data = new_data -> 2965 self._strl_blob = ssw.generate_blob(tab) 2966 return data 2967

~/anaconda3/lib/python3.6/site-packages/pandas/io/stata.py in generate_blob(self, gso_table) 2652 2653 # llll -> 2654 encoded = self._encode(strl) 2655 bio.write(struct.pack(len_type, len(encoded) + 1)) 2656

~/anaconda3/lib/python3.6/site-packages/pandas/io/stata.py in _encode(self, s) 2592 """ 2593 if compat.PY3: -> 2594 return s.encode(self._encoding) 2595 else: 2596 if isinstance(s, text_type):

AttributeError: 'NoneType' object has no attribute 'encode'

Problem description

This seems like a fairly straightforward regression in the Stata StrL part of the 117 writer compared to how the 114 writer does some more checking on regular strings before working with them.

Note: I'm not sure if other datatypes are handled differently and may have a similar issue, but this is the one I encountered and could reproduce.

If you're wondering why my actual data is so ugly that I encountered this, I'd like to blame something else, but I wrote the code that parses the data that ends up there.

Expected Output

I'd expect the 117 writer to clean up these StrL columns like the 114 writer cleans up standard strings (and so does the 117 writer, because it appears to use the same codepath for standard strings and only handles StrLs differently).

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Darwin
OS-release: 18.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 3.10.0
pip: 18.1
setuptools: 40.5.0
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: 0.9.0
xarray: None
IPython: 7.1.1
sphinx: 1.8.1
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 3.0.1
openpyxl: 2.5.9
xlrd: 1.1.0
xlwt: 1.2.0
xlsxwriter: 1.1.2
lxml: 4.2.5
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.13
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: 0.1.6
pandas_gbq: None
pandas_datareader: 0.7.0