SAS7BDAT parser: Fast byteswap by jonashaag · Pull Request #47403 · pandas-dev/pandas (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation27 Commits27 Checks0 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

jonashaag

Speed up SAS7BDAT int/float reading.

This is order of magnitude faster than using struct.unpack(fmt, data) or precompiled_unpacker = struct.Struct(fmt).unpack; ...; precompiled_unpacker(data).

Unfortunately Python does not expose a low-level interface to struct or a byteswapping interface. The byteswap implementation in this change is from pyreadstat.

Today this brings a modest 10-20% performance improvement. But together with the other changes I will be proposing it will be a major bottleneck.

@jonashaag jonashaag changed the titleFast byteswap SAS7BDAT parser: Fast byteswap

Jun 17, 2022

jbrockmendel

uint8_t,
uint16_t,
uint32_t,
uint64_t,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious, are these interchangeable with the versions of these we cimport from numpy?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't make a huge difference. on the one hand itd be nice to avoid dependency on numpy when possible, on the other im inevitably going to forget and ask again in 6 months if we dont use the numpy versions

WillAyd
cdef inline float _byteswap_float(float num):
cdef uint32_t answer = 0
memcpy(&answer, &num, 4)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use sizeof instead of hard coding the size?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a verbatim copy from ReadStat, do you still want me to make that modification?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok. I think fine to keep as is then

@jonashaag

@jonashaag

@jonashaag

One thing I realized is that we could also use NumPy's byteswapping, at the cost of around 10% performance relative to this impl.

@jbrockmendel

One thing I realized is that we could also use NumPy's byteswapping, at the cost of around 10% performance relative to this impl.

How much of this could be replaced with that?

@jonashaag

How much of this could be replaced with that?

Everything copied from readstat so ~ 50 lines of Cython code

jreback

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

asv's that cover this case?

can you add a whatsnew note

@@ -433,3 +439,73 @@ cdef class Parser:
self.current_row_on_page_index += 1
self.current_row_in_chunk_index += 1
self.current_row_in_file_index += 1
def read_float_with_byteswap(const uint8_t *data, bint byteswap):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add some comments here on what and why you are doing this.

@jonashaag

@jonashaag

@jonashaag

ASV from #47405:

       before           after         ratio
     [e915b0a4]       [435a003c]
     <main>           <sas/byteswap~1>
-      81.9±0.7ms       73.7±0.3ms     0.90  io.sas.SAS.time_read_sas7bdat_2_chunked
-      78.7±0.7ms       69.5±0.6ms     0.88  io.sas.SAS.time_read_sas7bdat_2

@jonashaag

@jonashaag

Test failure seems unrelated

@jonashaag

@jonashaag

@jonashaag

I'm trying an intrinsics-based version: Fewer SLOC and faster.

@datapythonista

Do you want me to add unit tests?

Personally I think it should be useful. Probably just one parametrized to test the function for every type is enough.

@datapythonista

The release note in the other PR is not accurate now. The changes in this PR won't be available until pandas 1.6/2.0, while for what you mention the release note in the other PR mentions this issue/PR for 1.5. Probably no big deal, but if you want to open a small PR to update that note in the releases to only include the issues/PRs that have already been merged, that would leave things more accurate (ping me in that PR, so I backport it to 1.5). And then you can add another release note for 1.6 here.

@jonashaag

Or we merge the PRs noted in those release notes to 1.5? The PRs have been ready for review with no code changes for a long time.

@jonashaag

@datapythonista

We already released pandas 1.5 release candidate, and we are only backporting regressions to it, not new features, performance improvements. It's also unclear to me this will be merged before the release.

I know this has been forgotten for a long time, sorry about that. But that's unfortunately part of how open source development works in a project like pandas.

@jonashaag

@jonashaag

I added tests using Hypothesis and moved the byteswapping code to a module. The byteswapping code also be moved somewhere else outside the SAS stuff.

@jonashaag

Updated release notes. I took the liberty of including #47656 already.

@jonashaag

mroeschke

mroeschke

@mroeschke

noatamir pushed a commit to noatamir/pandas that referenced this pull request

Nov 9, 2022

@jonashaag @noatamir