bpo-24937: Replace the extension module porting HOWTO by links to ext… · python/cpython@2d3ff2b (original) (raw)

`@@ -6,252 +6,21 @@

6

`Porting Extension Modules to Python 3

7

`*************************************

8

9

:author: Benjamin Peterson

10

-

11

-

12

.. topic:: Abstract

13

-

14

Although changing the C-API was not one of Python 3's objectives,

15

the many Python-level changes made leaving Python 2's API intact

16

`` -

impossible. In fact, some changes such as :func:int and

17

`` -

:func:long unification are more obvious on the C level. This

18

document endeavors to document incompatibilities and how they can

19

be worked around.

20

-

21

-

22

Conditional compilation

23

=======================

24

-

25

The easiest way to compile only some code for Python 3 is to check

26

`` -

if :c:macro:PY_MAJOR_VERSION is greater than or equal to 3. ::

27

-

28

#if PY_MAJOR_VERSION >= 3

29

#define IS_PY3K

30

#endif

31

-

32

API functions that are not present can be aliased to their equivalents within

33

conditional blocks.

34

-

35

-

36

Changes to Object APIs

37

======================

38

-

39

Python 3 merged together some types with similar functions while cleanly

40

separating others.

41

-

42

-

43

str/unicode Unification

44

45

-

46

`` -

Python 3's :func:str type is equivalent to Python 2's :func:unicode; the C

47


functions are called ``PyUnicode_*`` for both. The old 8-bit string type has become

48


:func:`bytes`, with C functions called ``PyBytes_*``. Python 2.6 and later provide a compatibility header,

49


:file:`bytesobject.h`, mapping ``PyBytes`` names to ``PyString`` ones. For best

50

`` -

compatibility with Python 3, :c:type:PyUnicode should be used for textual data and

51

`` -

:c:type:PyBytes for binary data. It's also important to remember that

52

`` -

:c:type:PyBytes and :c:type:PyUnicode in Python 3 are not interchangeable like

53

`` -

:c:type:PyString and :c:type:PyUnicode are in Python 2. The following example

54

`` -

shows best practices with regards to :c:type:PyUnicode, :c:type:PyString,

55

`` -

and :c:type:PyBytes. ::

56

-

57

#include "stdlib.h"

58

#include "Python.h"

59

#include "bytesobject.h"

60

-

61

/* text example */

62

static PyObject *

63

say_hello(PyObject *self, PyObject *args) {

64

PyObject *name, *result;

65

-

66

if (!PyArg_ParseTuple(args, "U:say_hello", &name))

67

return NULL;

68

-

69

result = PyUnicode_FromFormat("Hello, %S!", name);

70

return result;

71

}

72

-

73

/* just a forward */

74

static char * do_encode(PyObject *);

75

-

76

/* bytes example */

77

static PyObject *

78

encode_object(PyObject *self, PyObject *args) {

79

char *encoded;

80

PyObject *result, *myobj;

81

-

82

if (!PyArg_ParseTuple(args, "O:encode_object", &myobj))

83

return NULL;

84

-

85

encoded = do_encode(myobj);

86

if (encoded == NULL)

87

return NULL;

88

result = PyBytes_FromString(encoded);

89

free(encoded);

90

return result;

91

}

92

-

93

-

94

long/int Unification

95

96

-

97

`` -

Python 3 has only one integer type, :func:int. But it actually

98

`` -

corresponds to Python 2's :func:long type—the :func:int type

99


used in Python 2 was removed. In the C-API, ``PyInt_*`` functions

100


are replaced by their ``PyLong_*`` equivalents.

101

-

102

-

103

Module initialization and state

104

===============================

105

-

106

Python 3 has a revamped extension module initialization system. (See

107

`` -

:pep:3121.) Instead of storing module state in globals, they should

108

be stored in an interpreter specific structure. Creating modules that

109

act correctly in both Python 2 and Python 3 is tricky. The following

110

simple example demonstrates how. ::

111

-

112

#include "Python.h"

113

-

114

struct module_state {

115

PyObject *error;

116

};

117

-

118

#if PY_MAJOR_VERSION >= 3

119

#define GETSTATE(m) ((struct module_state*)PyModule_GetState(m))

120

#else

121

#define GETSTATE(m) (&_state)

122

static struct module_state _state;

123

#endif

124

-

125

static PyObject *

126

error_out(PyObject *m) {

127

struct module_state *st = GETSTATE(m);

128

PyErr_SetString(st->error, "something bad happened");

129

return NULL;

130

}

131

-

132

static PyMethodDef myextension_methods[] = {

133

{"error_out", (PyCFunction)error_out, METH_NOARGS, NULL},

134

{NULL, NULL}

135

};

136

-

137

#if PY_MAJOR_VERSION >= 3

138

-

139

static int myextension_traverse(PyObject *m, visitproc visit, void *arg) {

140

Py_VISIT(GETSTATE(m)->error);

141

return 0;

142

}

143

-

144

static int myextension_clear(PyObject *m) {

145

Py_CLEAR(GETSTATE(m)->error);

146

return 0;

147

}

148

-

149

-

150

static struct PyModuleDef moduledef = {

151

PyModuleDef_HEAD_INIT,

152

"myextension",

153

NULL,

154

sizeof(struct module_state),

155

myextension_methods,

156

NULL,

157

myextension_traverse,

158

myextension_clear,

159

NULL

160

};

161

-

162

#define INITERROR return NULL

163

-

164

PyMODINIT_FUNC

165

PyInit_myextension(void)

166

-

167

#else

168

#define INITERROR return

169

-

170

void

171

initmyextension(void)

172

#endif

173

{

174

#if PY_MAJOR_VERSION >= 3

175

PyObject *module = PyModule_Create(&moduledef);

176

#else

177

PyObject *module = Py_InitModule("myextension", myextension_methods);

178

#endif

179

-

180

if (module == NULL)

181

INITERROR;

182

struct module_state *st = GETSTATE(module);

183

-

184

st->error = PyErr_NewException("myextension.Error", NULL, NULL);

185

if (st->error == NULL) {

186

Py_DECREF(module);

187

INITERROR;

188

}

189

-

190

#if PY_MAJOR_VERSION >= 3

191

return module;

192

#endif

193

}

194

-

195

-

196

CObject replaced with Capsule

197

=============================

198

-

199

`` -

The :c:type:Capsule object was introduced in Python 3.1 and 2.7 to replace

200

`` -

:c:type:CObject. CObjects were useful,

201

`` -

but the :c:type:CObject API was problematic: it didn't permit distinguishing

202

between valid CObjects, which allowed mismatched CObjects to crash the

203

interpreter, and some of its APIs relied on undefined behavior in C.

204

`` -

(For further reading on the rationale behind Capsules, please see :issue:5630.)

205

-

206

If you're currently using CObjects, and you want to migrate to 3.1 or newer,

207

you'll need to switch to Capsules.

208

`` -

:c:type:CObject was deprecated in 3.1 and 2.7 and completely removed in

209

Python 3.2. If you only support 2.7, or 3.1 and above, you

210

`` -

can simply switch to :c:type:Capsule. If you need to support Python 3.0,

211

or versions of Python earlier than 2.7,

212

you'll have to support both CObjects and Capsules.

213

(Note that Python 3.0 is no longer supported, and it is not recommended

214

for production use.)

215

-

216

`` -

The following example header file :file:capsulethunk.h may

217

solve the problem for you. Simply write your code against the

218

`` -

:c:type:Capsule API and include this header file after

219

`` -

:file:Python.h. Your code will automatically use Capsules

220

in versions of Python with Capsules, and switch to CObjects

221

when Capsules are unavailable.

222

-

223

`` -

:file:capsulethunk.h simulates Capsules using CObjects. However,

224

`` -

:c:type:CObject provides no place to store the capsule's "name". As a

225

`` -

result the simulated :c:type:Capsule objects created by :file:capsulethunk.h

226

behave slightly differently from real Capsules. Specifically:

227

-

228

`` -

The name parameter passed in to :c:func:PyCapsule_New is ignored.

229

-

230

`` -

The name parameter passed in to :c:func:PyCapsule_IsValid and

231

`` -

:c:func:PyCapsule_GetPointer is ignored, and no error checking

232

of the name is performed.

233

-

234

`` -

:c:func:PyCapsule_GetName always returns NULL.

235

-

236

`` -

:c:func:PyCapsule_SetName always raises an exception and

237

returns failure. (Since there's no way to store a name

238

`` -

in a CObject, noisy failure of :c:func:PyCapsule_SetName

239

was deemed preferable to silent failure here. If this is

240

inconvenient, feel free to modify your local

241

copy as you see fit.)

242

-

243

`` -

You can find :file:capsulethunk.h in the Python source distribution

244

`` -

as :source:Doc/includes/capsulethunk.h. We also include it here for

245

your convenience:

246

-

247

.. literalinclude:: ../includes/capsulethunk.h

248

-

249

-

250

-

251

Other options

252

=============

253

-

254

`` -

If you are writing a new extension module, you might consider `Cython

255

`` -

http://cython.org/`_. It translates a Python-like language to C. The

256

extension modules it creates are compatible with Python 3 and Python 2.

257

-

9

We recommend the following resources for porting extension modules to Python 3:

10

+

11

`` +

The Migrating C extensions_ chapter from

12

Supporting Python 3: An in-depth guide, a book on moving from Python 2

13

to Python 3 in general, guides the reader through porting an extension

14

module.

15

`` +

The Porting guide_ from the py3c project provides opinionated

16

suggestions with supporting code.

17

`` +

The Cython_ and CFFI_ libraries offer abstractions over

18

Python's C API.

19

Extensions generally need to be re-written to use one of them,

20

but the library then handles differences between various Python

21

versions and implementations.

22

+

23

.. _Migrating C extensions: http://python3porting.com/cextensions.html

24

.. _Porting guide: https://py3c.readthedocs.io/en/latest/guide.html

25

.. _Cython: http://cython.org/

26

.. _CFFI: https://cffi.readthedocs.io/en/latest/