Apache Arrow

Language support

pyarrow

import pyarrow as pa

num  = pa.array([1, 2, 3, 2], type=pa.int8()); num

<pyarrow.lib.Int8Array object at 0x2b0bc3e20>
[
  1,
  2,
  3,
  2
]

year = pa.array([2019,2020,2021,2022]); year

<pyarrow.lib.Int64Array object at 0x2b0bc3e80>
[
  2019,
  2020,
  2021,
  2022
]

name = pa.array(
  ["Alice", "Bob", "Carol", "Dave"],
  type=pa.string()
)
name

<pyarrow.lib.StringArray object at 0x2b0bc3f40>
[
  "Alice",
  "Bob",
  "Carol",
  "Dave"
]

Tables

A table is created by combining multiple arrays together to form the columns while also attaching names for each column.

t = pa.table(
  [num, year, name],
  names = ["num", "year", "name"]
)
t

pyarrow.Table
num: int8
year: int64
name: string
----
num: [[1,2,3,2]]
year: [[2019,2020,2021,2022]]
name: [["Alice","Bob","Carol","Dave"]]

Array indexing

Elements of an array can be selected using [] with an integer index or a slice, the former returns a typed scalar the latter an array.

name[0]

<pyarrow.StringScalar: 'Alice'>

name[0:3]

<pyarrow.lib.StringArray object at 0x2b0bc3ee0>
[
  "Alice",
  "Bob",
  "Carol"
]

name[:]

<pyarrow.lib.StringArray object at 0x2b0c6c040>
[
  "Alice",
  "Bob",
  "Carol",
  "Dave"
]

name[-1]

<pyarrow.StringScalar: 'Dave'>

name[::-1]

<pyarrow.lib.StringArray object at 0x2b0c6d1e0>
[
  "Dave",
  "Carol",
  "Bob",
  "Alice"
]

name[4]

Error: IndexError: index out of bounds

name[0] = "Patty"

Error: TypeError: 'pyarrow.lib.StringArray' object does not support item assignment

Data Types

The following types are language agnostic for the purpose of portability, however some differ slightly from what is available from Numpy and Pandas (or R),

• Fixed-length primitive types - numbers, booleans, date and times, fixed size binary, decimals, and other values that fit into a given number

  • Examples: bool_(), uint64(), timestamp(), date64(), and many more

• Variable-length primitive types - binary, string

• Nested types - list, map, struct, and union

• Dictionary type - An encoded categorical type

Schemas

are a data structure that contains information on the names and types of columns for a table (or record batch),

t.schema

num: int8
year: int64
name: string

pa.schema([
  ('num', num.type),
  ('year', year.type),
  ('name', name.type)
])

num: int8
year: int64
name: string

Schema metadata

Schemas can also store additional metadata (e.g. codebook like textual descriptions) in the form of a string:string dictionary,

new_schema = t.schema.with_metadata({
  'num': "Favorite number",
  'year': "Year expected to graduate",
  'name': "First name"
})

new_schema

num: int8
year: int64
name: string
-- schema metadata --
num: 'Favorite number'
year: 'Year expected to graduate'
name: 'First name'

t.schema

num: int8
year: int64
name: string

t.cast(new_schema).schema

num: int8
year: int64
name: string
-- schema metadata --
num: 'Favorite number'
year: 'Year expected to graduate'
name: 'First name'

Missing values / None / NANs

pa.array([1,2,None,3])

<pyarrow.lib.Int64Array object at 0x2b0bc3ee0>
[
  1,
  2,
  null,
  3
]

pa.array([1.,2.,None,3.])

<pyarrow.lib.DoubleArray object at 0x2b0c6c040>
[
  1,
  2,
  null,
  3
]

pa.array([1,2,np.nan,3])

<pyarrow.lib.DoubleArray object at 0x2b0bc3ee0>
[
  1,
  2,
  nan,
  3
]

pa.array(["alice","bob",None,"dave"])

<pyarrow.lib.StringArray object at 0x2b0c6d1e0>
[
  "alice",
  "bob",
  null,
  "dave"
]

pa.array([1,2,None,3])[2]

<pyarrow.Int64Scalar: None>

pa.array([1.,2.,None,3.])[2]

<pyarrow.DoubleScalar: None>

pa.array([1,2,np.nan,3])[2]

<pyarrow.DoubleScalar: nan>

pa.array(["alice","bob",None,"dave"])[2]

<pyarrow.StringScalar: None>

Nest type arrays

pa.array([[1,2], [3,4], None, [5,None]])

<pyarrow.lib.ListArray object at 0x2b0bc3ee0>
[
  [
    1,
    2
  ],
  [
    3,
    4
  ],
  null,
  [
    5,
    null
  ]
]

pa.array([
  {'x': 1, 'y': True, 'z': "Alice"},
  {'x': 2,            'z': "Bob"  },
  {'x': 3, 'y': False             }
])

<pyarrow.lib.StructArray object at 0x2b0c6d240>
-- is_valid: all not null
-- child 0 type: int64
  [
    1,
    2,
    3
  ]
-- child 1 type: bool
  [
    true,
    null,
    false
  ]
-- child 2 type: string
  [
    "Alice",
    "Bob",
    null
  ]

Dictionary array

A dictionary array is the equivalent to a factor in R or pd.Categorical in Pandas,

dict_array = pa.DictionaryArray.from_arrays(
  indices = pa.array([0,0,2,1,3,None]), 
  dictionary = pa.array(['sun', 'rain', 'clouds', 'snow'])
)
dict_array

<pyarrow.lib.DictionaryArray object at 0x2b0b83d80>

-- dictionary:
  [
    "sun",
    "rain",
    "clouds",
    "snow"
  ]
-- indices:
  [
    0,
    0,
    2,
    1,
    3,
    null
  ]

Record Batches

Between a table and an array Arrow has the concept of a Record Batch - which represents a chunk of a larger table. They are composed of a named collection of equal-length arrays.

batch = pa.RecordBatch.from_arrays(
  arrays = [num, year, name],
  names = ["num", "year", "name"]
)
batch

pyarrow.RecordBatch
num: int8
year: int64
name: string

batch.num_columns

3

batch.num_rows

4

batch.nbytes

69

batch.schema

num: int8
year: int64
name: string

Batch indexing

[] can be used with a Record Batch to select columns (by name or index) or rows (by slice), additionally the slice() method can be used to select rows.

batch[0]

<pyarrow.lib.Int8Array object at 0x2b0c6d3c0>
[
  1,
  2,
  3,
  2
]

batch["name"]

<pyarrow.lib.StringArray object at 0x2b0c6d420>
[
  "Alice",
  "Bob",
  "Carol",
  "Dave"
]

batch[1::2].to_pandas()

   num  year  name
0    2  2020   Bob
1    2  2022  Dave

batch.slice(0,2).to_pandas()

   num  year   name
0    1  2019  Alice
1    2  2020    Bob

Tables vs Record Batches

As mentioned previously, table objects are not part of the Arrow specification - rather they are a convenience tool provided to help with the wrangling of multiple Record Batches.

table = pa.Table.from_batches([batch] * 3); table

pyarrow.Table
num: int8
year: int64
name: string
----
num: [[1,2,3,2],[1,2,3,2],[1,2,3,2]]
year: [[2019,2020,2021,2022],[2019,2020,2021,2022],[2019,2020,2021,2022]]
name: [["Alice","Bob","Carol","Dave"],["Alice","Bob","Carol","Dave"],["Alice","Bob","Carol","Dave"]]

table.num_columns

3

table.num_rows

12

table.to_pandas()

    num  year   name
0     1  2019  Alice
1     2  2020    Bob
2     3  2021  Carol
3     2  2022   Dave
4     1  2019  Alice
5     2  2020    Bob
6     3  2021  Carol
7     2  2022   Dave
8     1  2019  Alice
9     2  2020    Bob
10    3  2021  Carol
11    2  2022   Dave

Chunked Array

The columns of table are therefore composed of the columns of each of the batches, these are stored as ChunckedArrays instead of Arrays to reflect this.

table["name"]

<pyarrow.lib.ChunkedArray object at 0x2b0caa2f0>
[
  [
    "Alice",
    "Bob",
    "Carol",
    "Dave"
  ],
  [
    "Alice",
    "Bob",
    "Carol",
    "Dave"
  ],
  [
    "Alice",
    "Bob",
    "Carol",
    "Dave"
  ]
]

table[1]

<pyarrow.lib.ChunkedArray object at 0x2b0caa480>
[
  [
    2019,
    2020,
    2021,
    2022
  ],
  [
    2019,
    2020,
    2021,
    2022
  ],
  [
    2019,
    2020,
    2021,
    2022
  ]
]

Arrow + NumPy

Conversion between NumPy arrays and Arrow arrays is straight forward,

np.linspace(0,1,11)

array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ])

pa.array( np.linspace(0,1,6) )

<pyarrow.lib.DoubleArray object at 0x2b0c6dcc0>
[
  0,
  0.2,
  0.4,
  0.6000000000000001,
  0.8,
  1
]

pa.array(range(10)).to_numpy()

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

NumPy & data copies

pa.array(["hello", "world"]).to_numpy()

Error: pyarrow.lib.ArrowInvalid: Needed to copy 1 chunks with 0 nulls, but zero_copy_only was True

pa.array(["hello", "world"]).to_numpy(zero_copy_only=False)

array(['hello', 'world'], dtype=object)

pa.array([1,2,None,4]).to_numpy()

Error: pyarrow.lib.ArrowInvalid: Needed to copy 1 chunks with 1 nulls, but zero_copy_only was True

pa.array([1,2,None,4]).to_numpy(zero_copy_only=False)

array([ 1.,  2., nan,  4.])

pa.array([[1,2], [3,4], [5,6]]).to_numpy()

Error: pyarrow.lib.ArrowInvalid: Needed to copy 1 chunks with 0 nulls, but zero_copy_only was True

pa.array([[1,2], [3,4], [5,6]]).to_numpy(zero_copy_only=False)

array([array([1, 2]), array([3, 4]), array([5, 6])], dtype=object)

Pandas -> Arrow

We’ve already seen some basic conversion of Arrow table objects to Pandas, the conversions here are a bit more complex than with NumPy due in large part to how Pandas handles missing data.

Arrow -> Pandas

Series & data copies

pa.array([1,2,3,4]).to_pandas()

0    1
1    2
2    3
3    4
dtype: int64

pa.array(["hello", "world"]).to_pandas()

0    hello
1    world
dtype: object

pa.array(["hello", "world"]).dictionary_encode().to_pandas()

0    hello
1    world
dtype: category
Categories (2, object): ['hello', 'world']

pa.array([1,2,3,4]).to_pandas(zero_copy_only=True)

0    1
1    2
2    3
3    4
dtype: int64

pa.array(["hello", "world"]).to_pandas(zero_copy_only=True)

Error: pyarrow.lib.ArrowInvalid: Needed to copy 1 chunks with 0 nulls, but zero_copy_only was True

pa.array(["hello", "world"]).dictionary_encode().to_pandas(zero_copy_only=True)

Error: pyarrow.lib.ArrowInvalid: Needed to copy 1 chunks with 0 nulls, but zero_copy_only was True

Zero Copy Series conversions

Zero copy conversions from Array or ChunkedArray to NumPy arrays or pandas Series are possible in certain narrow cases:

• The Arrow data is stored in an integer (signed or unsigned int8 through int64) or floating point type (float16 through float64). This includes many numeric types as well as timestamps.

• The Arrow data has no null values (since these are represented using bitmaps which are not supported by pandas).

• For ChunkedArray, the data consists of a single chunk, i.e. arr.num_chunks == 1. Multiple chunks will always require a copy because of pandas’s contiguousness requirement.

In these scenarios, to_pandas or to_numpy will be zero copy. In all other scenarios, a copy will be required.

DataFrame & data copies

table.to_pandas()

    num  year   name
0     1  2019  Alice
1     2  2020    Bob
2     3  2021  Carol
3     2  2022   Dave
4     1  2019  Alice
5     2  2020    Bob
6     3  2021  Carol
7     2  2022   Dave
8     1  2019  Alice
9     2  2020    Bob
10    3  2021  Carol
11    2  2022   Dave

table.schema

num: int8
year: int64
name: string

table.to_pandas(zero_copy_only=True)

Error: pyarrow.lib.ArrowInvalid: Cannot do zero copy conversion into multi-column DataFrame block

table.drop(
  ['name']
).to_pandas(zero_copy_only=True)

Error: pyarrow.lib.ArrowInvalid: Cannot do zero copy conversion into multi-column DataFrame block

pa.table(
  [num,year], names=["num","year"]
).to_pandas(zero_copy_only=True)

Error: pyarrow.lib.ArrowInvalid: Cannot do zero copy conversion into multi-column DataFrame block

Pandas DF -> Arrow

To convert from a Pandas DataFrame to an Arrow Table we can use the from_pandas() method (schemas can also be inferred from DataFrames)

df = pd.DataFrame({
  'x': np.round(np.random.normal(size=5),3),
  'y': ["A","A","B","C","C"],
  'z': [1,2,3,4,5]
})

pa.Table.from_pandas(df)

pyarrow.Table
x: double
y: string
z: int64
----
x: [[0.502,-0.433,0.818,-0.408,0.039]]
y: [["A","A","B","C","C"]]
z: [[1,2,3,4,5]]

pa.Schema.from_pandas(df)

x: double
y: string
z: int64
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 563