Deepdive into data types

import pandas as pd
import numpy as np

from typing import Any
from strictly_typed_pandas import DataSet

Numeric types

Pandas stores all numeric data using numpy data types. For example, if we make the following DataFrame (where we explicitely define the data types using base python types):

df = pd.DataFrame(
    {
        "a": pd.Series([1, 2, 3], dtype=int),
        "b": pd.Series([1.0, 2.0, 3.0], dtype=float),
        "c": pd.Series([True, False, True], dtype=bool),
    }
)

df.dtypes

a      int64
b    float64
c       bool
dtype: object

Then we see that all columns have a numpy data type.

assert df.dtypes["a"] == np.int_
assert df.dtypes["b"] == np.float_
assert df.dtypes["c"] == np.bool_

Interestingly, numpy data types are by default equal to their base python counterparts.

assert df.dtypes["a"] == int
assert df.dtypes["b"] == float
assert df.dtypes["c"] == bool

Following this mindset, we allow the schemas to be defined using either numpy or base python data types.

class Schema:
    a: int
    b: float
    c: bool


df = DataSet[Schema]()
df.dtypes

a      int64
b    float64
c       bool
dtype: object

class Schema:
    a: np.int64
    b: np.float64
    c: np.bool_


df = DataSet[Schema]()
df.dtypes

a      int64
b    float64
c       bool
dtype: object

You can also define your schema with superclasses (e.g. np.integer) instead of specific classes (e.g. np.int64).

class Schema:
    a: np.integer
    b: np.float_


df = DataSet[Schema](
    {
        "a": pd.Series([1, 2, 3], dtype=np.int64),
        "b": pd.Series([1.0, 2.0, 3.0], dtype=np.float64),
    }
)
df.dtypes

a      int64
b    float64
dtype: object

Datetime and timedelta

These too are defined using numpy.

class Schema:
    a: np.datetime64
    b: np.timedelta64


df = DataSet[Schema]()
df.dtypes

a     datetime64[ns]
b    timedelta64[ns]
dtype: object

Pandas data types

Pandas has a number of its own data types, to allow for things like:

• Timezones

• Categorical values

• Sparse data

class Schema:
    a: pd.DatetimeTZDtype(tz="UTC")  # type: ignore  # noqa: F821
    b: pd.CategoricalDtype
    c: pd.PeriodDtype(freq="D")  # type: ignore  # noqa: F821
    d: pd.SparseDtype(dtype=np.int64)  # type: ignore
    e: pd.IntervalDtype
    f: pd.Int64Dtype
    h: pd.BooleanDtype


df = DataSet[Schema]()
df.dtypes

a       datetime64[ns, UTC]
b                  category
c                 period[D]
d          Sparse[int64, 0]
e    interval[int64, right]
f                     Int64
h                   boolean
dtype: object

Some of these types accept arguments (e.g. pd.DatetimeTZDtype(tz="UTC")). While this works perfectly well during run-time, it does result in linting errors. You can suppress these without any problems by using # type: ignore  # noqa: F821.

Note that the pandas data types are not considered equivalent to their numpy or base python equivalents.

class SchemaA:
    a: pd.Int64Dtype


class SchemaB:
    a: np.int64


try:
    DataSet[SchemaA]().pipe(DataSet[SchemaB])
except TypeError as e:
    print(e)

Column a is of type Int64, but the schema suggests <class 'numpy.int64'>

Strings

String types are complicated business in pandas. From pandas 1.0.0 and higher, we suggest using the string (i.e. pd.StringDtype) data type. When defining a schema, this data type is compatible with both the base python str annotation and the pandas pd.StringDtype annotation.

class Schema:
    a: str
    b: pd.StringDtype


df = DataSet[Schema](
    {
        "a": pd.Series(["a", "b", "c"], dtype="string"),
        "b": pd.Series(["a", "b", "c"], dtype="string"),
    }
)
df.dtypes

a    string[python]
b    string[python]
dtype: object

Unfortunately, pd.StringDtype has only been around briefly: it isn’t available in older versions of python, and as of yet it is still not used by default when creating a DataFrame with strings. Instead, strings are by default stored as the non-descript object type.

df = pd.DataFrame({"a": ["a", "b", "c"]})
df.dtypes

a    object
dtype: object

To be consistent, we have decided to set str == object when checking the schema, atleast until pd.StringDtype will be the default data type for strings in pandas.

class Schema:
    a: str


df = DataSet[Schema]({"a": ["a", "b", "c"]})
df.dtypes

a    object
dtype: object

Note that this is horribly unspecific. For example, the following DataSet contains a column a with data type object, which contains several things that are definitely not strings. However, since we had to agree that object == str, this currently passes without failure.

class Schema:
    a: str


df = DataSet[Schema]({"a": [None, 42, lambda x: x]})
df.dtypes

a    object
dtype: object

We hope that pd.StringDtype will soon be the default string type, so that we can avoid the problem outlined above. Until then, if you want to be sure that your string columns are actually strings, it’s best to use pd.StringDtype for your type annotations.

class Schema:
    a: pd.StringDtype


df = DataSet[Schema]({"a": pd.Series(["a", "b", "c"], dtype="string")})

try:
    DataSet[Schema]({"a": [None, 42, lambda x: x]})
except TypeError as e:
    print(e)

Column a is of type numpy.object, but the schema suggests <class 'pandas.core.arrays.string_.StringDtype'>

The Any type

In some cases it is useful to be able to define that a column can have Any type. This can either be a column of a specific type (e.g. int64) or a mix of data types (i.e. an object)

class Schema:
    a: Any
    b: Any


df = DataSet[Schema](
    {
        "a": [1, 2, 3],
        "b": ["1", 2, None],
    }
)
df.dtypes

a     int64
b    object
dtype: object

Anything missing?

There’s a zoo of data types used in pandas. Is anything missing? Contact me and I’ll look into it!