Saving raw Numpy arrays in a database via SQLAlchemy
Posted on fre 04 oktober 2019 in Programming
I am currently implementing a database where I need to store large chunks of raw data in spreadsheet format. After some initial - slow - attempts with Pandas' to_sql
method I looked around for a way to store the raw array in a Blob. I found out about the TypeDecorator class and inspired by the example at https://docs.sqlalchemy.org/en/13/core/custom_types.html#marshal-json-strings I created the following helper class.
import pickle
from sqlalchemy import Binary, TypeDecorator
class DBNumpyArray(TypeDecorator):
"""Represents a numpy array as a db BLOB
Usage::
DBNumpyArray(arr)
"""
impl = Binary
def process_bind_param(self, value, dialect):
if value is not None:
value = value.dumps()
return value
def process_result_value(self, value, dialect):
if value is not None:
value = pickle.loads(value)
return value
A simple snippet that made wonders to my application!
To further improve on the applicability, portability and speed, one could imagine to use BytesIO
and numpy.save instead of dumps
and loads
, but for me this was good enough!