Spectrify¶
A simple yet powerful tool to move your data from Redshift to Redshift Spectrum.
- Free software: MIT license
- Documentation: https://spectrify.readthedocs.io.
Features¶
One-liners to:
- Export a Redshift table to S3 (CSV)
- Convert exported CSVs to Parquet files in parallel
- Create the Spectrum table on your Redshift cluster
- Perform all 3 steps in sequence, essentially “copying” a Redshift table Spectrum in one command.
S3 credentials are specified using boto3. See http://boto3.readthedocs.io/en/latest/guide/configuration.html
Redshift credentials are supplied via environment variables, command-line parameters, or interactive prompt.
Install¶
$ pip install spectrify
Command-line Usage¶
Export Redshift table my_table to a folder of CSV files on S3:
Convert exported CSVs to Parquet:
Create Spectrum table from S3 folder:
Transform Redshift table by performing all 3 steps in sequence:
Python Usage¶
Export to S3:
from spectrify.export import RedshiftDataExporter
RedshiftDataExporter(sa_engine, s3_config).export_to_csv('my_table')
Convert exported CSVs to Parquet:
from spectrify.convert import ConcurrentManifestConverter
from spectrify.utils.schema import SqlAlchemySchemaReader
sa_table = SqlAlchemySchemaReader(engine).get_table_schema('my_table')
ConcurrentManifestConverter(sa_table, s3_config).convert_manifest()
Create Spectrum table from S3 parquet folder:
from spectrify.create import SpectrumTableCreator
from spectrify.utils.schema import SqlAlchemySchemaReader
sa_table = SqlAlchemySchemaReader(engine).get_table_schema('my_table')
SpectrumTableCreator(sa_engine, dest_schema, dest_table_name, sa_table, s3_config).create()
Transform Redshift table by performing all 3 steps in sequence:
from spectrify.transform import TableTransformer
transformer = TableTransformer(engine, 'my_table', s3_config, dest_schema, dest_table_name)
transformer.transform()
Contribute¶
Contributions always welcome! Read our guide on contributing here: http://spectrify.readthedocs.io/en/latest/contributing.html
License¶
MIT License. Copyright (c) 2017, The Narrativ Company, Inc.