View on GitHub

PyMADlib

A Python wrapper for MADlib - an open source library for scalable in-database machine learning algorithms

Download this project as a .zip file Download this project as a tar.gz file

Now you can have the number crunching power of MADlib along with the cool visualizations of Matplotlib, NetworkX and all your other Python libraries. Let MADlib do all the heavy-lifting on your Greenplum Appliance, while you program in your favorite language - Python!

Algorithms

PyMADlib currently has wrappers for the following algorithms in MADlib

  1. Linear regression
  2. Logistic Regression
  3. SVM (regression & classification)
  4. K-Means
  5. LDA

ReferMADlib User Docs for MADlib's user documentation.


Dependencies

  1. You'll need the python extension psycopg2 to use PyMADlib.
  2. If you have matplotlib installed, you'll see Matplotlib visualizations for Linear Regression demo.
  3. If you have installed networkx, you'll see a visualization of the k-means demo
  4. PyROC is included in the source of this distribution with permission from its developer. You'll see a visualization of the ROC curves for Logistic Regression.

Configurations

To configure your DB Connection parameters You should create a file in your home directory

~/.pymadlib.config 

that should look like so :

[db_connection]  
user = gpadmin  
password = XXXXX  
hostname = 127.0.0.1 (or the IP of your DB server)  
port = 5432 (the port# of your DB)  
database = vatsandb (the database you wish to connect to)  

Installation Instructions

  1. You may install pymadlib by downloading the source (from PyPI) and then run the following
sudo python setup.py build
sudo python setup.py install
  1. If you use easy_install or pip, simply run :
sudo easy_install pymadlib

Usage Tutorial

Visit PyMADlib Tutorial for a tutorial on using PyMADlib

Also visit PyMADlib IPython NB to download the IPython NB tutorial


Running the Demos

You may run the demo from the extracted directory of pymadlib like so :

python example.py

If you installed PyMADlib using instructions in the previous section, then simply run

python -c 'from pymadlib.example import runDemos; runDemos()'

Remember to close the Matplotlib windows that pop-up to continue with the rest of the demo.


Gallery

K-Means Cluster Visualization

Scatter Plot - Linear Regression (numeric attributes only)

Scatter Plot - Linear Regression (with categorical attributes)

ROC Curve - Logistic Regression

Random graph visualization - Networkx


Datasets packaged with this installation

PyMADlib packages publicly available datasets from the UCI machine learning repository and other sources.

  1. Wine quality dataset from UCI Machine Learning repository
  2. Auto MPG dataset from UCI ML repository from UCI Machine Learning repository
  3. Wine quality dataset from UCI Machine Learning repository
  4. Obama-Romney second presidential debate (2012) transcripts

Installation Issues

Installing PyMADlib using distutils should automatically install the dependent library psycopg2, which is required to connect to a Postgres database where MADlib is installed. If you are using Mac OSX 10.6.x you may run into issues with installing psycopg2.

psycopg2-and-postgresql-9-1-on-snow-leopard and links-about-building-psycopg-mac-os-x discuss some issues and offer some solutions.


Srivatsan Ramanujam vatsan.cs@utexas.edu, 3 Jan 2013