Developing Custom Machine Learning Algorithms in PySpark

This blog post introduces several improvements to PySpark that facilitate the development of custom ML algorithms and 3rd-party ML packages using Python. After introducing the main algorithm APIs in MLlib, we discuss current challenges in building custom ML algorithms on top of PySpark. We then describe our key improvements to PySpark for simplifying such customization.

One critical functionality in MLlib, however, is ML Persistence. Persistence allows users to save models and Pipelines to stable storage, for loading and reusing later or for passing to another team. The API is simple; the following code snippet fits a model using CrossValidator for parameter tuning, saves the fitted model, and loads it back:

Adding support for ML Persistence has traditionally required a Scala implementation. Up until now, the simplest way to implement persistence required the data scientist to implement the algorithm in Scala and write a Python wrapper. Implementing the algorithm in Scala would require knowing both languages, understanding the Java — Python communication interface, and writing duplicate APIs in the two languages.

These mixins dramatically reduce the development effort required to create custom ML algorithms on top of PySpark. Persistence functionality that used to take many lines of extra code can now be done in a single line in many cases.

The code snippets below demonstrate the code length of persisting an algorithm with a Scala implementation and a Python wrapper:

And this code snippet demonstrates using these mixins for a Python-only implementation of persistence:

Adding the mixins DefaultParamsReadable and DefaultParamsWritable to the MyShiftTransformer class allows us to eliminate a lot of code.

Related posts:

I looked at my credit history today and I had to dispute 3 items- for example- credit card totals that Ive already paid off!! 5 months ago. So my question is will they report these disputes to the…

Quem quer te escrever é a liberdade e ninguém pode conquistá-la por você, senão você mesmo. E quando ela te tomar, retome à escrita — divina liberdade de escrever — sem anseios (ou seriam cem…

These jobs often help people through tough times. But this bad rap is necessary sometimes. Especially when you’re freelancing and still feel the need to practice quiet quitting. More AI creators…