Navigate back to the homepage

My Semester in Review

Simar Kareer
April 30th, 2019 · 2 min read

Along with Vidhur, I explored the effects of boosting on adversarial robustness. This is a summary of that journey!

Intro

The inspiration for this project comes from Jake’s paper on boosting and adversarial robustness. That paper compares the adversarial robustness of smaller boosted neural nets in comparison with one larger network. In order to investigate this further, we sought to show a similar but more elementary result using decision trees/stumps.

Our Progression

Implementing Existing Algorithms

First and foremost, we implemented the standard adaboost framework using SKLearn decision trees. From here we realized that standard implementations of decision trees wouldn’t be sufficient for out purposes, specifically because we needed more flexibility with regard to sample weighting. The heart of Jake’s boosting framework is a weight matrix representing not only the importance of each $(x, y)$ pair, but also the cost of incorrectly classifying $ h(x) = y’$ for every $ y’ $. SKLearn does not give this level of verbosity with its weighting (SKLearn only let’s you weight specific samples, not individual classes). To counter this, Jake proposed a framework where we would still be able to use SKLearn Decision Trees. For every $ x, y $ pair, we would create other pseudo-examples $(x, y’)$ for all incorrect labels $y’$. This seems to work for the non-adversarial setting, but is not clearly extensible beyond that.

Custom Decision Tree

After that roadbloack, we decided to build our own decision tree which has the freedom to weight all $(x, y’)$ pairs. After much trial and error, we were able to use the initial adaboost framework, but with our custom trees and achieve the same error as the boosted SKLearn trees! At this point we were satisfied that our tree was implemented (mostly) correctly, so we were ready to tackle the adversarial boosting problem. With our custom tree we could now implement Jake’s framework directly. We tried this for a few weeks but the accuracy always seemed to have an unreasonably low upper bound. I decided I would try to use a framework with adaptive weights so I wouldn’t have to worry about tuning an $ \eta $ value. This led to a new framework which I hacked together. It worked very well in the 2 class setting, but seemed to fall apart in multiclass.

Current State / Further Improvements

It became clear we needed a robust implementation of multiclass boosting, so we turned to the literature. Schapire has written extensively on this topic so we decided to use his algorithm “ADABOOST.MM”. Interestingly we even found a typo in it, but once we cleared that up, it was relatively easy to use his boosting framework with our custom decision trees. Now, yet again we had to tackle the adversarial problem, but this time with a more robust framework. We edited Schapire’s boosting algorithm (described below), and we now see very promising results.

More articles from ML Theory

Article 1

With the growing community interest in Gatsby, we hope to create more resources that make it easier for anyone to grasp the power of this incredible tool.

April 30th, 2019 · 1 min read

My Semester in Review

Along with Vidhur, I explored the effects of boosting on adversarial robustness. This is a summary of that journey!

April 30th, 2019 · 2 min read
© 2019 ML Theory
Link to $https://twitter.com/narativeLink to $https://github.com/ugmltheorygtLink to $https://instagram.com/narative.coLink to $https://www.linkedin.com/company/narative/Link to $https://dribbble.com/narativestudio