Machine Learning with Friends, Part 2

Consistent operating systems and flexible compute

Photo by Louise Viallesoubranne on Unsplash

Again, skip to the end for the TLDR.

And again, not selling anything, especially cloud services. 

Motivation

In part 1 of this series, I laid out how to set up your team. Most of my recommendations there generally apply to any group of people collaborating semi-virtually so in part 2, we get into the systems you need specifically for machine learning. Specifically, how you set up computational systems so that your team can effectively collaborate on machine learning side hustles or hobby projects. This is less cut and dry than how to set up a team, but my goal is to give some concrete recommendations with a few guiding principles to help you make key decisions. Full disclosure, I have some biases here and my opinions have been largely influenced by my personal experience with various tools/systems. With that said, let’s get into it. 

Operating Systems

Some of you are old enough to remember the “Get a Mac'' commercials that ran from around 2006 to 2009. John Hodgman played “PC” in a drab suit and Justin Long was the hip “Mac” in casual attire. While I loathe the idea of picking a side here, just get a Mac if you haven’t already. I know this is prohibitively expensive for many people but if you can afford it, it will make your life easier in this world. There isn’t anything inherently wrong with Windows for regular work/life stuff, but the guts are different. Mac OSX runs on unix, which is all but indistinguishable from linux for most of what we do as machine learning practitioners. Another option would be to do everything in the cloud but that’s unnecessarily expensive so I recommend having a local setup at least for small-scale experimentation and testing. The most effective setup for most people is a combination of their local environment and virtual machines in the cloud. You’ll ultimately want multiple people contributing to the same body of work across various machines, which requires moving smoothly between environments and linux/unix is the operating system for that.

Linux Options

Most of us don’t run on linux to do things like email, virtual meetings, etc. because it doesn’t have the support of Apple or Microsoft behind it to make those things work seamlessly, especially with mobile. So unless you want to make your digital life arbitrarily difficult for the hell of it, just stick to whatever OS you’re comfortable with and set up a linux-like environment inside of it. If you’re running on Mac OSX, you already have unix so that’s it. If you use Windows, install WSL (Windows subsystem for linux) and do all of your machine learning development work there. Microsoft has good documentation on it, and here’s a nice guide to installing WSL2 on Windows 10.

Last thing, if you read the last few paragraphs and thought to yourself “this Apple fanboy doesn’t know what he’s talking about” then please pause before you go down a path that will discourage you from doing what you want, which is machine learning with friends. This is more like getting the right adapter so you can charge your phone in Europe than picking a sports team to align with until you die. I mentioned the “Buy a Mac” commercials for a reason: we’ve been conditioned to affiliate with products like religions and it’s not helpful when you’re a practitioner trying to find the best tools. Get over it, this isn’t a hill worth dying on. 

Flexible Compute

I have a somewhat strained relationship with cloud services as I oscillate between loving them and hating them. I love that the cloud has made all kinds of things possible. For example, I can buy time on machines that cost hundreds of thousands of dollars for about $30/hour. On the flip side, I hate the cloud when I waste hundreds of dollars because I unknowingly left a machine running somewhere and forgot to set a spending alert. Cloud services are like power tools: they can be incredibly useful and save you loads of time, but if you aren’t careful you could lose a limb. For machine learning with friends, we want to safely take advantage of the flexibility offered by cloud computing.

Full disclosure, I primarily use AWS because it’s what most of my employers have used so it’s where I have the most familiarity. I’ve used GCP for a few small projects, and I haven’t touched Azure or IBM Cloud. My goal is not to sell you on a particular cloud provider, but rather to give some guidance on how to use cloud services effectively for your machine learning projects. 

Why the Cloud?

The primary use cases for cloud services in your machine learning projects are data storage, which I’ll get into in the next article, and on-demand computing. There are many interesting and challenging problems you can tackle with just your laptop and a little patience, but when you hit the limits of your local setup, it’s extremely nice to have on-demand options. For example, you might be training a gradient boosted tree classifier on a few GB of data when you run into memory bottlenecks. It worked fine with 2GB of training data but at 5GB you get errors because your laptop only has 8GB of memory. Wouldn’t it be nice to spend a few bucks to spin up a virtual machine with 64GB of memory, train it there, and be done with it? Of course, and more importantly, you don’t want to have to learn how to set up and use one of these platforms when you’re in the middle of a project. 

Getting Started

If no one on your team knows anything about working in the cloud, I strongly suggest working through a fundamentals course on AWS or GCP as a group. Personally, I worked through this AWS course awhile back on Udemy and really enjoyed it. You could also try Coursera or any of the major online education platforms. Either AWS or GCP are great to learn from first principles, especially if you’re looking to level up in your career. You don’t need to go through the entire course but commit about 10-20 hours to it and use it to lay a foundation. If you or someone on your team has extensive experience with a cloud platform, have them take the lead then get the others up to speed. If you’ve only used cloud services but never setup or managed accounts, I still strongly recommend buying a fundamentals course to help with setup, permissions, alerts, etc. The last thing you want is someone accidentally spending more than you can afford. 

Budgets, Permissions and VMs

When you set up your account, someone has to put down a credit card so that should be the super admin. The super admin should first set a budget or spending limit. This acts as a fail safe if someone is overspending. You can configure it to alert you as you spend through different proportions of the budget. AWS has pretty thorough documentation on creating budgets. GCP has their budgets documentation too.

Next, you need to add users with defined permissions. I don’t recommend just making everyone an admin unless they all know what they’re doing and you trust them to be prudent with usage. For most machine learning projects in AWS, you’ll do fine starting with S3 for data storage and using SageMaker for virtual machines with jupyter notebooks. I recommend granting access to just the services people need. However you decide to do it though, the point is to get your team set up so they have access to on-demand compute.

Sagemaker is a managed service designed for data science and machine learning that is a bit more expensive than non-managed services like EC2, but they do make it fairly easy to get started. In GCP Google Storage is equivalent to S3 and Vertex AI is like SageMaker. Both Amazon and Google have decent guides for getting started with SageMaker and Vertex AI respectively. I recommend first getting comfortable with using notebooks to read/write data and build basic models in whichever platform you choose. This is a solid foundation and won’t cost you much if you use the small instances and shut them down when you aren’t using them. 

Guiding Principles

Machine learning is a team sport and to do it effectively, you need both consistency and flexibility. This will be a recurring theme more generally but especially with respect to computing, you need consistent operating systems so that different people can contribute to a broader effort on different devices. If I write a makefile to automate some configuration setup and preprocessing but I do it in OSX and you’re using Windows, then one of us is going to have to write a new version to work in Windows. This is a terrible waste of time. Additionally, you need the flexibility to scale hardware up or down depending on the task. For example, when my 16GB laptop runs into memory constraints for a training job, I could spend $5k on a better one or I could spend $10 for just the compute time I need in AWS. Ultimately, everyone needs to be working with the same basic stuff (the OS) with the ability to put more or less power behind it as needed. Some of these things will require some learning, but you can do it. 

TLDR

Too long; didn’t read? Same. Here you go:

1. Get everyone on unix/linux. If they use a Mac, all set. If they use Windows, install WSL2 and run linux on their PC. 

2. Pick a cloud platform for on-demand compute and commit to an educational resource if you aren’t experienced with setting up accounts and managing users. If this is all new to you, choose AWS or GCP and buy a course from Udemy or Coursera to learn how to set things up properly. The learning curve is steep if you’re brand new to this stuff and mistakes can be expensive. It’s worth it though. 

3. Set up your account with a budget and thoughtful permissions for your team. Don’t just make everyone an admin without a good reason. 

4. Pick an option for on-demand virtual machines. If you’re new to this and using AWS, SageMaker Notebook instances are easy to get started with. In GCP, try the Vertex AI Workbench. If someone on your team is more experienced and you don’t want to pay the premium on these services, I’ll encourage you to set up your own instances but make sure that it’s as easy for the rest of your team to use them as it is for you. 

5. Use cloud services as needed, don’t just do everything in the cloud by default. 

It’s hard to say what to expect in terms of cost because your needs will vary depending on what you’re doing and how many people are doing it. With that said, if you use cloud services thoughtfully and only spend as needed, I wouldn’t be surprised if most of your monthly bills are in the $10-$50 range. When it’s just me using notebook instances a few times a week to run memory-intensive jobs for a few hours at a time, I don’t spend more than $10/month. 

Next week, we’ll talk about data storage, organization and access.