Back in January, I learned about the Hugging Face Deep Reinforcement Learning Course from Thomas Simonini and decided it would be fun to participate. Now that I’ve finished the course, I thought it would be a good idea to reflect back on what I learned from the experience and offer some feedback from an educational perspective. For brevity’s sake, here’s a high level summary of what I found to be the pros and cons:

Pros:

• Using Google Colab made it very quick and easy to get started.
• The sequence of problems was very well thought out and flows nicely from one to the next. I felt like each unit does an excellent job of setting up a need that gets addressed by the following unit.
• It finds a nice balance between the mathematical and natural language descriptions of the discussed algorithms.
• I enjoyed the use of video games as a training environment. Watching the animation playback provided helpful feedback on agent progress and kept the lessons engaging.
• I developed an appreciation for how gym environments and wrappers could be used to standardize my code for training experiments.
• I feel I now have a much better understanding of how the Hugging Face Hub might be useful to me as a developer.

Cons:

• The usage limits on Google’s Colab eventually became a hinderance. It might be difficult to pass some of these lessons if that’s your only resource and some of the units suffer from nightmarish dependencies if you try to install locally.
• I really disliked the use of Google Drive links in lessons, particularly when they contained binaries. I’d feel a lot safer about trusting this content if it came from a “huggingface.co” source.
• Some of the later units felt little unpolished in comparison to the early ones. It was a little frustrating to spend increasing time debugging issues that turned out to be mere typos (“SoccerTows”) while also having drastically less scaffolding to work with.
• Accessibility of these resources seemed very limited. Some of the text in images was difficult to read and lacking alt text. Some of the video content would benefit from a larger font and slower narration.
• While training bots for Atari games was certainly fun, the lax attitude towards attribution is concerning from an ethical and legal standpoint.

Overall I had an enjoyable time going through the course. I found myself looking forward to each new unit and cheering on my model as I habitually refreshed the leaderboards. I did not, however, join the Hugging Face community on Discord, so I can’t comment on what the social elements of the course were like. Nothing personal, I just dislike Discord.

It’s probably also important to note that I came into the course with some prior experience with both Python and Deep Learning already. For me, I felt this course made a nice follow-up to Andrew Ng’s Deep Learning Specialization on Coursera in terms of content. While it might be possible to make it through the course without having used PyTorch or TensorFlow before, I feel like you’ll probably want to at least have a conceptual understanding of how neural networks work and some basic familiarity with Python package management before starting the course.

My favorite lesson in the course was the “Pixelcopter” problem in Unit 4. This was the first time in the course where hitting the cut-off score seemed like a real accomplishment. I probably spent more time here than in the rest of the course combined but felt like it was this productive struggle that enabled me to learn some important lessons as a result. Up until this point I felt like I had just been running somebody else’s scripts. Here, it felt like my choice of hyperparameters made a huge difference in the outcome.

Part of the problem with choosing hyperparameters for this Pixelcopter was trying to keep the training time under the time constraints of running in Google Colab. If the training time was too short the bot wouldn’t produce a viable strategy, and if the training time was too long then Colab would timeout. At this point, I went back to the bonus unit on Optuna to manage my hyperparameter optimization. I was able to get a model that produced what I though was a high score, but the variance was so high that I didn’t quite the cut-off score.

Eventually I got so frustrated with the situation that I set up a local Jupyter-Lab server on an old laptop so I could train unattended for longer periods of time. However, this came with its own set of problems because I mistakenly tried to install more recent versions of the modules. Apparently the “gym” module had become “gymnasium” and undergone changes that made it incompatible with the sample code. In an effort to keep things simple, I rolled gym back to an earlier version so I could concentrate on what the existing code was doing.

Once I got my local development environment going, I let Optuna run a bunch of tests with different hyperparameters. This gave me some insight into how sensitive these variables really were. Some bots would never find a strategy that worked. Other bots would find a strategy that seems to work at first, then something would go wrong in a training episode and the performance would start dropping instead. With this in mind, I decided to add an extra layer to my model and started looking more closely at the videos produced by my agents.

What I noticed from the videos was that some bots would attempt to take the “safe” route and some bots would take the “risky” route. The ones that take the safe route tended to do well in early parts of the episode, but started to crash once the blocks speed up enough. The ones that try to take the risky route do way better on later stages, but the early crashes result in unpredictable performance overall.

In an effort to stabilize my agent’s performance, l started playing around with using different environment wrappers. The “Time-Aware” Observation Wrapper seemed to help a little, but I ran into a problems with gym again when I attempted to implement a “Frame Stack”. Apparently the there was a bug in the specific version I had rolled back to, and explicitly pinning my gym version to 0.21 resolved the issue. With a flattened multi-frame time-aware observation the bot was able to come up with a more viable strategy.

What really drove this lesson home was that I felt it set up a real need for the actor-critic methods in Unit 6. I knew precisely what was meant by “significant variance in policy gradient estimation”. I also learned in Unit 6 that the reason the Normalization wrapper I tried before wasn’t working was that I didn’t know I had to load the weights into my evaluation environment. All of these small elements came together at the same time. My extensive trial and error time with Pixelcopter in Unit 4 had show me precisely why that approach would be insufficient for the robotics applications in Unit 6. I felt like understanding this need really solidified the driving ideas behind the actor-critic model.

I also thoroughly enjoyed the “SoccerTwos” problem in Unit 7. However, the part where I had to download the Unity binaries was very discomforting. Not only was the link hosted on “drive.google.com”, but the files inside the zip folder were misnamed “SoccerTows” instead of “SoccerTwos“. It looks like this issue may have been corrected since then, but I won’t deny it caused a moment of panic when I couldn’t find the model I’d been training because it wound up in a slightly different location that I expected. I feel like Hugging Face should have the resources to host these files on their own, and the fact there were typos in the filenames makes me wonder if enough attention is being paid to potential supply chain vulnerabilities.

My least favorite unit had to be Unit 8 Part 1. I felt like I was being expected to recreate the Mona Lisa after simply watching a video of someone painting it. I didn’t really feel like I was learning anything except how to copy and paste code into the right locations. And this might be a sign of my age, but it was extremely frustrating to not be able to clearly read the code in the video. Some of the commands run in the console are only on screen for a second and it’s not always clear where the cursor is at. The information may be good, but the presentation leaves much to be desired. As a suggestion to the authors, I’d consider maybe splitting this content up and showing how to set up a good development environment earlier in the course so you can focus more on the PPO details here.

While fun examples, I also felt a little uneasy with the way the Atari games were included in this course. Specifically, the course presents Space Invaders in a manner that seems to attribute it to Atari when it was technically made by Taito. I feel like this is more a complaint for OpenAI as the primary maintainer of gym than it is toward Hugging Face, but I got the distinct impression that these Atari games the RL zoo are technically being pirated. After finding this arxiv paper on the subject, it looks like OpenAI erroneously assumed that the liberal license to the code in this paper gave them justification to use the Atari ROMs as benchmarks for large scale development. Given that these ROMs are being used to derive new commercial products, what might have been “fair use” by the original paper is now potentially copyright infringement on a massive scale. I strongly believe the developers of Space Invaders deserve to both be cited and paid for their work if its going to be used by AI companies in this way.

In conclusion, I think completing this course gave me a better understanding of what Hugging Face is attempting to build. The course taught me the struggle of providing reproduceable machine learning experiments and demonstrated the need to have a standardized process for sharing pre-trained models. This free course is a great introduction to these resources. At the same time, the course also drew my attention to the ways this hub might be misused as well. I think I would feel more comfortable with using models from the Hugging Face Hub if I knew that the models hosted there were sourced from ethically collected data. A good starting point might be to add a clearly identified “code license” and “data license” on project home pages. While Hugging Face says this should be included in the model’s README, a lot of the projects I saw on the hub didn’t readily include this information. I sincerely hope Hugging Face will take appropriate efforts to enforce a level of community standards that prevent it from turning into into “the wild west” of AI.

In any event, thank you to Thomas Simonini and Hugging Face for putting this course together. I really did have a fun time and learned a good deal in the process!

The models I built during this course can be found through my Hugging Face profile here.

Do you ever wonder if we do a disservice to children by asking them to name a favorite color too early in life? With such limited life experience, by what criteria does a child even decide? It seems like innocuous small talk on the surface, but once you’ve named your favorite color there’s a tendency to commit to that decision for the long haul. It’s incredibly unlikely that I’ll wake up tomorrow and spontaneously declare that now my favorite color is blue. That doesn’t imply preferences can’t change over time either.

Maybe it’s the innocence intrinsic to the childhood experience that gives our answer such impact. When I asked my young nephew what his favorite color was, I could see clear physical indicators of the thought being put into his response. His eyes rolled up and back, he moved his hand to his slightly elevated chin, then replied “purple!” with an unmistakable air of confidence. The certainty of his assertion in juxtaposition to his age is so awe inspiring that I’m almost jealous. We know the kids are right. The fact that I feel a need to protect him from the prejudices of society makes me wonder just how much of my preferred palette is a product of the environment where I grew up.

The reality is that my favorite color was undeniably influenced through social interactions. I distinctly remember a time when would tell people that my favorite color was black. This caused problems with other kids my age. “Black’s not a color! It’s the absence of color!” was so frequent a response that the very question of my favorite color would make my blood boil. It would make me see red. At some point I grew so tired of always having to argue with people about the definitions of color and hue that I eventually gave up the fight and started saying red instead. It worked rather effectively. People tend to associate red with anger and cede you more space as result. This can be quite rewarding when you’re as introverted as I am. It’s no surprise that the choice stuck, but something about this never sat right with me.

Part of me feels guilty that I didn’t defend my choice of black more strongly than I did. Every computer I’ve ever worked with has treated black as a valid color. Looking back, there’s an obvious explanation for this anti-black sentiment that I was too young to understand at the time. No one notices the contrast of white on white. If I would have claimed white as my favorite color, would it have been subjected to the same level of scrutiny for being “not a color but all colors”? I’m not really sure. The people that seem to most enjoy the fog are the same people that get offended when light passes through a prism.

Another part of me feels slightly hypocritical for simultaneously loving red and hating pink when I know the two colors are essentially the same hue. There was a very prevalent stigma attached to boys who liked pink. Much like the people who attacked my choice of black as a non-color, I engaged in similar form of mental gymnastics to assert that my favorite color was red was not pink. Perhaps it’s not really a question of “if” I was pressured into my color choice by social factors, but to what degree I tried to be what other people think of when they see me. There’s no denying that I preferred blood over bubblegum in my shades of red.

I’ve never been particularly good at understanding how people see me. There’s this shared connection people tend have between colors and feelings but sometimes I’m not sure I experience things quite the same way. If you lead me down a rainbow hallway I’ll choose the red door, but I will simultaneously see that red door and want it painted black. It’s hard not compare the way I tend to suppress my feelings with the way I avoid bright colors, but I think saturation and intensity both fall short of the metaphor I’m looking for here. Black is not necessarily the absence of feeling. Black is the shade on a hot summer day that provides you with reprieve from the relentless sun. Black is a warm trench-coat on a cold winter’s morning. Sometimes I need that armor which only the void can provide.

To make this analogy between colors and emotions work, perhaps it would make sense to add the concept of an alpha channel. In graphics programming it’s common to augment our definition of color from red, green, and blue to include opacity so we can express an image in layers. Emotions work like this also. Sometimes an emotion is relatively translucent and I see through it well enough to go on with my day. Other times an emotion is so opaque that I literally can’t see anything else. Grief is often the primary culprit of this. Grief stacks on layer after layer of blue on blue, heartache on heartache, until your vision disappears completely.

I already had a hard time distinguishing feelings, so learning how to recognize the layers of multiple emotions together was a difficult process. I could normally manage my rage on its own, but this phenomena of simultaneously being sad and angry was a burden I wasn’t prepared for. I felt lost in this purple haze, not knowing if I’m coming up or down. It used to be that red made me happy by pulling me out of my blues, but now the misery just follows with me. It’s like this painful bruise under my skin that I just have to bear with until it heals. Perhaps this is how purple came to be associated with courage.

Perhaps somewhere deep down I’m afraid of what purple represents. It’s so much easier for me to logically separate purple into red and blue than emotionally engage in the combination. When you find your pleasure in clouds of bright red cotton candy, it’s trivial to direct your hostility towards the cold blue steel it’s locked behind. Having a clearly defined prey to hunt makes life simpler for the predator. It’s much harder to accept that I’m drawn to the prowl because the thirst for blood distracts me from the river of tears I’m floating down. The reality is that I was raised in a world which portrays feelings as an impediment to survival and purple got caught in the crossfire.

I find it interesting how if you filter out the red and blue from white light you’re left with green. It seems like an appropriate metaphor for the unsatisfied hunger I feel. When you’re surrounded by mountains of purple, the grass is always greener on the other side. Feeling nothing at all would be preferable to feeling pain, but for every step forward I take I wind up sleepwalking back again. It makes me wonder if my efforts at suppressing feelings are ultimately futile. Maybe I need to learn how to scout my feelings from a higher altitude so I can figure out which bridges actually lead to verdant pastures and which ones I’ve already reduced to embers.

It’s here on this boundary between red and green that I’m starting to find hope again. In this stop and go world, we tend to think of red and green as opposites because that’s how our eyes work. However, it’s important to remember that these colors can be combined. When you mix red and green paint together it turns a disgusting greyish-brown, but you add red and green light together it produces a brilliant shade of yellow. A simple change in perspective can have huge consequences. It used to be that I associated yellow with fear, but now I’m starting to see it more optimistically as an opportunity for growth. Embedded in this line between red and green is a whole spectrum of yellows from the shade of fertilizer to shining sun. When the traffic signal turns yellow, do you speed up or slow down? Sometimes we need to do something that seems scary at the time in order for it to turn into something beautiful later.

Back when I was a teenager, I owned a pair of sunglasses with amber lenses and noticed something interesting happen when I’d take them off after wearing them for a while. Everything turned blue. In much the same way our eyes treats red and green as opposites, they also treat blue and yellow. When everything you see is displaying shades of gold naturally, your brain gets used to seeing the world like that and begins to filter it out. It seems fitting that yellow and blue have this relation. Trying to see hope in everything is so exhausting that I wind up seeing only the sorrow.

I think what makes these feelings difficult is my lack of control over them. They come down on me like rain. Part of me knows that both the sun and rain are necessary for growth. That’s the only way the roses bloom. The other part of me is scared to embrace what I can’t control. Bruce Lee taught me that there was power in being like water, but sometimes when it rains it floods. In times like that it’s Brandon’s line from The Crow that keeps me moving forward. I tell myself “it can’t rain all the time” as I try desperately to make my peace with the tears.

The first book I that I recall reading by a Black author was Alice Walker’s The Color Purple, so I revisited the film recently in preparation for this post. To say I didn’t understand it at the time would be a massive understatement. Looking back, part of me is kind of glad that I didn’t get it. I was so privileged that I didn’t even have a frame of reference for that level of suffering. I didn’t know what it was like to stand in the purple rain and be resigned to watch as destruction falls from the sky on everything you care about. I couldn’t imagine the courage it takes to feel that kind of pain and still be capable of laughter and compassion. In some ways it was better for me to not know these things in the same way I do now. Some feelings are best left unfelt.

In an effort to show solidarity with my nephew, I made it a point to wear my purple shirt around him. Note that I say “my” instead of “a” because at the time of writing this I literally only have the one. I’m the type to primarily dress myself in blue, red, black and grey. Heck, I even bought myself a grey guitar to play. Yet, my lone purple shirt represents something special to me. I only own it because Dr. Val Brown and a compassionate group of educators decided to #ClearTheAir by openly talking about racism on Twitter. It features the following quote from Dr. Martin Luther King Jr.:

“…persistent trying, perpetual experimentation, persevering togetherness. Like life, racial understanding is not something we find but something that we must create.”

As I’ve started to wear it more, I’m finding that I’ve become increasingly more comfortable seeing myself in shades of lavender and indigo. Maybe I look good in purple. Maybe there’s some alternate universe where my favorite color is violet or magenta– a place where I felt truly free to dream in color. Yet I’m so used to dreaming in ones and zeros that I have a hard time even envisioning that in my head. Somehow my brain can imagine that it sounds something like music though– a harmonious cacophony.

It’s easy for me to imagine dreams of blue. It’s easy for me to imagine dreams of red. But dreams of purple still feel elusive. The color itself feels like an illusion. Colors are to light what pitch is to sound. Red light has a low frequency and blue has a high frequency. Purple is a frequency of light that’s so much higher than blue that it starts to appear red again. Because the range of our vision is limited, we perceive the top and bottom of the color spectrum as forming a loop. Perhaps dreaming in purple is like dropping or raising your voice an octave to harmonize with a signer that’s outside your range– two dreams coming together as one.

I know better than to assume I can just snap my fingers and be in a different world. At the same time, I don’t want to give up hope for a better one either. It’s not within my power to remove all possible sources of pain from the world, but maybe there are steps I can take to share the load. Just because red is my favorite color doesn’t mean I can’t dabble in adjacent colors. There’s plenty of passion to be found between grapes and gold. Maybe allowing myself to space to express myself through purple can be an act of resistance that provides a crack in the clouds for hope to shine through.

Maybe someone out there needs to see that it’s okay to be purple.

Maybe that someone is me.

[I’d like to thank the following sources for the vibe that helped me get through this: Robert DeLong, K. Flay, WALK THE MOON, Lola Blanc, Counting Crows, Elliot Lee, The Rolling Stones, Bobby Vinton, Jimi Hendrix, Oingo Boingo, Pink Floyd, Coldplay, 311, grandson, Jessie Reyez, Prince, and The Art of Noise. Thanks for reminding me what it means to feel human.]

Introduction

I’m planning to do something a little different with this post. It’s probably more code heavy than my usual writing and more wordy than my usual coding. I wanted to share it because I think it’s simultaneously an interesting story and potentially useful script. My hope is to provide a gentle introduction to Application Programming Interfaces (or APIs): what they are, how to use them, and why having open access to them is so important in today’s society. Some familiarity with Python will be helpful when it comes to understanding the code snippets, but feel free to skip over those sections if you’re not feeling it.

Furthermore, I’d also caution in advance that this post contains a brief discussion of self-harm related content and mature language. It’s okay to set boundaries for yourself and skip to the code if that topic is sensitive.

While the narrative is my own, the code here is based on code samples from Google and Spotify licensed under the Apache 2.0 License.

Google’s quick start guide is available at: https://developers.google.com/youtube/v3/quickstart/python

Spotify’s quick start guide is available at: https://developer.spotify.com/documentation/web-api/quick-start/

You may obtain a copy of the Apache 2.0 License at: http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

The Python code that follows is designed to be run in Google Colab. Hypothetically, you should be able to open this code in your own notebook by using the following link: https://colab.research.google.com/github/rruff82/misc/blob/main/YTM2Spotify_clean.ipynb

The reason I’m using Colab for this is that it makes it extremely easy to authenticate to your Google account. Please understand that doing things the easy way carries inherent risks, and you should be mindful of any code you run in this manner.

If you’ve never used an interactive notebook before, it’s broken down into “cells” which can be either text or code. I’m going to talk you through all the steps I went through to migrate my likes and describe what I was thinking at each stage of the process. You’ll need to run each code cell one at a time as you follow along. You can do this directly in your web browser by selecting a code cell and either clicking the “play” icon on the left or by using CTRL+Enter on your keyboard.

Background

This story begins back in 2014 when Google did a collabaration with Marvel to release the Guardians of the Galaxy soundtrack for free on Google Play Music. Total marketing campaign success. It got me using the app more frequently, and the more I used it, the more I loved it! Before long I was buying an album a month and it seemed totally reasonable to pay for a subcription to the service.

The performance of the Play Music app was top notch. I especially appreciated the option for caching songs while on WiFi so I could listen to my favorites without eating through my cell phone’s data plan. It also gave me the ability to upload some of the more obscure albums in my collection and easily integrate them into playlists with subscription content. As I started liking songs on the platform, the algorithm started to get really good at providing me with new music that I actually liked. Furthermore, the integration with Google’s voice activated assistant was perfect for controlling my music mix while driving.

Unfortunately, Google terminated the Play Music service in 2020 and replaced it with YouTube Music. They migrated over my library well and my Play Music subscription had been upgraded to suppress ads on YouTube, but as a music service it wasn’t really meeting my needs. Songs started cutting out when my phone changed networks. The voice commands I relied on while driving stopped working. I’m also pretty sure the web app had a memory leak which would cause it to crash if I listened to a single station for too long. Everything that I loved about “Play Music” gradually slipped away.

The nail in the coffin came in 2022 when Google started rolling out content filters on YouTube that flagged videos for depictions of “self-harm”. I can understand that this is a good idea in theory, but it essentially made YouTube Music inoperable for me because there was no way to bypass this warning within the YouTube Music app. Rage Against the Machine’s entire debut album became inaccessible on the platform overnight — presumably because of the cover art’s depiction of self-immolation. What was most frustrating was that this ban was predominantly affecting playlists I had explicitly made to cope with the grief of having lost close family members to suicide. When you’ve relied on certain playlists of music to help you process painful emotions for years, having that safety net abruptly taken away from you by a third-party vendor feels pretty fuckin’ horrible. And then they had the balls to try to raise the price of their shitty service too? I felt so betrayed.

I don’t think Google understood the role music really played in my life. They knew the music was a commodity that I was willing to pay for, but they didn’t know the extent to which music was the cornerstone of my emotional foundation. Heck, I don’t think I understood it fully either. Other people talk about feeling as originating from the heart, but I’ve started to wonder if I’m different. I feel like my feelings are closely tied to the physical sensations I associate with music. It’s in my fingers when I feel the vibration of the strings against my fingers as I press down up on the frets. I feel it in my lungs when I try to belt out that falsetto. I can feel it resonate through my whole body when I stand to close to the bass amp. Music is more than just a background soundtrack in my life; the rock opera is a mirror to my soul.

For better or worse, Google had forced me into paying closer attention to how much of my music library was connected to suicidal themes. It was disturbing see some corners of my music collection getting banned while also seeing obvious references go unnoticed by the algorithm. The lyrics to “Pardon Me” by Incubus describe the same event as shown in Rage Against the Machine’s album cover but somehow wasn’t affected by the filter. Who gives a damn what the album cover looks like if I’m listening to it in my car? Trying to protect me from unanticipated self-harm related imagery in music is a impossible task when I grew up listening to artists like Kurt Cobain, Chris Cornell, and Chester Bennington. The line between the music that heals and the music that hurts is one that that only I can draw.

The problem with migrating to a new music service lies in training it to understand which music I want and what I don’t. Over the 8 years I used Google’s music services I had accumulated over 2200 “likes” and built dozens of playlists. This music profile has an emotional value that made it difficult to leave the platform. Cory Doctorow recently wrote about the “enshittification” of social media platforms, and I couldn’t help but feel that’s essentially what happened to Play Music. I was locked in by my data and needed a way out.

Having made a decision to leave, signed up for Spotify and found a service called TuneMyMusic that imported some of my key playlists. However, there was a 500 song limit on what they’d transfer for free and the playlists I did import weren’t as accurate as a would have liked. There was also the problem of it playing a lot of music that I didn’t like on radio because my library of favorites was still empty. Hearing Creed in my alt rock radio is enough to make me want to break shit. That’s when I decided I would use the Spotify API to just move my Youtube Music “likes” over myself.

There’s something empowering about being able to do this. The continual enshittification of Twitter serves as a constant reminder of the powerful impact of API access. Despite all its flaws, Google had graciously provided me with all the tools I’d need to exit it’s own platform — provided I was willing to put in some effort. I respect that and it’s comforting to know I can always reverse this process later if I so choose.

At the same time, this is probably not something the “average person” would be able to do on their own. But I think they could do it with assistance! I wrote this code primarily for myself, but hopefully this will provide a detailed enough example for others to build on.

I don’t think of APIs as a difficult concept to understand. They’re basically just a computer’s version of a contract for certain services. The difficulty lies mostly in knowing of their existence in the first place. You have to know who provides the service and what they allow you to do with it. Without some idea of what’s possible using an API, you’d have no idea what you could use them for!

Code

First, we’ll need some credentials to use the APIs. These credentials contain sensitive information that proves we’re supposed to have access to the APIs we want to use. It’s good practice to keep these secrets separate from your code, so I’m storing them in Google Drive because it’s convenient.

PLEASE DON’T SHARE THESE FILES WITH ANYONE!

For Youtube Music, you’ll need to first open up the Google Cloud Console:

• https://console.cloud.google.com/

Go to “Credentials” and select “Create OAuth client ID”. Configure the consent screen then create and download OAuth client secret. Upload this file to google drive as “creds_google.json”.

For Spotify, you’ll need to go to the Developer Dashboard:

• https://developer.spotify.com/dashboard/applications

Select the option to create a new application. Where it asks you for a “redirect URI”, add “https://localhost:8888/callback”. We won’t actually use this since we’re not building a full web application, but we need to put in something here so we’re choosing something that would be useful in a debugging environment later.

Once you’ve created the credentials, take note of the “client_id” and “client_secret”. You’ll also need your Spotify username which you can find on the account overview page:

• https://www.spotify.com/us/account/overview/

Using your favorite text editor, create a new text file containing the information above using the following JSON format:

{
  "client_id":"REPLACE WITH CLIENT ID",
  "client_secret":"REPLACE WITH SECRET",
  "user_id":"REPLACE WITH SPOTIFY USERNAME"
}

Save and upload to Google Drive as “creds_spotify.json”.

Next, we’re going to install a Python module to simplify access to the Spotify API. The following command will tell Colab to install the latest “Spotipy” package from the PyPI repository. We do this first because we might need to restart the interpreter afterwards.

!pip install spotipy

Now that the “Spotipy” package is installed, we’ll import all the modules we’ll need for the rest of the script. Typically I import them as I produce my code but collect them at the top. This makes it easy to check that all my prerequisites are accounted for before I get too deep into my code.

# The first section are modules we'll need to read our credentials files in
import json
import os
import os.path

# The following are modules for accessing the Spotify API using the package
# we just installed.
import spotipy
import spotipy.util as util
from spotipy.oauth2 import SpotifyOAuth
from spotipy.oauth2 import SpotifyClientCredentials

#  We'll need "sleep" so that I can limit the rate of my Spotify API calls
from time import sleep

# The following are modules that we'll need to authenticate to the Youtube API
import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors

from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials

from google.colab import drive

Since we stored our credentials in Google Drive, we’re going need to give ourself access to it from this notebook. The following line will tell the interpreter to treat our Google Drive as a local folder. Please exercise caution when doing this as we’re technically granting more access than we actually need.

drive.mount('/content/drive',readonly=True)

Now that we have access to our credentials files, we’ll need to authorize the application to read our playlists from Youtube. The following code will prompt you to log in with your Google account and warn you that you are providing your information to an application that’s under development. This is okay because we created the app. Copy and paste the code from the confirmation screen into the running cell to complete the authorization process.

api_service_name = "youtube"
api_version = "v3"
client_secrets_file = "/content/drive/My Drive/creds_google.json"
scopes = ["https://www.googleapis.com/auth/youtube.readonly"]

flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(
    client_secrets_file, scopes)
credentials = flow.run_console()

The following code may prompt you to enable the YouTube API for the project in the Google Cloud Console the first time you run it. Follow the directions as needed.

youtube = googleapiclient.discovery.build(
        api_service_name, api_version, credentials=credentials)

Getting this far means we now have access to our YouTube data. Youtube Music basically stores all your likes in a playlist called “LM”. I’m presuming it stands for “Liked Music”. The API has a limit to how many results it would return at a time, so the following code will loop through the pages of the API response until we’ve collected the entire list.

def get_all_items_by_playlist(playlistId="LM"):
    request = youtube.playlistItems().list(
        playlistId=playlistId,
        part="contentDetails",
        maxResults=50
    )
    response = request.execute()
    all_items = response['items']
    while response.get('nextPageToken',"") != "":
        request = youtube.playlistItems().list(
          playlistId=playlistId,
          part="contentDetails",
          pageToken=response['nextPageToken'],
          maxResults=50
        )
        response = request.execute()
        all_items.extend(response['items'])
    return all_items
all_items = get_all_items_by_playlist()

As a quick check, I compared the length of this list with my expected value of 2255 to ensure I’m getting the correct playlist.

print(len(all_items))

Next I took a peek at the structure of the data by printing the first 5 items.

all_items[0:5]

It turns that “contentDetails” didn’t contain very many details about the content. There’s not enough information to identify a song based on this, so I reached back out to the YouTube API and asked for a “snippet” instead.

def get_music_metadata(playlistItem):
    request = youtube.videos().list(
        id=playlistItem['contentDetails']['videoId'],
        part="snippet"
    )
    response = request.execute()
    return response

test_md = get_music_metadata(all_items[1])
print(test_md)

At least now we have some information to work with here! There’s no clear cut way to pick out the song and artist though. Some YouTube videos have both the artist and the song in the title, but the majority only include the artist name in the channel. For my purposes, I’m going to treat the channel name up to the first dash as a proxy for the artist and video title as the song title. It’s not going to be 100% accurate, but hopefully will be close enough that Spotify can figure it out.

def guess_song_title(metadata):
  if len(metadata['items']):
    return metadata['items'][0]['snippet']['title']
  else:
    print('Failed to find song name for item:')
    print(metadata)
    return "Unknown Song"

def guess_artist_name(metadata):
  if len(metadata['items']):
    return metadata['items'][0]['snippet']['channelTitle'].split('-')[0].strip()
  else:
    print('Failed to find artist for item:')
    print(metadata)
    return "Unknown Artist"

print(guess_song_title(test_md))
print(guess_artist_name(test_md))

Having verified that I get a reasonable guess as to the song and artist associated with a sample video ID, I looped back through my list of all likes to acquire the rest of the dataset.

for i,item in enumerate(all_items):
    all_items[i]['metadata'] = get_music_metadata(item)
    all_items[i]['artist'] = guess_artist_name(all_items[i]['metadata'])
    all_items[i]['song'] = guess_song_title(all_items[i]['metadata'])

---

My results showed that my script failed on 1 item which seems to have been deleted, but that’s not a bad success rate considering how many it ran through. The next step is to see if we can find the songs on Spotify, so lets pull our credentials from the file we created earlier.

SPOTIFY_CREDS = "/content/drive/My Drive/creds_spotify.json"
with open(SPOTIFY_CREDS,"r") as f:
  spotify_credentials = json.load(f)

Now that we have our credentials in hand, we need to authenticate to the Spotify API like we did with Google earlier. Note that the redirect URI appears again here, so you may need to update this if you used something different.

market = [ "AD", "AR", "AT", "AU", "BE", "BG", "BO", "BR", "CA", "CH", "CL", "CO", "CR", "CY", 
      "CZ", "DE", "DK", "DO", "EC", "EE", "ES", "FI", "FR", "GB", "GR", "GT", "HK", "HN", "HU", 
      "ID", "IE", "IS", "IT", "JP", "LI", "LT", "LU", "LV", "MC", "MT", "MX", "MY", "NI", "NL", 
      "NO", "NZ", "PA", "PE", "PH", "PL", "PT", "PY", "SE", "SG", "SK", "SV", "TH", "TR", "TW", 
      "US", "UY", "VN" ]

sp_scope = "user-library-read user-library-modify"
sp_credentials = SpotifyClientCredentials(
        client_id=spotify_credentials["client_id"],
        client_secret=spotify_credentials["client_secret"])

auth_manager = SpotifyClientCredentials(client_id=spotify_credentials["client_id"],client_secret=spotify_credentials["client_secret"])     

sp_oauth = SpotifyOAuth(
    username=spotify_credentials["user_id"],
    scope=sp_scope,
    client_id=spotify_credentials["client_id"],
    client_secret=spotify_credentials["client_secret"],
    redirect_uri="https://localhost:8888/callback",
    open_browser=False
)

Like the Google API before it, Spotify will also ask us to provide an access token. We don’t actually have a web server set up to provide a nice user interface here, so you’re going to need to pull the access token from the address bar in your web browser.

When you run the following cell, it provide a link that opens in your web browser. When Spotify completes the authorization, it sends you to the redirect URI which will likely fail because your computer is probably not running a web server on the specified port (if you are, you might need to adapt a little here). This “Site can’t be reached” error is okay, but what we need here is the URL from the address bar. It should look something like this:

• https://localhost:8888/callback?code=long-sequence-of-random-characters

This whole URL will needed to be copy and pasted into this notebook to complete the authorization process.

sp_token = sp_oauth.get_access_token()

Now that we have an access token, let’s finish the the log in process and verify that we’re authenticated as the correct user.

sp = spotipy.Spotify(auth_manager=sp_oauth)
print(sp.current_user())

The next step is to use the artists and song titles we extracted from YouTube to find the corresponding track on Spotify. I’m going to put a dash between the artist name and the song title for my search string, then use the first hit in the results so see what I found.

def get_search_string(item):
  return item['artist']+" - "+item['song']

res = sp.search(get_search_string(all_items[0]), type="track", market=market, limit=1)
print(res)

The key piece of information we need here is the “track id”.

res['tracks']['items'][0]['id']

Since we need to do this for a lot of songs, we’ll wrap this process in a function and verify that it works.

def get_track_id(item):
  if item['song']=="Unknown Song" or item['artist']=="Unknown Artist":
    print(f"Skipping unknown item: {item}")
    return None
  search_str = get_search_string(item)
  res = sp.search(search_str, type="track", market=market, limit=1)
  if len(res['tracks']['items']):
    return res['tracks']['items'][0]['id']
  else:
    return None
my_track_id = get_track_id(all_items[0])
print(my_track_id)

Having extracted this track id, we can use it to add the item to our Spotify library. Before we do that, we should probably make sure it’s not already in the collection.

sp.current_user_saved_tracks_contains([my_track_id])

If it’s your first time running the above code, it should return “[False]” because we haven’t added anything to our library yet. Next, we’ll add it and try again.

def add_track_to_library(item):
  track_id = get_track_id(item)
  track_name = get_search_string(item)
  if track_id is None:
    print(f"Couldn't find track for: {track_name}")
    return
  if sp.current_user_saved_tracks_contains([track_id])[0]:
    print(f"Track is already liked: {track_name}")
    return
  print(f"Attempting to add track: {track_name}")
  sp.current_user_saved_tracks_add([track_id])

add_track_to_library(all_items[0])
sp.current_user_saved_tracks_contains([my_track_id])

The cell above should return “[True]” to indicate that the item has been successfully added to our library. The only thing left is to apply this process to the rest of our list. I’ve added a small sleep step to this loop to ensure that I stay within the Spotify usage limits

for i,item in enumerate(all_items):
  add_track_to_library(item)
  sleep(0.1)

Conclusion

That’s it! We’ve successfully migrated our YouTube Music Likes to Spotify Favorites. Now when Spotify tries to dynamically generate radio stations, it at least has some general information about what I’d like to listen to. There’s plenty of room for improvement in this code, but it got the job I needed done so I probably won’t be coming back to it any time soon.

If you’re looking for some project ideas to further develop your own understanding of these APIs, here are some potential modifications you might consider:

• Try to improve the search criteria for better accuracy.
• Try to create a new Spotify playlist from a Youtube Music playlist.
• Try to sync Spotify favorites back to YouTube Music.
• Try to implement this as a complete web application.
• Try to analyze patterns in your music collection using data available through Spotify’s “Audio Features”.

There’s just so much you can do with these APIs! All it takes is a little imagination and some elbow grease!