Newgrounds.com — Everything, By Everyone.

A while back I posted a blog in which I pondered the ingredients necessary to make chart topping hit songs. I decided to do a research project to see if I could discover those ingredients. Here is a status update on that project:

I collected weekly data on the #1 songs from Billboard's Top 100 chart, starting January 1, 1990, and ending December 31, 2011. That time period spans 22 years, and 1148 weeks. During this time period, there were 306 unique chart toppers. This is where I hit a pitfall. You see, in order to collect data like the key signature, chord progressions, tempo, and the record company that produced the song, I would have to do a LOT of research. I started searching for that information. I had to search Wikipedia and guitar tab web sites to find a lot of it, and even then, there were some songs that just didn't have that kind of information available. The bottom line is that I would have to manually search the web for each and every song. I am extraordinarily tenacious when I'm curious about something, but this would be too much work for curiosity's sake, even for me. So I've decided to make some amendments to the project:

1. Regression analysis is out of the question now. My original goal was to test for correlations between variables like key, tempo, and chord progressions, etc., but due to the amount of time required to gather that data, I simply cannot do it. Besides, I have gained a lot of insight just browsing the data that I did collect. More on that in a bit...
2. I will complete the experiment with data that's easy to collect and analyze.

As you might expect, the #1 predictor of a song's success is if the artist has had previous hits. However, you might be surprised to find out just how important prior success is. Out of the 1148 songs, there were only 192 unique artists. That means that if you were to pick a random week from that 22 year time-span, there would be at least an 83.28% chance that the #1 song for that week was by an artist with multiple hit songs. That's amazing! I say at least 83.28%, because some of the artists near the beginning of that time-span had hits before 1/1/1990, and faded out during the 90's. Jon Bon Jovi, for example, had chart topping hits in the 80's, but his last #1 hit on Billboard was "Blaze of Glory" the week of September 3, 1990. So the 83.28% may be a low estimate. This probability will also increase if an artist with only 1 hit in my study comes out with another chart topper in the future.

If I were to run a regression for the data, I would have had a binary explanatory variable to account for an artist's prior successes. This dummy variable would have been VERY strongly correlated with the dependent variable. I wouldn't be surprised if it was 0.8 or higher, and the p-value was low. That's how significant I interpreted prior success to be!

My goal was to find similarities between popular songs that didn't exist in unpopular ones. Perhaps, given the importance of an artist's prior success, it would be more interesting to learn how songs by breakout artists become popular. Between 1990 and 2011 there were artists that lived and died, like Boyz II Men, one hit wonder artists, like Hanson, and artists that surfaced from seemingly nowhere, like Adele. Breakout artists like Adele were able to put together songs with all the necessary ingredients for success, besides prior success. Do breakout hits like "Rolling in the Deep" have similarities between each other? Is there a way to predict the next breakout artist? Can you combine possible similarities between breakout hits into your own song and be guaranteed success?

After reviewing the data, I have come to some conclusions, even without using statistical analysis:

1. Hit songs are most commonly Pop, with an increasing trend toward Electro/Pop fusions.
2. Lesser known artists collaborating with successful ones are not guaranteed future success.
3. Chart toppers are almost always in 4/4 time, and many of them have a bass hit on every quarter note. The songs without a quarter note bass hit have variations of it, like a bass kick on beats 1 and 3, followed by a snare on beats 2 and 4. Notice I said "many of them" not "all of them". There may only be half of the songs in my study that obey this rule, but they would still be the majority.
4. The most common theme in popular songs is love, followed by sex and partying, respectively. The most common lyrics are: love, heart, come, away, life, your, and variations of the word baby, (or in 1 particular case, MMMbop.)
5. In almost every song about sex, the topic is discussed with innuendos, rather than mentioned explicitly. (haha, nobody outright says "I want to have sex with you," probably because it wouldn't be cool.)
6. The longer a song is #1, the less likely it is to reclaim the top spot after usurpation.
7. Breakout artists have to wait longer from the release of a song to when it becomes a hit. I suspected this, and it was confirmed by the data I started collecting to measure the time from release to #1, but since I didn't finish collecting it, I can't say for certain that this rule is always true.

I would expect most or all of my conclusions to be validated by the statistics, if I were to run the regression. If I ran it, I might multiply the equation by (1-Dummy for prior success) to target the breakout artists' song similarities. Given what I was able to conclude already, I'm satisfied with the experiment. I just wish I could have put numerical values on the data. I still think that there might be a magic formula to forecast hit songs, but without incentive to spend time finishing this project, I'll never know.

Let me know what you think of all this! Would you have done anything differently?