Skip to content

Blog

Iterating Through the Folder Dictionary

After taking a break and coming back to it, I finally was able to iterate through the dictionary containing the folders I’ve sorted my records in to on Discogs.

Dictionaries are a basic object in Python, but my struggles show that to be good - at anything - takes practice. And I haven’t been practicing. Thank goodness for Stack Overflow!

I embedded all kinds of print statements trying to figure out how to play with the JSON from Discogs.

After lots of trial and error, Google searches and Stack Overflow searches, I’ve got it.

After making the API call to Discogs, we get the JSON back:

record_json = response.json()

json_data = record_json
json_folders = json_data[”folders”]

json_folders is a list, so we add the last line above. Now I had to make a for loop to iterate through that list - which is a list of dictionaries.

The if statement looks to see if folder - which we passed from the controller when the user clicked which kind of album they want a random answer to, matches the folder ID found in the JSON.

for get_folder_id in json_folders:
    print(json_folders)
    print(len(json_folders))

    if get_folder_id[”id”] == folder:

        lp_count = get_folder_id[”count”]
        print(”Folder passed = “, folder, ”LP Count is :”, lp_count)
        # folder_test = 2162484
        print(folder, lp_count)

Voila! From there, I get a random number between one and the count found in the JSON. That number is then passed to the pagination request to get the album ID.

Next up: Either fix the pagination method to reduce the if / else statement or move on to getting the album release information. I also need to remove a ton of print statements and clean up and remove all the methods I don’t have to use after refactoring this. Progress!

Silver Saucer Random Play Progress

I’ve made some good progress over the last few days on being able to pick an album at random out of a given folder and get information about it.

All of the work has focused on the back-end. Once I know I’m getting the right data out of Discogs, I’ll hook up the front-end to display it, but for now, this has been a little more complex than I expected. (It always is, isn’t it?)

One thing that’s jumped out at me was I don’t understand how people do TDD (Test Driven Development). The amount of trial and error I do, even on “simple” things like building the right URL for the API call is ridiculous. I know it’s because I’m a novice at Python and I don’t do this for a living all day every day, but I have a lot of respect for those that do TDD. One of my original goals was to do this in TDD, but I have my excuses.

You can tell I’m a novice by how procedural my code is. Do this, then do that. It was working well in a very procedural manner, so yesterday I refactored it all. Yup, I ripped it up and started over. Well, not from scratch. I could visualize in my head how to make the code more Pythonic and I was able to re-use some of the code, but most of it was refactored in the play_service. A good example is going back to the folder example. I had hard coded each folder ID and each folder had it’s own method to get a random album release ID and its data.

I’ve been able to take the folder selected by the user (LP, EP or single) and just pass variable that to the method in play_service.py. This allowed me to use the same method to get the release ID information and the method doesn’t care if it’s a single or full album.

I still need to fix the loop I’m iterating through in the JSON, but I’m really close. This is one of the downsides to being a hobbyist - so much I don’t remember, including basic things, like how to iterate through a dictionary.

Practice makes perfect and I needs lots of practice. I told myself I need to think of some more projects to work on, but one project at a time. In the meantime, I should sign up for something like Python Morsels and keep up with it.

This weekend I’m hoping to fix looping over the list in the dictionary and also fix API call that deals with the pagination of results from Discogs. You can see what a mess it is in the play_service.py file and I know there has to be a better way to programmatically determine the pagination than a long if / else statement. (And if I ever get over a 1000 records it will break! Or my wife will break me when I have that many…)

Discogs Authentication

I was ready this morning to write a new blog post chronicling all the problems I had yesterday trying to authenticate against the Discogs API. After finding the motivation to sit down and code for a few hours, I ended the day with almost nothing done. I could access the parts of my collection that were public, but the parts that required authentication I couldn’t figure out.

I sat down with my copy of coffee this morning, re-read the Discogs API authentication documentation, and voila! It just clicked. Sometimes you just need a good night’s sleep.

You have two options for authentication when building an app with the Discogs API: full Oauth that allows users to login and validate their login, or pass a user token as part of the API query. Since my app is just going to use my Discogs information, the user token is perfect. Both authentication options give you options for a higher rate limit and the ability to return an image - and this last one will be important later.

I thought I would tackle what would be the easiest part first - have the app pick a random album for me to play.

It was a lot harder than I expected, but I’ve made good progress.

Now that I could get back details of my collection via authentication, I had Discogs return a list of all the folders I’ve organized my music into. I put each of the following into its own folder, as when I hit the random button, I don’t want it to pick a 45 or 7” record with only two songs on it. I have 10 folders, with 0 and 9 included by default:

  • 0 - all
  • 1 - 10” (10” records, mostly EPs)
  • 2 - 12” (12” singles or EPs)
  • 3 - 7” (7” or 45 rpm singles)
  • 4 - Autographed (currently empty, but you can have a release in more than one folder)
  • 5 - Cassette
  • 6 - CD
  • 7 - Digital
  • 8 - LP
  • 9 - Uncategorized

The JSON returned looks like this:

{
    "folders": [
        {
            "id": 0,
            "name": "All",
            "count": 799,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/0"
        },
        {
            "id": 2162486,
            "name": "10\"",
            "count": 19,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162486"
        },
        {
            "id": 2198941,
            "name": "12\"",
            "count": 30,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2198941"
        },
        {
            "id": 2162483,
            "name": "7\"",
            "count": 76,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162483"
        },
        {
            "id": 2162485,
            "name": "Autographed",
            "count": 0,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162485"
        },
        {
            "id": 2162487,
            "name": "Cassette",
            "count": 1,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162487"
        },
        {
            "id": 2162488,
            "name": "CD",
            "count": 6,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162488"
        },
        {
            "id": 2198943,
            "name": "Digital",
            "count": 1,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2198943"
        },
        {
            "id": 2162484,
            "name": "LP",
            "count": 666,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162484"
        },
        {
            "id": 1,
            "name": "Uncategorized",
            "count": 0,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/1"
        }
    ]
}

I then got stuck for a while before I realized I should be using the folder IDs in the resource_url and not the JSON number of 0-9.

That - plus authentication - are challenges for someone like me still learning. But progress is being made. Lots of trial and error with string concatenation to get the URLs and tokens right. The next thing to solve for: Discogs only returns 100 items in a list, making you interact with pagination to go deeper. I’ll need to do this to take the random number generated of while album to play to get its data, including the image to show, when a random album is chosen.

Pre-Commit Git Hooks

It was a pleasant surprise recently to get an email from Michael Kennedy at Talk Python. As a previous buyer of one of the Talk Python Everything Bundle which includes a number of different Python training courses, I was eligible for the “Pro” Talk Python podcast subscription, which doesn’t included ads.

Going through the backlog, I added a bunch of episodes, and the timing of Episode 282: Pre-Commit Framework was perfect. Just as I’m getting back into working on my projects, now is a perfect time to set up Black to help format my code. The documentation was easy to follow as well as other walk throughs on the web, and now every time I check in code, it runs black to format it Pythonically, isort to automatically sort my imports in each of my files, and I’ll be adding flake8 this weekend, too.

Thanks again for the Pro podcast subscription, Talk Python!

The one piece of data Discogs doesn’t have

As a record collector, Discogs is invaluable. What started as a crowd sourced website to catalog electronic music, it now has a catalog of almost every piece of recorded music. They’ve also added robust e-commerce abilities so it makes buying and selling easy - whether it’s for something popular or a rare record only made in Greece.

It can take a little work to figure out which record you have, especially if it’s something popular like Fleetwood Mac’s Rumours, which has literally hundreds of different releases, some of which come down to figuring out the engraving on the inside of the record to figure out which pressing plant made it.

The other thing that makes Discogs great is they have a Developer API available, making it easy to interact with all of the data they have stored programmatically. All of the data is available in JSON and there is also a Python library available, making it even easier.

The amount of data about each album and each of its variants is amazing. I won’t go through all of it and you can get an overview here , but it includes almost everything you could probably think of off the top of your head.

Each album release has a “master” release, that represent a set of similar releases. If an album is released this Friday, for example, and comes out on CD, cassette and vinyl, the master release would catalog each of these individually. Each individual one is a “release”. (And here’s to hoping Discogs changes the word master sometime in the near future).

Here is the master release page for Pearl Jam’s Ten, released in 1991:

Pearl Jam Ten Master Release

You can also see there are 251 different versions (or releases) of Ten on different formats. The list is currently sorted by chronological year of release and if we scroll all the way down to 2009, we can find the repress I own on vinyl. You can click through each individual release to see how it’s unique or find that particular version for sale. Clicking through we see:

Pearl Jam Vinyl Repress Release

There is one thing I am looking for to build one part of my app: What date was an album released? Neither the master release or the release of the album I own supplies anything but the year, not the full date, which is what I want to know to celebrate an album’s anniversary.

That’s by design. If you check out Discogs' Submission Guidelines , only the year is required. If you know the actual date, great, if you know the year and month, you can also input that.

I also found a forum post from a few years ago, which I can’t find now, where the Discogs developers stated they wouldn’t change the release date field to require a full year / month / day and they didn’t want to change the database schema to support that (which I don’t blame them).

But that doesn’t help me in building a web app that shows which albums were released today in history.

In addition to my pandemic induced loss of motivation, this has blocked me from moving forward. I have a couple of potential ideas to move forward.

The first is interfacing with the MusicBrainz API in addition to Discogs. MusicBrainz is a full open and free music database licensed under Creative Commons. They even link back to Discogs to make it easy to link releases. My thought is if Discogs returns a date field that doesn’t include the month and day, to then query MusicBrainz to see if they have it. I’m a little worried about responsiveness in making two queries like that, plus dealing with two different JSON schemas and matching.

The second idea may work for the majority of releases, but there is a catch at the end. Doing a random check of albums, when you look at a master release, the very first release chronologically almost always has the full date attached. I could query the API, and slice the list returned and use the date provided in the first entry in the list. But I checked one random album I had that I knew is an obscure release (Maggie’s Dream if you’re curious) and there is no date.

So I think what I’m going to do - as I need to get going and do something - is to build out and test the second option. This keeps me with just one API to learn in the short term and I can probably build a query pretty easily that will sort all of my releases by chronological order (using the trick above) that will show me how many albums don’t have a full release date. 10%? More? Less? Might as well build it out and get something done!

Stupid Pandemic

The COVID-19 pandemic has changed a lot of things for a lot of people. One of the things it’s done to me is made it harder to focus on my coding projects. (Part of that is I’m very deadline driven, and well, no deadlines on personal projects…)

Looking back on this year, I’ve actually accomplished a lot, but just not on the coding side. I’m just getting back to it and I have two projects I’m going to make progress on. The first one is my music centric web app built with Pyramid that interacts with my Discogs profile via the Discogs API. The second project involves a Raspberry Pi, but I’m much better at the coding side of that than I am on the hardware side. (And that’s saying something with my novice coding skills).

Introducing Silver Saucer

I’ve started my next Python project: Silver Saucer, which will be hosted at silversaucer.com. I originally bought the domain about ten years ago when I was thinking of going into business for myself, which never happened and I hung onto the domain name because I liked it. It was inspired by Neil Gaiman’s poem, The Day The Saucers Came, which I also have a framed art print of:

{{< figure src="day-the-saucers-came.jpg" title="The Day the Saucers Came" link="day-the-saucers-came-original.jpeg" >}}

Once upon a time, I was also a big fan of The X-Files, so it seems fitting.

With the pandemic here, I’ve had a little time to work on some long dormant projects, this being one of them. I’m stuck on another Python project (more because of the hardware than the coding), so I thought I’d find some time to work on this.

I have a few different goals of things I want to learn and practice:

Pyramid

It’s my third project using the Pyramid web framework. I don’t have an urge to learn Flask or another framework - I would like to get better at Pyramid, which I know a little. I also really like the Pyramid community. The few times I’ve become stuck and asked for help in the Pyramid IRC channel, they’ve been both welcoming and helpful. My first two Pyramid projects were based on the first training Talk Python offered for Pyramid, which used a package called pyramid_handlers which is no longer the recommend way to build a web app in Pyramid. I’m doing it the recommended way this time, using a class based approach.

Bootstrap and CSS

I know a little of HTML, enough to get by. But CSS and Bootstrap, not so much. I’ve already integrated a Bootstrap theme and tweaked it where it’s almost working, but I’m just hacking at it - I don’t really know what I’m doing and it’s something I want to get better at. I should probably find some good HTML / CSS tutorials and go through those.

Discogs API Integration

The goal for Silversaucer.com is to integrate with the Discogs API). Discogs.com is a website that lets you catalog your record collection and also includes community features and a marketplace where you can buy and sell records (or CDs or almost any kind of media). There are two things I want to build using the Discogs API:

Play a random record

I know that Discogs already has a feature on their website where you can have it randomly choose an item in your collection and if you shake the mobile app it will also show you a random item in your collection. I want to take that to the next level and sort by type (record, 45, CD, etc.).

On this day

The second things I want to build is a page that shows all records released for today’s date. This one is going to be more complicated and I’ll share more in a separate blog post.

Testing

I’ve blogged about it before, but I struggle to learn pytest. I’m going to continue to try and learn more about software testing, starting with this app.

Infrastructure

I’ve already hooked up Silver Saucer to Azure Pipelines to automatically do continuous integration. I want to integrate pytest and code coverage next. After that I may look into continuous delivery, but there’s is a lot about Azure that I don’t know.

Plenty of things to learn and keep me busy during these interesting times. Here's a teaser for making it to the end:

{{< figure src="silversaucer.png" title="Silver Saucer screenshot" link="silversaucer.png" >}}

Another Reason to Love Open Source

I've been using free and open source software (FOSS) for over twenty years and I've involved in with different FOSS projects for ten to fifteen years (on and off). Thare are so many different aspects of the open source community that I love (community being a big one to start with) and here's another example.

I'm trying to build a new website using Python and the Pyramid framework. I want to interact with APIs on the Discogs website. Discogs is a website and community that allows you to catalog your music and also buy and sell music. For me, this is all about my record collection. I'll be posting more about what I want to do, but right now I'm just trying to learn what my options are with the Discogs API and how to connect and interact with their API. I'm struggling a little bit to get authentication working.

I'm also using Postman, an application that makes it easy to test different APIs and see how they return data. I first learned about Postman through the Talk Python trainings (another reason to love the Talk Python trainings) and it was a core tool in building NFLPool and MLBPool to understand how MySportsFeeds showed sports data to integrate into those two apps.

If I'm struggling to authenticate and login to the Discogs API, I'm never going to get any results.

And here is why I love open source: Github user leopuleo has a Discogs-Postman repository where he has taken the time to not only put together almost 60 different ways to use the Discogs API, it also includes documentation and tests!

Postman Discogs

Postman Discogs Tests

This is all licensed under the MIT License, giving you the ability to take it and modify as you need to. leopuleo didn't have to do this, nor did he have to share it, or give it a license that just gives it away and let's you do anything you want with it. This is just one reason why open source is awesome - someone builds something, shares it with the world, and even better, let's you build on top of it.

Open source and free software rock.

RetroPie Update

I hooked up the Raspberry Pi to test it out. I did an update and it broke the boot up process and now wants the username and password. Think I remember that? Nope! Of course I don’t remember the password. So I downloaded a new RetroPie image which has led me down a rabbit hole of MAME configuration. The buttons on the controller don’t seem to be mapped correctly.

I did downloaded a new romset from the Internet Archive for MAME 0.209. Doing further research, I tried both the mame4all emulator and libretro. Both had unexpected results. This led me into further research and it looks like I want to use a MAME 0.78 romset and the mame2003 emulator. (This is due to RetroPie using the libretro emulation engine as it's primary emulator). So I downloaded another 50GB from the Internet Archive and I’ll try out those ROMs to see if the buttons map correctly on the knockoff PS3 controller I’m using for testing. It’s interesting that the joystick buttons are different based on what emulator I’m using.

I did briefly consider not using the Raspberry Pi 3B+ for the arcade cabinet and switching to a PC. Reading some forums and Reddit posts, a PC is always recommended for MAME. All I really want to do is play classic 80s arcade games that should mostly work with the Raspberry Pi. A PC is mostly tempting to add the Dolphin emulator so I can play Super Mario Strikers for the Nintendo Gamecube, but I don’t know if that’s worth the expense or the trouble. The Pi should more than handle the classic arcade games, Sega Genesis and Nintendo games I want to play.

But doing all the research is fun!

Practice Cuts

A few days ago I asked Kelly for help in drawing the arcade on the plywood. I had tried to draw the lines for the top of the cabinet, but I just couldn’t visualize it. (It’s both an ongoing joke and the truth that her spacial relationships are a hundred times better than mine. You don’t want to see me try to even pack a suitcase, I’m terrible). I also needed help with how to draw the curves along the sides. Luckily, she made some time Saturday morning and by afternoon I had the saws out.

Practice Cuts

We have two old pieces of plywood that we’ve used for desks at LAN / gaming parties over the last twenty years, with the last time a few years ago here at the new house. This is what I used for my first practice round with the saws.

The cuts with the circular saw went pretty well. They were fairly straight. It was the jigsaw that was harder than expected. I underestimated the strength needed to move the jigsaw. I made three mistakes: one of the big curves isn’t as smooth as it should be, though it could be sanded down; the marquee cut didn’t follow the line correctly, though you can’t really tell; and I over cut a point where the jigsaw and circular saw cuts meet.

Overall, it was good for a first time. I’m going to make one more practice run and then on to the real thing. It’s going to snow the next few days, but this weekend looks nice...

Right Side Panel

Left Side Panel