Skip to content

Blog

Learning pytest using continuous integration with Azure Pipelines (or SSH key hell) - Part 1

Introduction

I’m still on my quest to learn more Python and at the top of that list is learning pytest. I just can’t wrap my head around testing and I know my two Pyramid apps aren’t “complete” until there are tests. (I did write docs, so I have that going for me).

A couple months ago I (very easily) added continuous integration to NFLPool using Microsoft’s Azure Pipelines. I, like many other people, have been blown away by the right turn Microsoft made a few years back to embrace open source, and wanted to give Azure Pipelines a try. Every time I make a commit or Pyup.io submits a pull request to update a Python package, Azure Pipelines builds NFLPool. Of course it fails, because the tests I’ve written so far fail. To end this cycle, I really need to learn how to write tests!

I have a domain I don’t use, silversaucer.com. I have a few different ideas for some projects for the domain. I’m going to build another web app using Pyramid to start, with a goal of properly using classes in Pyramid (and not Pyramid handlers) to start. I’ve also been reading Test-Driven Development with Python, 2nd Edition, that I received from a recent Humble Bundle filled with Python books.

I was thinking that this was a perfect opportunity to learn pytest. I would create a new project in Pyramid and use TDD to write my tests as I write my code. If I’m going to do that, I might as well set up continuous integration right away and pretend I’m a real developer.

Setting up Pyramid

Since I’m going to be using a Test Driven Development philosophy, all I’m going to do is create a Pyramid project and not make any changes to it yet. I used Pyramid’s cookiecutter to create the project and then committed it to my Github repository. That’s it!

Hooking up Azure Pipelines

Here is where the fun starts. I’m not going to go through this process as Microsoft has great documentation for Azure and set up.

Walk through the setup and connect to your repository on Github. Azure Pipelines will create the needed YAML file, commit it and run your first build.

Here is where my build failed for Silver Saucer. I was getting the following error:

Obtaining silversaucer from git+git@github.com:prcutler/silversaucer.git@75932f389536b59993fa780b281170849ff92238#egg=silversaucer (from -r requirements.txt (line 38)) Cloning git@github.com:prcutler/silversaucer.git (to revision 75932f389536b59993fa780b281170849ff92238) to ./src/silversaucer Running command git clone -q git@github.com:prcutler/silversaucer.git /home/vsts/work/1/s/src/silversaucer Host key verification failed. fatal: Could not read from remote repository.

Here is where I first lost hours over the course of a few days. Azure is pulling my repository using git, not https. I would compare this to my Azure Pipeline for NFLPool, which for some reason pulls my repository using https and works fine. I know a little bit about SSH keys. I’m no expert, but all of my Digital Ocean and servers at home use SSH key authentication to log in and not passwords (yay me for good opsec!) and I have my SSH key on multiple computers without any issues.

Lots of search queries later, I learned how you can change a git repository’s remote URL, but this appears just to be on your local machine. I’m sure I’m missing something simple to make it a global change, but I never figured it out.

Ok, let’s add my SSH key to Azure Pipelines. Again, Microsoft has good developer documentation on how to do this.

  • Step 1: Add your public key to your Azure profile.
  • Step 2: In your projects in Azure Pipelines, go to Pipelines -> Library and choose Secure files. Add your private key (usually id_rsa).
  • Step 3: Add the SSH Task to Azure Pipelines and make sure you authorize the private key - follow Microsoft’s developer documentation for the SSH Task. Update your YAML file:

Install SSH Key

Install an SSH key prior to a build or release

  • task: InstallSSHKey@0 inputs: hostName: sshPublicKey: #sshPassphrase: # Optional sshKeySecureFile:

The hostname input confused me at first, but here you’re going to go into your ~/.ssh directory and copy and paste the Github entry in your known_hosts file. (This is a hidden directory in your home folder on macOS or Linux. I’m not sure where it is on Windows, sorry!) Paste your public key in sshPublicKey: and the name of your private key that you uploaded in Step 2 above. If your repository is public on Github, you are not going to want to add your sshPassphrase to your YAML file.

My SSH key fails

Again, I lost hours here. I have no idea why, but the SSH key I’ve been using for the last couple of years will not work. I had to create a second SSH key for Azure Pipelines and delete my known_hosts file, clone the repository again (which then updated known_hosts) and paste in the new fingerprint from known_hosts to the YAML file. The new key works fine, but I’ll be damned if I can figure out why my normal key doesn’t work. I’ve compared the fingerprint of my usual key, used it to clone other repositories, but Azure Pipelines refuses to work with it as I would just receive the above error over and over again.

I’m not thrilled with having a second SSH key, and now I have to go to the other two computers I work with and copy it over and add it to my keyring. But it works.

What’s next

Coming up in Part 2: Pytest works in Azure Pipelines in Python 3.6 (but not Python 3.7!)

It was a good week of Python and me

It was a good week for Python and me.

In no particular order:

MLBPool2

I pushed a major re-write of MLBPool2 to production just days before the baseball season ended. Why didn’t I wait until the baseball season was over before pushing a major update? Because MySportsFeeds this summer upgraded their API from 1.x to 2.0 and it included a big change to how they track team standings, especially playoff standings. To give MLBPool2 players better visibility to how they were doing, it was important to get the update out there. There were a few more things pushed in that update which deserve their own write-up.

100 Days of Python

I started the 100 Days of Python training from Talk Python and Pybites. The first couple of days are training on the ``datetimemodule, which just reinforced to me how much more I like Pendulum. I haven’t been updating the daily log (or tweeting), but I have been working on Python daily.

Pyramid Documentation

I mentioned in my last blog post a few weeks ago how I’ve been learning about testing. One of the things I noticed when reading up on Pyramid and pytest was that the Pyramid documentation refers to pytest as py.test in its documentation, which is an old way of doing it.

GitHub and partners started Hacktoberfest this week, so it was a perfect time to go through all of the Pyramid documentation and update it. So I did, and it was a good learning opportunity. It’s not as simple as an easy search and replace - reading through the docs to make sure the changes are done right also teaches me about the features, and learning about pytest and especially coverage was great. Talking to the developers, Pyramid has a 1.10 release coming very soon, so I got those changes in and then dived into another discussion topic on IRC, cookiecutters.

With the upcoming 1.10 release, Pyramid will be merging three of it’s cookiecutters into one. Even though we’ll update the documentation, including the README, there are probably some developers who have a workflow to just pull the cookiecutter from the command line. I added a message to the two deprecated cookiecutters that they are deprecated and to use the correct one. I did spend some time going through the Talk Python course on cookiecutters, which was good, too. I was looking for a way to update the pre-commit hook to add the deprecation message, but no luck on that.

And speaking of Hacktoberfest, I met the goals! (Not that I’m done yet).

It was a good week.

Hacktoberfest 2018

There is No Offseason

Alternative title: There is No Offseason (or writing my Python apps never ends and that's ok)

The blogging might slow down some, but the coding never stops. Ok, well maybe a little. After creating and launching MLBPool2 this past spring, I took a small break and then back at it for the upcoming season of NFLPool.

I have two big goals for NFLPool before the 2018 season starts:

  1. Fix the time / timezone issue where a player tried to submit his picks before the first game of the season started to, but was denied. Rip out all references to Python’s datetime module and replace it with the Pendulum module instead.
  2. Allow NFLPool players to make changes to their picks if the season hasn’t started yet. This build on existing functionality in MLBPool2.

And a few smaller things to add including some updated admin functionality. I didn’t get to a few of the big things I wanted to do this offseason, which included unit tests and porting to MySQL. But it’s good to have goals for next offseason.

I’ll be writing about some of the changes over the few weeks leading up to this season’s kickoff.

  • in Python
  • 2 min read

Python Dev Kit Humble Bundle

Humble Bundle has done it again. They’ve released an awesome bundle of Python trainings, books and tools all for a low price and also supports charity!

Included in the bundle are three trainings from Talk Python (which I’ve already raved about a lot). I’ve been an ardent supporter of Talk Python trainings and have recommended them numerous times on the Python subreddits, and usually get comments that they’re so expensive (when compared to Udemy’s “sale prices” of $13). But they’re worth every penny and now people have a chance to get three of them for just $20.

But wait, there's more! Fluent Python, which I have not read, but is almost always one of the most highly recommended books for intermediate Python programmers and up is also included. Dan Bader’s Python Tricks book, Matt Harrison’s Illustrated Guide to Python 3 and Thoughtful Machine Learning by Matthew Kirk are also included. I’m personally excited to get Matt Harrison’s book - Talk Python launched a new training this week and if you buy it within the first week that book is included. I’m going to buy the Talk Python Everything Bundle in a month, so I would have missed out on the promo to get the book, but now I get it anyway!

You also can get a number of software tools. A 1 year subscription of Postman Pro is included - I’ve used the free version since Talk Python introduced it to me. It’s been fantastic for learning the MySportsFeeds API and looking through rows and rows of JSON. I’m curious to see what the Pro version brings. GitKracken looks interesting, but PyUp is one I really want to try out.

Last, but not least, is a 6 month subscription to egghead.io, featuring video tutorials to learn JavaScript. I wanted to get better at Python before learning JavaScript, but this makes starting to learn it a no-brainer.

Even though I have two of the three Talk Python trainings in the bundle, a PyCharm Pro license and am an existing DigitalOcean customer, it’s still an unbelievable deal. Go get it now! (Not an affiliate link).

  • in Python
  • 2 min read

Password Validation in NFLPool and MLBPool2

When I first learned to work with Pyramid thanks to the Talk Python course Python for Entrepreneurs, I used the account registration system directly from the course for NFLPool. When I wrote MLBPool2, I augmented it to require the user to use a much stronger password than the course taught. (You can see the original code from NFLPool here).

In MLBPool2, I required the user to use a password between 8 and 24 characters and it must have at least one lowercase letter, at least one upper case letter, a number, and a symbol:

A nice error message is shown each time the user doesn’t do it correctly. I was pretty proud of myself for figuring this out (and for using Stack Overflow to get some tips on how to do it).

But…. It hit me a couple weeks ago that it works great for user registration, but if they lose their password or want to reset it, none of that functionality is there. When resetting a password, the user clicks the reset password link which emails them a one-time link to reset their password. And none of the password requirements I was so proud of are in the methods for resetting a password.

It’s going to require a big re-write - right now it only has one input field for the new password. I’ll need to add a second field to require the user to enter the new password twice (similar to registration) and then add all of the requirements and the error messages to the viewmodel. So I’ll be working on that over the next week or so.

  • in Python
  • 3 min read

NFLPool - The Work Continues

For once in my life, I’m not procrastinating. With the NFL season just over four and a half months away, I’ve already started on working to update NFLPool.

I’m really enjoying working with Python and don’t want to let the few things I’ve learned get rusty. This includes back porting a number of updates from MLBPool2.

I've updated the Standings page to display all seasons played (before it defaulted to just the current season). This uses traversal in Pyramid to create a GET request based on the season year and automatically creates the links. This feature within Pyramid is so cool, I guess I should expand on it:

I have a standings_controller.py that creates all of the routes for the pages in Standings:

I call the StandingsService to get a list of all the seasons that exist in the database to create the index page and it shows a bullet list of each season. (Currently, this returns the one season that has been played, 2017). You can see it in action here.

You can then click on the 2017 bullet to see the player standings for the 2017 season. The code for the player standings in each season looks like:

The cool thing is season = self.request.matchdict['id'] above - when the user clicks on 2017, that’s passed to this method, that passes 2017 and makes season equal to whatever bullet the user clicked on in the list on the Standings index page, which is 2017 in this example and returns the 2017 Standings. The rest of the code passes the season variable (which is now 2017) to the database to show all of the players and their score for the 2017 season. I also updated the template that if the week is 17 or greater (there are only 17 weeks in an NFL season), the header on the page now says 2017 Final Standings, otherwise it would return 2017 Week x Standings, where x is equal to whatever week the standings have been updated through. This was helpful as last year there was an issue with how division standings were pulled from MySportsFeeds because they don’t calculate the tiebreakers in their API and they had to fix it - after they fixed it, I updated the stats from their API, but now it was “Week 18”, which technically doesn’t exist in the NFL. Now it just says final.

I also added Slack messaging that is present in MLBPool2. When a new user registers or submits their picks, I get a message in my NFLPool Slack channel.

A minor thing, but when players are selecting the NFL player they think will win in a category (such as who has the most passing yards), the dropdown box that shows a list of all NFL players now includes the player’s team and position in addition to the player’s name. Another small thing is the site administrator can update when a player has paid the annual league fee and when the admin updates to a new season the list players who have paid is reset.

Last, but not least, I’ve been working on documentation. I had already written the User Guide on how to play and make your picks, and over the last week I’ve added an Administrator’s Guide, including how to install, first time setup, how to update to a new season and how to manage NFLPool players. This is all written in reStructured Text and uses Sphinx to create the documentation and is hosted on ReadTheDocs. I still need to write the developer documentation - I have no idea how to write developer docs and am still looking for some good examples of developer documentation to crib from.

It’s a good thing I’m using Github’s issue tracker - a couple of those fixes above I had added last September after NFLPool launched and probably would have forgotten about. Between that and the documentation, it’s almost like I’m running a real open source project!

The work doesn’t end there, though. The big one I need to add the TimeService and GamedayService to make datetime management easier, especially for calculating when picks are due and displaying it on the pick submission page. I also have a couple of more bug fixes to get to and I’d like to add Twitter support to the app as well. The journey never ends.

  • in Blog
  • 2 min read

Goodbye Wordpress, Hello Hugo

I’ve been using Wordpressfor over 15 years. In fact, I started blogging before Wordpress existed, and even used b2 / cafelog, from which Wordpress was forked.

But over the last couple of years as I’ve maintained three separate Wordpress sites (the first for paulcutler.org, then Stone Open, and later MLBPool2 (which is now using Pyramid instead), I’ve had constant crashes and memory issues. I don’t know if it’s because of the number of blog posts I have, but both my personal blog and Stone Open have tables corrupted and I have to go in and fix them whenever the site crashes.

That, combined with the constant updates for Wordpress and it’s plugins, as well as security fixes, made me say enough is enough.

I’ve been looking at Hugo (written in Go) for the last month or so to replace Wordpress with. I also looked briefly looked at Jekyll and Pelican, which are also both static site generators. Pelican interested me, as it is written in Python, but Hugo seems to be a bit more actively maintained.

As of this morning, paulcutler.org is now running Hugo. I updated the web server, installed a new SSL certificate (thanks Let’s Encrypt!) and moved it to my other DigitalOcean droplet that serves MLBPool2.com and NFLPool.xyz.

One of the advantages of using Hugo is that all of the blog posts are written with Markdown. I’ve been using Markdown more and more, as most of the daily apps I use on macOS use Markdown (Day One, Bear Notes, and Ulysses (which I’m using to write this now with).

I’m a little worried I’m going to lose some Google search magic for the old URLs, but we’ll see. I also need to see if this theme I’m using has search functionality to use. The site should be faster now as there is no database in the middle and I’m just serving static HTML pages. I’m sure I’ll continue to tweak it over time as I learn more about Hugo. Now to get automated deployment working!

  • in Python
  • 3 min read

MLBPool2 – Letting a Player Change their Picks

I’ve been blogging a little bit about MLBPool2 the last couple of weeks and now the last three months of work is complete.

I already touched on two of the biggest differences between NFLPool and MLBPool2 (the time service using Pendulum and using MySQL / MariaDB instead of SQLite).

The biggest difference between NFLPool and MLBPool2 though is players have the ability to change their picks. At the All-Star Break, MLBPool2 players can change up to 14 of their 37 picks, but those changes are only worth half points.

This required a major re-write in the way I capture and store each player’s picks. I also figured, based on how NFLPool went when I launched it, if a player was going to be able to change their picks at the All-Star Break, I might as well let them change their picks before the season starts. (I had a couple NFLPool players request to make a change, which required me to delete all their picks from the database and I just told them to do it again. But I hate touching the database.). I wrote a new service (gameday_service.py) that the app uses to figure out a few different things:

  • Season start
  • When picks are due
  • The All-Star Break (48 hours before and after the All-Star Game)
  • Season end

I already was using two methods and one for the initial picks submission and one for changes in the PlayerPicks service. If the player was changing their picks, I used an if / else statement that compared the current time to when the season started:

now_time = TimeService.get_time()

if GameDayService.season_opener_date() > now_time:
"""Update the picks passed from change-picks. If the season start date is later than the current time,
make the new changed picks equal to the original picks, making original_pick equal to the new pick."""

Update Pick Type 1
Update the AL East Winner Pick - check to see if it has been changed
if al_east_winner_pick != session.query(PlayerPicks).filter(PlayerPicks.user_id == user_id) \
.filter(PlayerPicks.season == season) \
.filter(PlayerPicks.pick_type == 1) \
.filter(PlayerPicks.rank == 1) \
.filter(PlayerPicks.league_id == 0) \
.filter(PlayerPicks.division_id == 1):
session.query(PlayerPicks).filter(PlayerPicks.user_id == user_id).filter(PlayerPicks.pick_type == 1) \
.filter(PlayerPicks.season == season) \
.filter(PlayerPicks.rank == 1) \
.filter(PlayerPicks.league_id == 0) \
.filter(PlayerPicks.division_id == 1) \
.update({"team_id": al_east_winner_pick, "date_submitted": now_time,
"original_pick": al_east_winner_pick})

And do a session.update to update the database.

But if it’s during the All-Star Break we need to capture that the pick has been changed (changing the value in the PlayerPicks table from 0 to 1) where the UniquePicks service will credit the player with half points:

else:
"""If the season has started, update picks at the All-Star Break. Do not change the original pick column
and update the changed column to 1."""

Update the AL East Winner Pick
for pick in session.query(PlayerPicks.team_id).filter(PlayerPicks.user_id == user_id) \
.filter(PlayerPicks.season == season) \
.filter(PlayerPicks.pick_type == 1) \
.filter(PlayerPicks.rank == 1) \
.filter(PlayerPicks.league_id == 0) \
.filter(PlayerPicks.division_id == 1).first():

if pick != int(al_east_winner_pick):
session.query(PlayerPicks).filter(PlayerPicks.user_id == user_id) \
.filter(PlayerPicks.pick_type == 1) \
.filter(PlayerPicks.rank == 1).filter(PlayerPicks.league_id == 0) \
.filter(PlayerPicks.division_id == 1) \
.update({"team_id": al_east_winner_pick, "date_submitted": now_time, "changed": 1, "multiplier": 1})

When the change picks form is submitted, all 37 picks are sent from the form to the POST and viewmodel. In the controller, I checked if the number of changes was greater than 14 and would redirect them to an error page. But if there were 14 or less changes, the above code would run. If the new pick was not equal to the pick stored in the database, we’d update the pick, capture the time the player submitted changes and flag that the pick was changed. The code works, but there are 36 blocks like the one above and I couldn’t figure out a method that would work where I could pass it parameters. (36, not 37 as the tiebreaker pick can never be changed). But the key is that it works.

The multiplier is reset to 1 and if their original pick had qualified for the double points bonus, it needs to reset back to one as the UniquePicks service is re-run after the All-Star Break and then will assign a multiplier of 2 if the new pick qualifies for the double points bonus.

I plan on porting the code to allow a player to change their picks before the season starts to NFLPool to make the player’s life easier. But really, it’s to make mine easier as I don’t want to have to manually delete their picks from the database.

  • in Python
  • 2 min read

MLBPool2 is live!

I deployed MLBPool2 on Monday. I had a flurry of activity over the weekend to fix a scoring bug. No matter how hard I tried, I couldn’t get MLBPool2 to match the 2017 results that were done by hand. I learned that I don’t have the patience to hand enter the picks for 16 players and then all of their All-Star Break changes. I did it three times and every time I would catch a mistake that I made. And with the way the UniquePicks service works, one mistake is all it takes to throw off the entire score. I also decided to add the ability to add an administrator and let him or her update the database if a player had paid the league fee.

Once that was done, it was time to test my mediocre (at best) sysadmin skills. Adding the server blocks to nginx was easier than expected and it took me a half hour to figure out why the app wasn’t loading – when using nginx with Waitress for the wsgi server, the app name has to be an exact match. (Note to self: The app name is mlbpool, not mlbpool2).

I changed the DNS to point from one DigitalOcean droplet to another and Let’s Encrypt made it a breeze to get a new SSL certificate. (Of course https is served by default. I haven’t been a member of the EFF for 14 years for nothing).

The database hasn’t crashed yet, so I’m hopeful all my session.close() statements are where they should be, though Bing’s webcrawler apparently crawls by IP and not domain and I woke up to a Rollbar notification of an error thanks to Bing looking for a URL on NFLPool.

It’s been fun having Slack notifications for when a new user registers and makes their picks and seeing those roll in.

But the work doesn’t stop there. Yesterday I wrote user documentation using reStructuredText and Sphinx (yay, another markup language to learn. This is only the fourth language I’ve used to write user help.). I created an account on Read the Docs, connected my Github repo, and voila! User documentation for MLBPool2 on how to create and manage an account, now to submit and change picks, league rules and scoring. This was all just a precursor to what I really want to do, which is write the developer documentation.

I also started working on tests for MLBPool2. I’m still so confused on testing, specifically what tests I need to right to improve code coverage. But I successfully wrote my first test using pytest today – just a small one to check the MySportsFeed API and make sure it’s up and returns a 200 status code. Baby steps.

I’ve already filed a bug and a couple of enhancements that I want to work on once the season is over. When I set out to learn Python to build these apps, I did it because I wanted a hobby, and I sure have one now.

  • in Python
  • 3 min read

Where I get stats for MLBPool and NFLPool: MySportsFeeds (and it’s awesome)

A few years ago I started to look into how I could build apps to manage MLBPool and NFLPool. The key would be how to integrate all of the team and player statistics and where to get that data. I was floored when I saw the pricing of how much companies charge to provide those stats – it was hundreds to thousands of dollars per month to get access to baseball or football stats. The closer you got to real time statistics, the more it was. Most of these companies are providing statistics to commercial services that run fantasy leagues (I’m guessing), which is why fantasy leagues charge a fee to host your fantasy league.

It’s been a couple years now and I don’t remember how I came across MySportsFeeds, but they offer a commercial service for companies like I mentioned above, but they also have a key differentiator. For educational purposes, developers or research, they offer a free service to access stats for completed seaons. Best of all, you can also subscribe for a very low price to their Patreon to get access to live data. I’m happy to say I was one of the first Patreon subscribers and for $5 / month I get access with a 3 minute delay. (I really only need data overnight and not real time for my apps, but what an awesome price). MySportsFeeds currently offers statistics for the NHL, NBA, MLB and NFL and statistics are available in JSON, XML or CSV.

There are a few different things I love. One, there is a Slack channel where the owner and lead developer, Brad Barkhouse, helps out. He’s extremely responsive to community questions and is always around. Two, the service is always getting better. Last summer they launched wrapper libraries for popular programming languages including Python, Ruby, R, NodeJS and more. Three, they have fantastic documentation that includes all the parameters you can pass to the different feeds to help filter what information or statistics you might need.

There are a couple quirks. For NFLPool, the Team Standings feeds don’t account for tiebreakers. I can’t fault them for that as the NFL tiebreaker calculations can be complex. After the NFL season ended, I pinged Mr. Barkhouse and he quickly updated the feed to match the NFL standings, which I needed for my app.

In baseball, I started to enter all of the 2017 MLBPool picks for testing. I need to make sure that the app works and matches the scoring that was done by hand last year. When entering picks, one player had chosen Yu Darvish for one of the pitching categories. When I went to make the pick in MLBPool, he wasn’t available as an American League pitcher. MySportsFeeds showed him on the Los Angeles Dodgers – but he wasn’t traded from the Rangers to the Dodgers until July 31, 2017. Brad will fix his roster information, but MySportsFeeds uses a cumulative player statistics so the feed shows Darvish’s stats for the entire year. But in Major League Baseball by rule, a player’s stats when traded between leagues are NOT cumulative. This is obviously an edge case for MySportsFeeds, but something I’m going to have to account for before MLBPool launches. (I currently store all baseball player statistics in one table – I’m going to need to split this and have two tables, one for the American League and one for the National League to account for this).

MySportsFeeds is under constant development and always improving. Mr. Barkhouse and team updated the API last year from version 1.1 to 1.2 and work is underway for 2.0. They added Daily Fantasy data last year. Users can also file issues in Github or ping him in Slack to get items added to the roadmap. Overall I am extremely happy with the service and highly recommend MySportsFeeds.