One Step Back, Two Steps Forward

I’m so far behind that I have to blog about what happened last weekend. My wife was away so I had a bit of time to myself which I planned to use doing some work on the site, instead of acting like a human being and having a life.

The first plan was to fix the Login button on the front page. What happens is that I display both a Login and a Logout button, then I make a call to the server to find out whether you are logged in or not, and then I remove the irrelevant button. That was working, but it was very slow due to Lambda startup time, so the two buttons both displayed for a disturbingly long time. So I wanted to move the API for whether you’re logged in or not into the “Elastic Beanstalk” server. I put that in quotes because although I still call it that, Elastic Beanstalk is not involved at all. Elastic Beanstalk is a service which would start another one of those servers if the first got overwhelmed. I had that working until I discovered that Elastic Beanstalk costs extra money, and as it was unnecessary I took it out.

Anyway, I moved that service as planned, and it didn’t work. It turns out that API Gateway – the bit where I say which server or Lambda each URL goes to – has very firm poorly documented opinions about HTTP headers transiting through itself. And since the cookie which says whether you are logged in goes in a header, it didn’t want to send it. In fact I still don’t have that working. And as much as I understand how HTTP and CORS and API Gateway work, I’m not sure how it is ever going to work, although it seems very much like the sort of thing that should. So after many many hours, that problem got put on the backburner.

The check for whether you’re logged in or not is a bit quicker, but it always says you’re not logged in, because the cookie is never sent to the server. I’ll figure it out.

The other thing I worked on last weekend was also supposed to be easy. It was the Plays of Games Owned by Published Year graph, which looks like this:

This graph goes on the Owned page, which I hadn’t touched in a while, so I started by updating the libraries I use to make it. Well, that was a mistake. For four hours I juggled library versions, checked commit logs, and researched why the heck I was getting the wrong version of TypeScript. OK, the wrong version of TypeScript was totally my fault, but the rest of it was not and it was annoying.

But then I got the page running again, and I could add the new graph. That just involved confusion about how Vega allocates colours to data points, and a few things being upside down and the wrong size and so on, as happens in this sort of work. And then it was finished and I sent it out to the site and dragged my sorry backside to bed. It really does get a bit wearing, doing things that don’t work, all the time.

But of course this morning I was optimistic and naive again, and at 7am I started adding another feature to the Owned page. This time I didn’t need to upgrade any of the libraries, but I did discover that some of the data I wanted to display, in particular whether you had a game for trade, was not available in the data set I was using, so I had to fix that. And then I decided that rather than improving the REST data set, I should use the GraphQL data set and improve that. So I ended up creating a new version of extstats-core, a new version of extstats-angular, a new release of the API, and converted the whole page over to GraphQL. And then, only 6 hours after I started (some of which were spent on domestic duties), I got the “Why Do You Even Own This?” table working on the new site.

On the whole though, today’s mucking around was much more productive than last weekend’s. GraphQL is kinda lovely, and much easier to work with than REST, and the changes I made today should be useful for all of the other pages as well. I think about 4 of them out of 9 or 10 use GraphQL so far. However I think if I do some more work this weekend I’ll probably just try to get some features out, rather than mucking with stuff.

Lift and Shift

As mentioned in the previous post, the old Extended Stats server blew a foofoo valve and needs parts. The fan I ordered on EBay arrived and when I went to install it I discovered that I’d ordered the wrong size. This is one reason I hate hardware. Although I love the details of software, the details of hardware bore me senseless. Why can’t they just make it out of Lego so I can hack something together rather than having to figure out exactly what thing I want?

Anyway, I ordered another fan and it didn’t arrive yet. In the mean time the idea of doing a lift and shift continued to grow in my mind. A “lift and shift” in cloud computing is when you take an existing service and dump it in the cloud without adapting it to cloud architecture at all. It’s kind of the worst, most expensive, way to get into the cloud, but it’s also the quickest.

Another of the things I use the site for (other than being there for you guys) is experimenting with stuff. There’s a type of database that I’d like to try at work, which would be a huge risk for the company, but I can muck with Extended Stats for a few hours without anyone losing too much money. I mentioned the idea of experimenting on the stats site to my boss, and I suspect I might be able to get him to throw me some funding… so yeah, that’s gonna happen.

So as you might guess I did the lift and shift. It took a few hours, spread over two weekends, as as usual everything in Django seems to have changed between releases. And then I had an annoying problem where the web server couldn’t find the code, even though everything was configured perfectly, and that took me a good 3 hours to figure out – my home directory had permissions which prevented the web server from seeing the code! Late last night I got that sorted, and the new site started to work.

Then this morning I spent another hour or so getting the downloader to work. And then, in a small brain explosion, I told git (the version control system) to undo all of my changes… so then I had to go and fix all the things again, which was not at all what I had planned.

But now it seems that http://stats3.drfriendless.com is working OK. You might notice that there’s an S missing from HTTP. That’s another problem for me to sort out! I’ve come to realise that I don’t particularly enjoy solving AWS configuration problems, so I’m not really looking forward to that one. I could do it with a load balancer (which is expensive) or with a CloudFront CDN (which is annoying to configure but I suspect I’m getting OK at it by now). Or I could try to do SSL termination in Apache – actually, nah, that’s the sort of thing that will just make more work for me later.

Well, that’s how I spend my weekends! I’ve been hearing cries from the crowd asking for utilisation numbers on the new site, and I have no idea why I haven’t done them yet. So I might try to do some programming and get those happening. I like programming, it’s better than configuring web sites.