Category: Uncategorized

Working on the User page is a bit of a pain in the backside.  For those of you who are new here, let me explain. After you create an account and log in, the Log Out button appears with your (Extended Stats) user name underneath it. This works extremely unreliably! I’m *still* debugging that.

If you click on that user name, you get taken to the user page, which contains the user widget, which is the bit that is supposed to let you choose your BGG user name and make geek buddy lists. It also shows you the information that Extended Stats stores about you, which actually does work, even it looks like a lot of waffle (so far).

Now I spent quite a bit of time before I got addicted to Assassin’s Creed: Origins making the user widget work. In particular I built an Express server and coded the autocomplete functionality in that and deployed it with Elastic Beanstalk and then undeployed it from Elastic Beanstalk and just stuck it on an EC2… and now Chrome says that as the user page was loaded using HTTPS (secure), it can’t call the Express server on the EC2 using HTTP (insecure) as that would break the security. It is correct to do that. So today I needed to add a security certificate to the EC2, but with AWS you just can’t do that. There is no way.

So what I figured out I could do is I could configure api.drfriendless.com, which does have HTTPS, to direct calls to the autocomplete code on the EC2. And then I need to change the user widget to use https://api.drfriendless.com instead of http://eb.drfriendless.com.

As if all of this isn’t a complete pain in the bum enough, extstats.drfriendless.com is served by the CDN, Cloudfront, so it caches all of its resources. I am supposed to be able to tell it to dump particular resources from the cache, but that hasn’t been working for me. It used to, but now I must be doin’ it wrong. So after I change all of those things, I have to wait a day for the changes to take effect. This is a seriously painful development process.

Of course, I need a test server to try this out on. I do have test.drfriendless.com. The test server doesn’t use CloudFront, which means that it doesn’t have HTTPS so it won’t have the problem anyway. But also, because it doesn’t have HTTPS, auth0 (which manages the logins and passwords) won’t talk to it. So in this particular case, the test server cannot possibly work.

Ah well, enough bitching. I got two new patrons yesterday, for a total of 4. Four wonderful patrons! Muahahahahaa! At this rate I’ll be rich about when I get the user page working properly.

 

This Christmas season, I am being, as usual, as unChristmassy as I possibly can. Quite apart from work never actually stopping for me, and this being a busy time of year, I’m a grumpy old man. However as this blog is not about my misanthropy, let me tell you about the code I’ve been writing.

On December 1 (oh my goodness, so long ago) I showed off the component I had got going which allowed you to enter a list of geeks. I also complained about Angular Material. Since then I’ve been working on the user data page, and maintaining the love-hate relationship with Angular Material. Mostly hate.

The user data page does two things – it allows you to edit data attached to your login on Extended Stats, and it tells you what that data is. The first is obvious, the second is my interpretation of a GDPR requirement. It might seem silly, but I kinda believe in what GDPR is trying to achieve and want to build accountability in from the ground up.

Now, editing the user data. The first thing you can edit is your BGG user name. As anyone can see the data on Extended Stats relating to your BGG user name, even without being logged in, you don’t need to fill that in. However my intention is that when you’re logged in, I’ll give you hyperlinks to the pages for that BGG user. And in fact, you can have multiple BGG user names, which will be convenient if you maintain stats on behalf of your spouse or your board game group.

Then there’s buddy groups. A few years ago I made this graph of new games played over time for my local gaming group. And I want so badly to be able to provide this graph to all of you!

So the plan is that this sort of graph will apply to a group of geek buddies. And to save you entering that group of geek buddies every time, I will store it in your account. And if you want to do it for your other gaming group as well, you can create a different buddy group.

Sadly, the work on the user page is still not complete. However most of the wrangling with Angular Material seems to be over, and I have to connect the results to the database. As the authentication system refuses to work with the test system, I need to test that out on the real system, which is kind of painful. So, work continues at its usual crawl.

I was away last weekend on Family Business, so I just let the site run, with the beanstalk going. The bill for November arrived, showing a dramatic increase in EC2 (virtual server) costs.

I’ve been very slack about posting about costs (because it’s so boring), so I made a (quite bad) chart of the components of the costs over time (the last 5 months). In July, the Cloudwatch  costs were high, because I was doing too much logging, and I fixed that. In September and October the Lambda costs were high, so I brought in Elastic Beanstalk to fix that. In November the EC2 costs were high because of the Elastic Beanstalk, so today I tried to fix that.

The reason Elastic Beanstalk is expensive is because it adds another server – so now I have two servers, one for Express and one for the blog – and that has a certain fixed cost. But then there’s another machine called the load balancer, which decides which of the Express servers will handle the request. In my scenario, the load balancer seems like an unnecessary luxury, given there is only one Express server. So at $8 for the month, that had to go. The virtual machine that Express runs on was more expensive than it needed to be, too, so I decided to make that an even smaller one. Those changes should save more than $10 / month.

Of course it wasn’t that easy. One of the nice things that Elastic Beanstalk does for you is create the server environment that the application runs in, so I had to redo that bit myself. So today I learn how to create an Application Machine Image,  how to deploy my Express application onto a server, and how to run an Express server with permissions to use port 80. The last bit was much harder than it deserved to be.

So apart from this recurring trauma, the site is going quite well.

The dithering of the updates has really evened out the load on the database, so now the database WHICH IS COSTING ME $30 / month looks like it might actually be able to do its job for a while. The AWS database service seems like extraordinarily poor value. On the other hand, it does backups and stuff for me.

The Lambda performance issues seem to be fixed for the moment as well.

It looks like it might be time to write some functionality.

So after arguing with Angular Material for most of the day, to the great disgust of my dog, I got this going:

It’s for editing your geek buddy list. Angular Material looks quite nice, when I can get it to work. I also quite like this idea of integrating demos into the blog posts. On the other hand, the demos are useless if I don’t then go put them into something on the site.

Well, after a week where everything was broken, I’ve made some progress. First of all, the big laptop got repaired and is better now. That means I could get the autocomplete demo from it, and then I was able to fix the bug in the web site deployment script (by abandoning the bit that mysteriously broke). That means the autocomplete demo is working, and here it is. Type someone’s geek name in:

That’s running against the Elastic Beanstalk server that I was waffling about a couple of weeks ago. So that’s all good. Next I have to remember why I was doing that.

Now I have also been working on the downloader performance, and there’s some good news about that too. I found a couple of places where I could do multiple database inserts in one transaction, and that has made a great difference. Here are some graphs of database performance over the last 2 weeks:

In the bottom right graph, notice that the green line has stopped bottoming out. That causes no more spikes in the bottom left graph, which is good because those are essentially site breakages. And the top two graphs show how the load on the database has spread out a bit.

This change in behaviour is also obvious in the Lambda performance graphs.

The nasty spikes in the top graph have gone, which is good because I think those are the ones which cost me money. In the bottom graph, the load seems to be continuing to flatten out, which is also a good thing.

OK, now I’ve solved those problems I really should get back to doing whatever the site was intended to do.

Well, I had a bit of a disaster during the week. On Thursday morning my laptop stopped charging, and now it’s in the shop being repaired for a hefty fee. It’ll be back some time during the week.

Luckily I have this other, smaller, laptop, so I can still do my work. It only took 7 hours to update it with all the work stuff… Well I guess that’s quicker than many other problems could have been resolved.

The bad news is that this laptop seems to be too underpowered to work on Extended Stats, which is not that big of a project, but it is a kinda complex one – it includes about a dozen Angular applications, and a couple of dozen Lambdas in a variety of locations, and I think the editor gets confused by the non-standard organisation.

Even worse, one of the bits of code that’s on the broken laptop is the autocomplete demo, which is one reasonably small thing that I could potentially have worked on on the smaller laptop. So that sucks as well.

Nevertheless, there is something to talk about. The work I did last weekend on putting the code in Elastic Beanstalk was aimed at preventing the really spiky bits in the Lambda performance graph (because they cost me money). Over the last couple of days we’ve been going through a period which should be spiky, and it has definitely changed.

By the way, these bits correspond to the update of all the users’ collections and played games, which happen every three days. I did two things.

First of all, I rewrote the bit that was appearing light blue in the bottom graph, because as explained in this post:

Bugs Bugs Bugs!

it was costing me a lot of money.

The second thing I did was to change “every 3 days” to “about every 3 days”. So when I schedule an update, I don’t schedule it for exactly 3 days, because all that does is preserve the periodic load. And as you can see, that flattened out the load a bit, and should continue to do so. I was calling this “dithering” to myself, but now checking on Wikipedia, dithering may be something more specific than that. Nevertheless, that plan is sort of working.

On the other hand, as Tevye would say, it didn’t seem to really help the database performance. The orange line is how hard the database is working, and making that change seemed to spread the peak out, but increase its total area. And then that caused the green line to hit the bottom for longer. And that’s bad. I’m hoping that with more dithering the green line might not get to the bottom at all.

I miss my big laptop. You don’t know what you’ve got till it’s gone.

As it’s Monday morning and I’m supposed to be going to work, this one will be short. I spent yesterday working on the autocomplete functionality, and after a great deal of mucking with Express security, I got it working. It looks lovely! However then when I went to put it on the site, the application I use to copy all of my HTML and so on up to AWS just refused to do anything. Computer says no.

That’s  the “fun” and frustration of this project. The tech is pretty much bleeding edge, so stuff just fails quite often. My test site, http://test.drfriendless.com, is completely broken at the moment!

I penetrated the impenetrable twaddle! After getting very frustrated last week with AWS security, I did some research, and finally found a description of a few things that I could understand. I now basically understand what a virtual private cloud is, and sort of how it works. That was really awesome news on Monday. Then on Tuesday we had some system problems at work and I got distracted sorting out other (AWS) stuff.

I was finally able to get back to the Express on Elastic Beanstalk project today. As I now understood the VPC stuff, I configured EB to use the VPC I already had. And then I commanded “Work now!” And behold, it did not.

I screwed around with it for a few hours, until I came to the conclusion that the reason it wasn’t working was because it thought it wasn’t working. No really, that makes sense.

Elastic Beanstalk is designed for implementing a web site (which looks like one computer) by using a bunch of computers in the background. If one of those bunch of computers goes bad for some reason, EB throws it out and replaces it with a new one. So to do this, EB has to know how to tell whether one of those computers is bad. And for a long time, I had that wrong.

However even after sorting that out, I still can’t get it to work. I’m beginning to suspect it’s a networking problem again – although Express “works” when I run it on my own machine, it doesn’t work when I connect to either the load balancer (the part of EB which farms work off to the worker machines) nor when I try to connect to the individual machines. My recollection from how we do this at work is that *should* work. So I’m back trying to figure out networking. And since I have known that stuff since Monday, I thought I got it…

I’ve also been considering whether I should actually get back to working on functionality (i.e. pretty things) instead of this AWS navel-gazing. But I should not. The bill for October (which I still have not blogged about) was a bit high, and I need to get this project working to cut down the costs. While it’s costing me too much, this project is at risk, so I need to get that stuff sorted.

The good news is that I was plagued by problems like this for about a year while I was trying to get this project off the ground, and with perseverance and experience I managed to figure them out. I feel like I”m making progress, even if it’s just developing a callus where I’m bashing my head on the wall.

I’m struggling to get back into my motivated groove for Extended Stats since all the holiday stuff. Although my wife is away and hence is unable to distract me, the dog is not and she demands an inordinate amount of attention. I’m also doing a lot of cooking of things that I can’t make for my vegetarian wife. This evening’s nasi goreng was pretty good if I do say so myself. But not much coding got done.

Today’s project was the top-right sticky note, which says “Express & EB”. Express is node.js software for writing a web server, and EB is Elastic Beanstalk, an AWS technology for scaling web servers – essentially you tell it “Here’s my code, put it on a machine. If it gets busy start some more machines running the same code.” I use EB at work, but I’ve never used Express before.

I’ve been getting the feeling that AWS Lambda is a little expensive for some of the things I want to use it for.  In particular, the note below “Express & EB” which says “geek buddies”. The plan for that note is that if you’re logged in you’ll be able to tell the site which other users are in your game group / family / other peer group. And the plan for that involves an autocomplete field where you start typing a BGG user’s name, and I give you all the valid completions for what you’re typing.

That’s easy enough to code, but it usually involves one HTTP call for each character typed, and as I pay for each HTTP call to a Lambda, I’m not very keen on that. So the plan is to make an Express server which can handle very tiny calls like that for a pretty much flat rate of a couple of bucks a month. That should decrease my Lambda costs (which I should also write a post about).

However, today’s plan to run Express on Elastic Beanstalk foundered at the bit where I tell Elastic Beanstalk about the database. Omigod, AWS security is basically impenetrable twaddle. At least, it is to me. At work we tell the servers about the database a different way which I don’t want to use here, so I would like to configure the database in EB. I ended up with EB and the database both in virtual private clouds, but in different virtual private clouds. Is that good? I don’t know. It was like when you can’t find two socks the same colour. So in the end I got cranky and dumped the EBs and will try to do some reading so I can get half a clue for the next attempt.

In happier news, I did manage to fix a big. BGG started sending my ratings for games people hadn’t rated is “N/A” instead of just completely absent, so I was trying to store “N/A” as a number which broke stuff. Thanks to a new user for pointing out to me that something was broken.

The next bug I found was that if you have many many plays in a month, for example 3750, BGG tells me to stop asking for so many files all the time. As the Lambda wants to get its job done quickly because I’m being billed for the time it runs for, I can’t just sit around and do nothing. So I need to come up with a plan for dealing with many pages of plays over several lambda invocations. Something nice-ish will spring to mind eventually, I suppose. That technique could possibly also be adapted for users with very large (BGG) collections, e.g. 100000 as I have seen. Where’s my thinking cap? I need my thinking cap.

 

Hello, I’m back! I spent two weeks gallivanting around the world with my wife (who was working), and then I spent one more weekend playing Ingress in Canberra. And now finally after all the unpacking is done and the clothes washing is finished, I can get down to some programming. I last saw my wife in Dubai and don’t expect her home for a while yet, so I hope for minimal distractions.

So while I was away the system was running happily by itself, by which I mean it was unattended and nobody much was looking at the site 🙁 so really all it was doing was costing me money. The last bill arrived while I was in Italy so I haven’t looked at it in detail yet, but I did investigate why the Lambdas were still costing me more money than I expected.

I found a new way of arranging the Lambda usage graphs that makes more sense (and that’s what the pretty picture above is). It’s pretty clear that blue was doing way too much work, so I investigated that. That’s the bit which takes the list of what games someone has in their collection and puts it into the database. One user had many versions of games in their collection, so I kept trying to put the same game into the database over and over. I’ve sort of fixed that (but what can I do if someone gives the same game two different ratings?) and that particular problem seems to be fixed. In a week or two I’ll know how that has affected the performance (and hence cost) of the system.

The next bug I addressed was one that was found by alert reader Jeroen, who pointed out that if someone recorded more than 100 plays in a month, I only used the first 100. I knew what that was, so I investigated and fixed it. Then came the hard bit – fixing all the data that was wrong. Even finding it took a while – essentially I had to find all the months for all the geeks where there were 100 plays recorded. There were 2750 of those, so I then had to mark those as needing to be reprocessed. That was quite difficult to figure out. Ideally I would have just said “reprocess all the months”, but I know that that takes more than a month to do, so I had to get exactly those ones. I seem to have achieved it, but after all that hard work, the database was a bit puffed out and needed a rest. In technical speak, I exceeded my CPU quota and the burst balance was very quickly heading to zero as well. So now that I’ve figured out all of the months to fix, the database is too tired to do it and I’ve told the downloader to rest. I’ll turn it back on in the morning.

So one of the good things about my wife being away is that I can now put sticky notes of things I need to do all over the house. There’s a rather nice Japanese flower picture in the living room here which is now covered in notes saying things like “geek buddies” and “API keys”. Actual user-facing features haven’t even made it to the sticky notes yet. However I shall be extremely pleased if I can get these things done in a couple of weeks.