Over summer, I was working towards a particular goal. I mean the Australian summer, so that was more than 6 months ago now. My plan was to put into the new site a feature that I had mocked up on the old site 2 years before. However even then the old site was becoming difficult to work on.
I actually implemented the feature with the server in Kotlin, and the web page in AngularJS. AngularJS has since died, and I am now using Angular 7 (which is sooooo much nicer!). Kotlin is a language I do love, but the architecture of the server was so 2015, so that code could not continue to live in the serverless world. And then when I did start writing serverless code, TypeScript on Node.js seemed like a better choice.
So anyway, what did I do? Well, it’s just the Rate of Play of Games New to You, for multiple users (which the page could already do), but now it’s EASY TO USE! So people can actually see that the page can do that.
I tried to add a few more people to the list, but I ran into a problem I haven’t encountered before – there was too much data for the Lambda to return! It seems there’s a 6 megabyte limit. After I return that data, it gets compressed before sending it back to the browser, to only about 10% of the original size, but the limit is before compression.
To tell the truth I’m a little nervous about the amount of data I send back, as it costs me money, so maybe it’s worth some work to make that data smaller.
This new feature is on the test system and is coming to the live system within 24 hours.
One of my favourite features on the old site is the Plays of Games Owned Graph. I’ve been intending to put it on the new site for a long time, but have finally achieved it! As far as I can recall it has all the same features as the old one, including the click-through to BGG.
This graph is now on the Owned page. It’s available now on test.drfriendless.com, and tomorrow on extstats.drfriendless.com.
When I started writing the user interface parts of this project, I knew I would need a table component. Angular has quite a few libraries, but I couldn’t find a table component that was useful and that I could get to work… well until the one that I ended up with and used for the War Table and the Rankings Table. It wasn’t great, but it sort of worked and got the data out there.
However I always aspired to something nicer – I aspire to make the whole site better, but I have to do it bit by bit! So I prepared to make the table component nicer by reading the source code, but at first I didn’t understand it. It didn’t do complicated stuff, it just used Angular features that I didn’t know existed. This is what I get for being a newb! However today I managed to get myself sorted to make an improved version, and I’m rather pleased how it came out.
The War Table, updated
I moved the pagination stuff to the top of the table. If it’s at the bottom, you may not see it to use it. I also got the CSS under control so that the pagination bar looks like a natural extension to the table. I also took control of the page numbers you could go to so that it’s easier to scroll through the pages with some sort of intent.
And with all that extra space I had available, I added a Search field. One thing that I did not like about those tables was that it was really hard to find yourself – nobody wants to scroll through lots of pages searching for something. So if you type a geek name into the search field, it will take you to the page with that name on. How cool is that! It makes all the data behind that table much more accessible.
I guess tomorrow’s job will be to fix the rankings table, because IMHO this design is too sweet to ignore.
Of course the Plays page which I was touting last week failed miserably. It complains about the base setting being incorrect. I suppose it’s telling the truth, but it’s a bit annoying to find these things out in production. So I’ve deployed a new version and due to the CDN, you might see that new version tomorrow. In the meantime, you can play with the test site with URLs like this: http://test.drfriendless.com/plays.html?geeks=Friendless,karlsen,jmdsplotter which show off the colours and the multiple geeks.
Another thing I managed to sort of achieve today was compression. A graph such as the one above retrieves a lot of data from the database – maybe 2 megabytes. Sooner or later someone has to pay for that data transfer, and someone is gonna be me, so I wanted to cut that down. Luckily, web servers and web browsers have this agreement that the browser can ask for the data to be gzipped, and if the server can do it will. One of the things AWS’s API Gateway gives me for free (for which I forgive it for being a massive pain in the arse other times) is that I can just tell it so send data compressed, if the web browser asks for it.
So I changed my code to ask for the data to be compressed, and Angular (the JavaScript framework I use to write the pages) said “NO! You are not allowed to ask for that!” Apparently it’s a sin to be running around asking for stuff to be compressed willy-nilly. And the word on the street was that the browser does it automatically.
Which it plainly was not doing. So I had a think and a search and another think, and remembered that as the request for the data was coming from extstats.drfriendless.com and going to api.drfriendless.com – a totally different place – the two ends don’t trust each other. So extstats.drfriendless.com could only ask for compressed data if it first got permission from api.drfriendless.com to do so. This is a thing called CORS, but don’t Google it, it’s horrendously complicated annoying crap that just prevents things from working. For example, it prevents sites from stealing your bank logins and that sort of stuff. But after I configured CORS, the browser was allowed to ask for the data to be compressed, and the server was configured to do it, and hey presto 90% data compression.
Although the site doesn’t have a lot of the functionality I want on it, yet, I feel that mechanically it’s approaching stability enough that I can start inviting lots of people to use it. They’ll be disappointed I expect, but I will use their misery as motivation.
The old site annoys me more and more each time I look at it. During the week we had a technician come to convert our internet connection from “cable in the house connected to cable in the street” to “cable in the house connected to fibre in the street”. That may make the old site run a little faster, but probably not a whole lot as that site’s big problems are the database architecture and the wifi link from the machine to the home network. But I had to spend an hour or two of misery trying to understand networking and Ubuntu config things that are just not problems with the new site. I think I’ll be pleased when I can close that site down.
Last time I blogged I mentioned how frustrated I was with Vega. Yesterday, Saturday, I had a look at the problem and I still had no idea. So I posted a big description on reddit and went and walked the dog or something. When I got back, a useful person had replied, but I couldn’t get what they were telling me. After a bit more head scratching this morning, I finally figured it out and I got the plays page doing something.
Number of different games played over time, for Karlsen
I’m claiming it mostly wasn’t my fault! I thought that if I imported Vega it would import an appropriate set of sub-libraries. Not at all, it imported an inconsistent set of sub-libraries. So two versions of things I didn’t specify didn’t with each other. I am grumpy about that, but when it works Vega is so beautiful I’ll have to put up with it.
Not that this graph is particularly beautiful, yet. I probably should have used a colour other than black for the line. When I start adding more users, each line will be a different colour and rendered with a different symbol.
I’ve been working towards that goal for MONTHS. After I got this much going, I had a look at how to get the geek buddy lists from the user data into this page. Then I got stuck with another bug for way too long, because Angular doesn’t tell you the URL of the page you’re on… and it doesn’t even say that it does, it’s just that it looks very much like it might, and that confuses a great many people.
So I’m continuing work on adding multiple users to this page. It was very nice though to be able to add a news item to the news table in the database for the first time since September.
