Month: August 2018

I mentioned a couple of weeks ago that I needed to do a big survey of all of the features in the old site, and make a plan for getting them onto the new site. And that I  had a big block of drawing paper specifically for such situations. Well, that bit’s true, but when it came to the crunch, I was sitting in bed with my laptop and couldn’t be bothered getting the drawing paper, so I had a(nother) look at Trello, and discovered that Trello is a great tool for organising a bunch of pictures into groups. So now I have a Trello board with the work that needs to be done to get the new site being as useful as the old site.

I’ve been doing a lot of under the water work on the Favourites page, and spent the morning extracting bits of that code in the hope that I could use them again for other things. It was pretty cool from a programmer point of view, but useless from a user point of view. But then I decided that I would take the good bits that I’d extracted from the Favourites page and apply them to the User Collection page, which is the first table I did. I sort of wondered where that page was going, but it was supposed to correspond to one of the columns on the Trello board.

About then was when I realised that I’d deleted the user collection source code in a fit of cleaning one day. It’s on Github, I could have got it back, but when I consulted Trello to find out what the table on that page was supposed to look like, there wasn’t one. So I didn’t bother. Technically that page could hold a table which included differences between your rating and BGG averages, but that’s on BGG as Aldie stole the idea from me many years ago. Maybe I’ll make that table one day, but this is not that day.

There were three things in that column, but I only had time to get one done, so I chose a pretty one. It was the ratings by published year graph which looks like this on the old site:

You may remember from previous posts that I’m using a charting package called Vega. Vega is lovely and horrible at the same time.  It’s very powerful and the charts are very beautiful, but Vega does not play very well with Angular, so it’s always a bit of a pain to get it working. However after a couple of hours I got it sorted.

If you click through that image you will get to a live demo on the test site. (I usually link to the test site from the blog, as the live site gets changes about 24 hours later due to the CDN.) As I write this, there’s still more work to be done on that chart (e.g. tooltips, configurable start year), but it’s about as good as the old one. So I’m faintly pleased with that work.

In other news, the Favourites table got The Most Advanced Feature Yet! As well as having the Documentation button, and the Charts button, it got a Configuration button. The configuration button lets you change the selector that the table uses to load games. And of course if you do that then do charts, they are based on the new data that you loaded. This is very exciting to me, and maybe one other person.

The plan is that logged-in users will be able to store their preferred selector for each page. I’m not so far off achieving that, despite the login button being a bit broken at the moment, and not much functionality having been achieved on other parts of the site. But, it’s a brave new world that has such features in it.

My wife is an information science academic, meaning that she spends her time thinking about how people interact with information. For example, what’s in your mind when you’re doing a Google search. Or when the doctor tells you you have to stop smoking or you’ll die, what bit of that don’t you understand. She talks about things like “affordances”, which means sort of “a thing that’s available for you to do things”, e.g. a button or a door handle.

So user experience is not an alien concept to me, but you wouldn’t know that given how crappy the site has been so far. I’ve been doing that thing like a duck does, with a whole lot of furious activity under the water, while above the water the site has been serenely useless. Today’s job was to address that.

During the week I got the geek page working. Actually, no, that was this morning. It’s been a long day. The geek page is like this:

https://extstats.drfriendless.com/geek.html?geek=Friendless

It’s a page all about one BGG user. It has links to various bits of the site, some of which I’ve even done and work properly. So if you can get to the geek page, you can find your stats.

Then today I coded up an idea I had last night for the front page of the site (extstats.drfriendless.com). It’s like a FAQ with stats. I think geeks will get it, and they’ll play with it, and be able to find the rest of the site. If you have ideas for FAQs, please suggest them to me.

Also, calloo callay, I got the blasted login button to go to the right hand side of the page.

I’ve spent the day working on cleaning up a few things. When I started this project I didn’t know what could succeed, and I tried a few different ways of doing things. Time has shown me better ways, so I’ve gone back and revised a lot of things.

I’ve extracted two JavaScript libraries – one for the various data structures that I send around between the server and the client, and one for common user interface components (in particular, the charts). Extracting the charts out should make it easy to stick them into other views. I also need to extract the documentation component, and push some more stuff into the table component so that they can be reused as well. The tabbed view in the documentation has me a bit confused at the moment. I don’t really understand the template stuff they’re using, so I can’t fiddle with it.

I also updated the navigation bar to be more garish, and kinda like how it has turned out. It’s starting to give the site that look of game pieces all over the table that I’m going for.

I’ve also been working on a page that represents a geek. The more I think about the old site with the geek page with a variety of tables and diagrams on it, the more I think that’s not suitable for this site. On that site you scroll through a lot of stuff, I want this one to be more about poking around and discovering things.

The way I see it at the moment, there will be a relatively small number of different pages, but each of them will be quite complex. I need to sit down with the old site and plan out what those pages will be. Then from the geek page shown below, there will be a number of coloured squares that link to those pages. The squares will contain a little bit of data of their own, but I was too tired today to put it in there so they’re mostly blank at the moment.

You might also notice the login button hanging in there causing trouble still. Between the CDN caching stuff, Auth0 demanding https, and insolent damn thing not want to go to the right-hand side of the page, it’s wearing me down. However at least now it’s a nice colour. Work will continue on that.

But anyway, this thought process led to the conclusion that the user collection page, e.g. http://test.drfriendless.com/collection.html?geek=Gecko3D, which is not very useful, is not very useful because I’m not clear where it’s going. And that’s practically the last step before it going somewhere.

In my method of work, ideas come together when I make a Big Diagram. I even have a big drawing book for such events. I think I’ve arrived at a point where I know what the next Big Diagram has to have in it. No really, this is a great thing!

I went to a MongoDB meetup a couple of weeks ago. I don’t use MongoDB, but I do like the meetup and I do think I need to know about NoSQL databases. There was a talk about MongoDB Charts, which looks like a pretty whizzy product if you use MongoDB. I was particularly impressed by the pretty charts, so I had a chat to the speaker afterwards, and he told me that they use Vega, and that I could too. So I put that on my list of things to do.

Now having spent way too many days writing code for the site that was not very cool, I decided to spend today trying to shoehorn Vega into the site. It looks like it worked!

If you want to try it yourself, the place where it definitely works at the moment is http://test.drfriendless.com/favourites.html?geek=Friendless. test.drfriendless.com is like extstats.drfriendless.com but without the https – the content distribution network which gives me the s in https also caches my pages all over the world, and it doesn’t like it when I update stuff all the time. So when I’m programming I send the changes to test.drfriendless.com.

At the top of the table is the Documentation button, where you can find out what the table data is about. So next to that is the Charts button, which reveals more buttons to display charts. When you use one of those, the I give the data from the table to Vega and tell it to draw a chart. Because I’m using the data from the table, there’s no network call related to any of the charts at all, and it’s really quick.

So far there are just 3 charts, because as I was approaching getting them working I realised that I really don’t know anything about Vega. Vega is very clever and very complex, and I haven’t read any of the doco yet. I think I’m off to a good start though!

Of course once I’ve modified the tables so I can hook in the selectors, you’ll be able to control the range of the data. And when I figure out how to work Vega, there should be a huge number of charts to choose from.

I’m very excited. I’d better go read that doco.

 

I spent most of yesterday working on login, AGAIN. I think that’s the fourth full day I’ve put into that subsystem. Except for the few minutes I spent to stick the navbar onto all of the pages. It’s ugly, but it’s kind of a fundamental component as it helps users discover what pages there are on the site.

One of the entries in the navbar is “Data Protection”.  This is not a very interesting topic, but thanks to the European Union’s General Data Protection Regulations, it’s something I probably need to deal with at some point. Having contemplated retrofitting GDPR compliance to the website at work (which is unnecessary because we only conduct business in Australia), I decided that the best way to deal with GDPR would be head-on up-front.

This is why the only thing on the User page (https://extstats.drfriendless.com/user.html) is a JSON dump of the data I store about a user. I was starting work on configuring user options, but I got bored and will have to spend probably another day on that. The sorts of things I would add first would be:

• screen display name as other users see it
• whether you consent to have your personal data shared with advertisers
• what your BGG username is
• in fact, what your BGG usernames ARE.

That suggests a few things about the future of the site. First of all, advertising. Yes, there will be some, unless I make sufficient money to fund the site some other way. In fact I tried to turn ads on already but Google refused for unexplained reasons. One day I’ll try again.

Secondly, other users seeing your screen display name. I have to admit I don’t know exactly where that’s going, but maybe one day add the ability to play games online. I’m very interested in coding games, but I guess I had better get the site going first.

Oh yeah, and your BGG user names. Some people manage multiple BGG accounts – theirs and their spouse, and their games group. So I would like to make switching between those accounts easy. Maybe I will also add geekbuddies, so that you can easily compare yourself to your buddies, or treat them as a group, or something. I did once develop an app to help decide what you and a bunch of friends might like to play, and I would love to bring that onto the new site.

The missing page in all of this is the Geek page. The Geek page will be the central place for the site’s data about a particular BGG user. Naively, I expect it to be similar to the pages in the old site where a bunch of tables and graphs are presented. I don’t think that’s practical though, for several reasons.

The web technology that I’m using to write the interactive parts of the pages, Angular, is very clever. However the one thing it can’t do is play nice with other Angular applications on the same page. Even on a page like the front page or the user page, the application on the page and the login button interfere with each other. I haven’t seen any particular errors related to that, but I don’t want to take any risks. So I need to figure out some solution to that – maybe isolating them from each other in IFRAMEs, or packaging them as Angular Elements (which don’t work very well yet).

Another argument against the pages full of features like the old site has, is that that’s a pretty archaic way to do it. All of the graphs there are generated on the server side, and the PNG data is sent to the client. As JavaScript can do some very cool stuff these, the modern way is to use the data that was loaded from the server and draw the graphs in the browser. In fact, see my next post which I haven’t written yet for more details.

This suggests that the new site will be substantially different from the old site. With selectors being more shareable, pages being much more suited to interactivity, and fancy things like graphs being easier to do, I am guessing it will be a lot more interactive and encourage exploration. What sort of experience that adds up to remains to be seen.

Finally, as I write this, the downloader is struggling to complete the download of the plays data. By struggling, I mean there are 16 files left to process out of about 300000, and those seem to be broken. So I’ll go fix that and then write another very exciting blog post.

I got the AWS bill for July, and it’s not quite as amusing as the one for June was. It’s not going to destroy the project, but it does show that I need to keep an eye on costs.

Going through the significant line items, CloudWatch is one that I didn’t really want to see there. CloudWatch is cool – it’s where a lot of the graphs that I post come from. I had 4 dashboards showing me graphs, and it seems I only get 3 for free, and the 4th cost me $2.87. As I didn’t use it anyway, I trashed it.

The other component of CloudWatch is a charge for data sent to the lambda logs. It’s not for storing that data, it’s just for getting it there. I was warned weeks ago that that was going to be a problem so I cut down the amount of logging I was doing. Still, there has been a lot of activity this month due to the plays download, so there was always a bit of logging from that. Let’s see how it goes next month.

The Data Transfer costs are small so far, but I’m not sure whether they’re for me downloading data or for me sending out requested data. If it’s the latter, the site is in trouble when I start advertising it to 3000 users. Let’s watch that one.

The Elastic Compute Cloud costs are for the blog. I realised that cost was coming so during the month I decreased the size of the virtual machine that the blog was hosted on, and that cost should be less next month.

The costs for Lambda are for way exceeding the free quota of time and compute power. I suspect that is also related to the plays download.

The Relational Database Service costs are for the database, and that one will probably go higher next month. As mentioned in previous posts I had to upgrade the database during the month so that it could cope with the plays download load, and it’s still not doing that well. The good news is that for about double the cost I think I can start using AWS Aurora, which allegedly has very good performance. So when I organise a revenue stream for the site that might be resolved. Until then it will be coming out of my pocket.

As mentioned last month, it’s not too bad, but the costs definitely remain one of my priorities. When I start asking for Patreon contributions and putting ads on the site, you’ll know why. It’s not because I’m trying to profit from the site, but because something of this scale costs money under any circumstances.

 

One of the more complicated features of the old site is a thing called selectors. They were an idea I thought of long after the initial code was written, but they were so handy to me that I retrofitted them a bit.

Essentially a selector is a rule that describes a set of games. As there are many sets of games involved in the site, having an easy way to describe them is useful. In the old site, it was mostly the case that the rules for choosing games for a feature were hard-coded, meaning that if I hadn’t chosen the right set of games for every interesting case, the site continued to suck.

However having had that idea once, I am too smart to let it go this time. So for the Favourites and Collection pages (the only ones that are working at the moment) I’ve encoded the rule about what games are displayed as a selector. When I get around to working on the table widget, I’ll make the selector editable, so then you’ll be able to choose your own set of games for each widget.

Another thing that has changed since I wrote the first site is a technology called Ajax, or XML HTTP Request. Those of you who are old may remember a time when you loaded a web page and you sat there and looked at it and it sat there and looked at you, and you were both stolidly passive. And then some pages started doing this thing where you would do a thing and they would change. Well the thing that enabled that was Ajax, which allowed the web page to load new data from the server after the page was loaded.  That technology only entered the standards process about when I started designing the site, so I didn’t know about it and didn’t use it.

These days though, it’s very easy to do. It’s built-in to the technology I’m using to develop the widgets on the web pages (Angular, by the way), so there will be a whole lot of it going on. Which means that generally the widgets on the pages will be a whole lot more interactive and fun to fiddle with. And selectors is one of the things you’ll be able to fidde with.

There’s a test page for selectors, which has the documentation and a test bed so you can try selectors out. It’s supposed to be here:

https://extstats.drfriendless.com/selectors.html

At the time of writing I’m waiting for the CDN to find that page and make it available. As I add more selectors I’ll add them to that page, because that’s the page I’ll be using to see whether they’re working.

One more thing before I go. I went to a talk at MongoDB the other night, and they are using this library called Vega for their charting tool. I think I might have a use for it myself.