Posted on

This Ain’t the Panama Canal

“A Man, a Plan, a Canal – Panama”

The lovely palindrome suggests that the building of the Panama Canal was planned and executed in a rather neat and reversible way. So when I consider my development process which is kinda haphazard, I feel a bit inadequate and envious. Fortunately, a quick glance at the Wikipedia entry on the building of the Panama Canal suggests that its 33 year construction was not without drama, difficulty or incompetence. Ferdinand de Lesseps, the man with the plan, was sentenced to go to jail for his role, but never actually did.

So that makes me feel better. The first Extended Stats site started as a few Python scripts. I spent years running them nightly and uploading them to free hosting sites, which were universally shitty, resulting in my decision to host the site at home. Maybe only a year after that I rewrote the scripts as a proper web site. Between the shitty free hosting and the world’s mistrust of dynamic DNS, the site name kept changing. As web users continue to expect more (e.g. https) and I become more professional in my approach, it continues to evolve.

The same thing is happening with the new site. I spent a good deal of 2015 and 2016 thinking “I should make a new site”, then a good deal of 2017 thinking “hmm, what would be the best way?” And then I spent a few more months trying to get those ideas to work. It was only in June 2018 that the technology started to work for me. About then I realised that Node.js, which I had chosen to write the server-side in, was not one of my talents. So I wrote some fairly bad Node code for a while, and maybe I am still doing so. Though it’s a few months since I’ve found a way I like better.

Similar things happened with the web pages. Although I am competent with Angular, when I started I didn’t know what I wanted on the pages, and I still don’t, really. However they are slowly converging. Whenever I write a new page there are some bits I like and some bits that are horrible. When I’m done, I take the bits that I like and put them into a library on NPM, and use them again later. That makes it easier to re-use those ideas on later pages, or on revisions of existing pages.

So this morning I sat down, determined to put the Plays Of Games Owned table into the new site. The first task was to find the right place for it, which would be on an existing page which already has the correct data. I looked at the Owned Games page, and noticed that it was the first page where I had implemented the pattern of the page loading a set of data, and multiple components rendering their views on it. However some code from that page had been put into a library, so I fixed up the page to use the library and stuck in a loader component while I was there (the coloured bobbles that indicate that the data hasn’t arrived yet).

However the Owned Games page doesn’t have plays data, so it’s not the right page for that table. So I next considered the Favourites table, which does include plays. The selector isn’t quite right though, as the Favourites table defaults to games played and rated, and isn’t anything to do with owning them.

The Favourites page one of the first pages to use Vega charts, so it had a Charts button that you could click on to get to the charts. That preceded the way I do it now where each chart appears on the page as its own thing, and uses the data the page provides. So I moved those charts out onto the main Favourites page, and wired them up to receive the data. Then I noticed that the Favourites page was the one where you can change the selector, but the new selector was injected into the table, rather than into the page. So I went back to the library and changed the way the config component worked, so now you can change the selector on the Favourites page, and the table and all of the charts update. It’s still clunky, but it works. Oh, and the Favourites page had one of the old-style tables without the handy header bar, so I fixed that up as well.

And that’s why it’s now late afternoon and I have not yet achieved the first thing I set out to do today. The site has evolved a little, and I look forward to wiring the selector changer into other pages, and hopefully making it nicer when I do so. I’ll see if I get to try again tomorrow.

“A man, no plan, a clusterfuck, a clean up.”

Posted on

A Bad Reaction

I may have mentioned several times on this blog that I use a technology called Angular to write the web pages on the site. That’s what I use at work so I have some experience with it. There’s a competing technology called React, which I have heard a lot about, in that I have subscribed to various React podcasts and so on, and generally opened myself up to it in the hope that understanding would come without too much effort.

Several BGG users who use React have told me “Angular’s dead! React has won that battle!” So, in response to such taunting, and also in response to a weakness in Angular where I can’t have two Angular applications on the same page, I decided to rewrite the navigation bar in React. The nav bar is a terribly simple component, and it couldn’t be that hard, could it?

Indeed, it was not. React (like Angular) has a nice feature where I can change the code and see it instantly changed in my web browser. So, following my usual process of one Google search for one line of code, I got it working to my satisfaction. And it was so simple I really did not need React, but maybe some other ideas I have will justify my choice in the future.

So that was good, I had some working React, now I just needed to update my process to take the JavaScript that React produced and put it onto my web site. React produced 3 files, with extremely unhelpful obfuscated names that change every time you change anything. This is bad because the rest of my site, which loads those files, does not change very much. I edit those files by hand, and there’s no way I’m going to edit them to put unhelpful obfuscated file names into them on a regular basis.

No problem! There’s a thing called webpack which does that. Oh yeah, except for the problem that webpack is badly documented, hard to configure, and keeps changing. My first attempt to use it involved copying some configuration I found on the internet (it seemed like a good idea at the time). And that seemed to work at first, but then it stopped. So I had to poke around and debug a whole lot of things to figure out what was going on, and that took WAY TOO LONG.

Of course, this could have been a whole lot easier if React had not generated the unhelpful obfuscated file names, but the React developers refuse to understand why anyone might not want that. It seems that React is the whole universe, and once the code works in any fashion, their job is done. They have no concern with fitting in with larger processes. Nevertheless despite my fury on that particular point, I liked React and I’ll use it again. Webpack, on the other hand, needs a bullet in the head.

Well, I hope this blog post provides me some catharsis! I’ve spent several evenings reading up on how webpack works when I never cared to know in the first place.

One final word. Angular is not dead. In my time outside the React world, I heard a lot about state management. There’s a technology called Redux which is used to help with that. It’s totally irrelevant in the Angular world, because we don’t have the state management problem – the solution is built in. So it worries me a bit that React requires a complex (and by all accounts, really boring) solution to a non-problem. Angular does have its problems (I nominate zone.js) but I’d say otherwise the technologies are equally capable, but different. And different is OK.

Posted on

Ooh Er, That’s a Bit Fancy!

When I started writing the user interface parts of this project, I knew I would need a table component. Angular has quite a few libraries, but I couldn’t find a table component that was useful and that I could get to work… well until the one that I ended up with and used for the War Table and the Rankings Table. It wasn’t great, but it sort of worked and got the data out there.

However I always aspired to something nicer – I aspire to make the whole site better, but I have to do it bit by bit! So I prepared to make the table component nicer by reading the source code, but at first I didn’t understand it. It didn’t do complicated stuff, it just used Angular features that I didn’t know existed. This is what I get for being a newb! However today I managed to get myself sorted to make an improved version, and I’m rather pleased how it came out.

The War Table, updated

I moved the pagination stuff to the top of the table. If it’s at the bottom, you may not see it to use it. I also got the CSS under control so that the pagination bar looks like a natural extension to the table. I also took control of the page numbers you could go to so that it’s easier to scroll through the pages with some sort of intent.

And with all that extra space I had available, I added a Search field. One thing that I did not like about those tables was that it was really hard to find yourself – nobody wants to scroll through lots of pages searching for something. So if you type a geek name into the search field, it will take you to the page with that name on. How cool is that! It makes all the data behind that table much more accessible.

I guess tomorrow’s job will be to fix the rankings table, because IMHO this design is too sweet to ignore.

Posted on

Swaggering, or Staggering?

During the week (well, last week, I think), a strange thing happened. I got an email from GitHub, where I keep the source code for the site, saying that someone wanted to fix a bug in the site. So I had a look at what they suggested and it made perfect sense, so I accepted the change. So since then that developer and I have been chatting, and now I have a collaborator. Or at the very least, a second opinion on many things. This is very odd for a doctor of friendlessness such as myself.

Now I have been a programmer longer than most people have been alive (source) so I have a great many opinons about programming things, but for much of the technology used in Extended Stats Serverless, I’m quite new to it. And being a humble motherfucker with a big ass appetite to learn, I’m very happy to take advice on these techs.

So one thing that has been on my mind is a product called Swagger (now changed to the Open API Initiative, which is a very dull name). Swagger is a system for describing APIs. An API is an application programming interface, which means it’s a way to talk to an application by programming. These days, it is coming to mostly mean a way to talk to a web application by making calls to the web site. One way to use an API is as the back-end to the web site, which is what I do.

When you go to extstats.drfriendless.com, all of that stuff is being loaded from AWS S3 (think of that as Amazon’s hard drive). Then the code runs in your web browser and talks to api.drfriendless.com to retrieve data – api.drfriendless.com implements the API. So what I was doing with Swagger was documenting what you could say to the API and what it would say back.

That doesn’t sound hard. And indeed it wasn’t, to do the sort of half-assed job that I did. Nevertheless it came to about a thousand lines, which took quite a while to do. I’m sure you’re desperate to see what I produced, so here it is:

https://app.swaggerhub.com/apis/DrFriendless/ExtendedStatsServerless/1.0.0

Yeah. Although the swaggerhub interface is nice, it’s still pretty dry stuff. Nevertheless I felt it was an essential bit of due diligence that I should do. And while doing I found a few cases where I thought “huh? why did I do it like that?” which I will probably go back and redo to be more sensible. So that’s a good thing.

Posted on

Scheming

I decided I needed a loading widget for the pages where a great deal of data needs to be loaded (i.e. all the useful ones). So after copying some stuff and fiddling with it, I came up with this:

I kinda like it. I like the colours. If I knew anything about graphic design I’d know how to proceed from that to make a colour scheme for the site.

To the uninitiated, developing a web site like this is just all beer and coding and living the high life from the Patreon proceeds. I have other responsibilities though! I’m fully conscious that the site looks a bit amateur, but that’s because I am indeed an amateur in many aspects of web development. I’ve been reading books about colour, and design – in the aesthetic sense, not in a how-to-do-it-in-HTML sense. So I’m hoping that over time some sense of style will infect my brain, and I’ll apply it to the site.

Posted on

Lots of Things Almost Work

Of course the Plays page which I was touting last week failed miserably. It complains about the base setting being incorrect. I suppose it’s telling the truth, but it’s a bit annoying to find these things out in production. So I’ve deployed a new version and due to the CDN, you might see that new version tomorrow. In the meantime, you can play with the test site with URLs like this: http://test.drfriendless.com/plays.html?geeks=Friendless,karlsen,jmdsplotter which show off the colours and the multiple geeks.

Another thing I managed to sort of achieve today was compression. A graph such as the one above retrieves a lot of data from the database – maybe 2 megabytes. Sooner or later someone has to pay for that data transfer, and someone is gonna be me, so I wanted to cut that down. Luckily, web servers and web browsers have this agreement that the browser can ask for the data to be gzipped, and if the server can do it will. One of the things AWS’s API Gateway gives me for free (for which I forgive it for being a massive pain in the arse other times) is that I can just tell it so send data compressed, if the web browser asks for it.

So I changed my code to ask for the data to be compressed, and Angular (the JavaScript framework I use to write the pages) said “NO! You are not allowed to ask for that!” Apparently it’s a sin to be running around asking for stuff to be compressed willy-nilly. And the word on the street was that the browser does it automatically.

Which it plainly was not doing. So I had a think and a search and another think, and remembered that as the request for the data was coming from extstats.drfriendless.com and going to api.drfriendless.com – a totally different place – the two ends don’t trust each other. So extstats.drfriendless.com could only ask for compressed data if it first got permission from api.drfriendless.com to do so. This is a thing called CORS, but don’t Google it, it’s horrendously complicated annoying crap that just prevents things from working. For example, it prevents sites from stealing your bank logins and that sort of stuff. But after I configured CORS, the browser was allowed to ask for the data to be compressed, and the server was configured to do it, and hey presto 90% data compression.

Although the site doesn’t have a lot of the functionality I want on it, yet, I feel that mechanically it’s approaching stability enough that I can start inviting lots of people to use it. They’ll be disappointed I expect, but I will use their misery as motivation.

The old site annoys me more and more each time I look at it. During the week we had a technician come to convert our internet connection from “cable in the house connected to cable in the street” to “cable in the house connected to fibre in the street”. That may make the old site run a little faster, but probably not a whole lot as that site’s big problems are the database architecture and the wifi link from the machine to the home network. But I had to spend an hour or two of misery trying to understand networking and Ubuntu config things that are just not problems with the new site. I think I’ll be pleased when I can close that site down.

Posted on

Goodness, a thing happened

Last time I blogged I mentioned how frustrated I was with Vega. Yesterday, Saturday, I had a look at the problem and I still had no idea. So I posted a big description on reddit and went and walked the dog or something. When I got back, a useful person had replied, but I couldn’t get what they were telling me. After a bit more head scratching this morning, I finally figured it out and I got the plays page doing something.

Number of different games played over time, for Karlsen

I’m claiming it mostly wasn’t my fault! I thought that if I imported Vega it would import an appropriate set of sub-libraries. Not at all, it imported an inconsistent set of sub-libraries. So two versions of things I didn’t specify didn’t with each other. I am grumpy about that, but when it works Vega is so beautiful I’ll have to put up with it.

Not that this graph is particularly beautiful, yet. I probably should have used a colour other than black for the line. When I start adding more users, each line will be a different colour and rendered with a different symbol.

I’ve been working towards that goal for MONTHS. After I got this much going, I had a look at how to get the geek buddy lists from the user data into this page. Then I got stuck with another bug for way too long, because Angular doesn’t tell you the URL of the page you’re on… and it doesn’t even say that it does, it’s just that it looks very much like it might, and that confuses a great many people.

So I’m continuing work on adding multiple users to this page. It was very nice though to be able to add a news item to the news table in the database for the first time since September.

Posted on

Much Frustration

Sometimes it would be more fun to bang my head against a brick wall rather than be a programmer. However programming pays better and I don’t have much experience in headbanging. This is no time for a career change.

I managed to make some progress on the User page. That’s the one you get to after you log in and click on your user name. It seems that it can now store your BGG name and your buddy lists, and then display them again afterwards. It has taken WAY TOO LONG to get that working.

I then continued work on the Plays page – the one that uses the data about all the games you’ve played. I’m working on a chart to compare how many new games you’ve played when with the same for your buddies from a buddy list. I’ve had the data gathering working for a few weeks, and I have just finished the chart. Well, sort of.

The charting tool, Vega, is the most beautiful and argumentative contrary piece of shit software I’ve ever used. After spending a couple of hours getting the chart to work in the debugger, I had to address why it didn’t work in the page. My first idea was to upgrade some versions of some things.

That was a complete disaster. Vega has internal conflicts, so now my code can’t compile and won’t run. Using the latest versions of everything, it doesn’t work. Using the same versions of everything from this morning, it doesn’t work. I have turned it off and on again so many times… so now I’m bored and I’m going to do read a book instead. This happens every time with Vega, I seem to be missing something.

Posted on

The Bills, Master, the Bills

I hate banking and I hate bills. Not because it’s money going out – I can cope with that – it just gives me anxiety. I am forever doomed to be a programmer (or possibly a novelist) because I hate thinking about money and I hate doing banking. This extends even to the email from AWS about the bills giving me anxiety. That’s why I had 3 unread ones in my mailbox until today, and they are issued monthly.

Despite my personal issues, the bills were great news, as illustrated in drop in the graph above. The database continues to cost a fixed amount (variations are due to varying numbers of days in the month), and the other costs are all dropping. I could probably do even more about the EC2 costs now that I have the lambdas sorted, but I feel that I should get some functionality working on the site, for a change.

Posted on

Family Business

Before I fall asleep, let me bash out a quick blog post so that you know I’m not dead. Instead I have been occupied with visiting family, which is the next best thing.

I have been doing some work on the site, but it hasn’t turned into useful functionality yet. I believe I’ve got the API for retrieving plays of games working, so next I have to produce a user interface which displays some of that data. I did a little bit of work on that this evening before I had to drive somebody somewhere.

I’ve also been chatting with the guys from https://www.rollforgroup.com. There are more of them than there are of me! I met the head honcho there at a MongoDB meetup before I started this rewrite, and when you find another programmer obsessed with board games you’ve got to be friends with them, right?

I also got a couple of new Patreon patrons early this year, which was wonderful! But that was exactly at the time that I went away for a week to play board games (including Gloomhaven, for the first time), and then when I got back the Family Business started. Now I remember why it took so many years for the old site to evolve to its current state.