Personal Project Journal 2021 July

July has been a month where I put a lot of my spare time working on various personal projects of mine, and one that managed to make a lot of progress on several projects of mine.

Data House

Data house is something that I've dreamed in my head for a while, so it was good to be able to make it into a reality. I didn't really talk about it apart from several screenshots I've posted on Twitter and Instagram.

I started collecting data about myself about a decade ago when I saw Feltron do these amazing data visualizations. I felt that I needed to collect some data, and turns out, I collected a lot of data, both manually and automatically. I think this is the most unified I've tried to tie all my data bases together.

I'm very happy with how it has turned out, I need to go back and spend more time curating it, but being able to create a Trips page, a full hour by hour breakdown of what I was doing and some mini visualizations of the data in that hour.

This will be an ongoing project of mine.

I'm especially proud of the fact that I managed to get all the data to be hosted in the cloud, under my control. The raw data itself is checked in to github for redundancy and tracability.

Mem

Mem is a bit of an itch scratcher. Mem is my idealized version of what I want a note taking system to be. It's probably closest to what Notion and Evernote is, but it is a lot more simple. It's like the very basic version of Evernote, Notion, Pinterest, Delicious and Instapaper combined. It's my little pack rat project that allows me to store anything into there from as many entrypoints as possible. I can send a few words, a few paragraphs, a URL and even a photo to it. And it just stores it in a basic structured data format that will allow me to query and visualize it in the future.

I was able to build this from scratch within a week. Still a lot of things not yet working, for instance, I discovered that Instagram serves up a log in page if you hit a instagram post URL from a Google server, but if you try from a non-datacenter server, everything is returned fine. No combination of headers has yet gotten me around this, but there must be a way to.

This can evolve in all sorts of different ways, it could become my manual collection endpoint, I could be sending me mood, random events and notes in to there like I do with Foursquare, if Foursquare ever shuts down. I'm pretty excited to build more on this.

Snow Map

I went back to doing more with Snow Map. Snow map has a lot of different aspects to it, the website snowm.app, the manually curated data set, semi automated crawled data that I'm manually aggregating into a master index of all the snow resorts in Japan.

The flaw so far with this is a lot of runs on my local machine and so I can't get stuff out, and the local dev environment is complex. So instead, I spent a lot of my time rewriting the whole internal guts of this so that all the data is in Cloud Storage, running on Cloud Functions. I was already half runninig on Cloud Functions but asumed that I downloaded all the data locally to script it. But instead, I've rewritten everything to be runnable locally, but actually natively in the cloud. So while I run some command line scripts on my machine, the real work is being done using Cloud Storage.

I'm pretty happy now that at this point, everything from the crawling, data classification, data generation and data labelling and machine learning model training is now in the cloud, nearly nothing is local on my machine except for the web app. Even that can be on the cloud, except it is still much faster to have hot-reload using Vue JS.

I've also scratched a really big visualization itch which has got me to learn how to be more proficient at using Three JS, but also overlay Three JS on Mapping APIs to create really cool visualizations.

Right now, I've built some really great tools, now I need to use them to train a machine learning model to help me classify GPX traces for mountains in order for me to produce a data-based summary of mountains.

Blur App

Also quickly hacked up a image blurring web app on a whim. I got quite a lot of play on Twitter and Instagram, though I'm not good a generating buzz and making things into a real app. I need to get better at that, figure out how to put things on to Product Hunt and getting some Google Juice out of it.

Wherecam is kind of like that too, it works great for me, it's public, but I don't know how to promote it.