Welcome to the first edition of the MyData DevLog! My aim is to write these weekly to keep interested people (like yourself!) updated about the project! I’m hoping that I’ll be able to provide commentary on daily activities. However as time goes on and I start my new job, I’ll only be able to talk about progress more generally.
Tuesday 12th December 2017
I’m bored and stumble onto GitHub. I notice that my activity sucks and realize I need to do better
I’ve been sitting on my Fitbit data for some time. I’d only started seriously collecting data since September and this is only a limited amount (Exercise and Sleep data). These two datasets are the easiest to record so I should make a greater effort collecting other types of data.
Right now, I’m struggling to understand how the API works. In the future, I’ll sit down for a day and focus on understanding it. Once this is done, I’ll be able to capture more meaningful insights into my data rather than the limited CSV files I’m extracting from my dashboard.
Setting up the repo isn’t too hard. Just a case of remembering how to do it and looking at GitHub cheat sheets. By the end of the evening, I’ve committed my datasets, uploaded a preliminary node.js app and some jupyter notebooks.
Wednesday 13th December 2017
I’m determined to have the website deployed by the end of the day. Getting back into Node was fun and I want it to continue. I decide on using the Express framework for the app (As I’d learnt it doing a Node course earlier this year) and it’s just a case of installing the bare minimum packages required to get the app going. Once I’ve created a basic landing page, I commit again to GitHub forgetting that I need a .gitignore file! Luckily I only had a couple of packages on there so it didn’t take ages to commit. Removing the packages from the repo was easy enough.
The difficult thing was deploying the app to Azure. I decided on Azure since Visual Studio has a Dev Benefits scheme for casual users. I only need a free app for now so you’d think it would just be a case of uploading it to Azure and it’s done, right?
I tried two different tactics:
- Use Docker to deploy the app. I know nothing about Docker expect it made my computer slow right down.
- Connect my GitHub repo as a deployment slot. I had to create a separate repo for the web app for this to work. This was around hour five and was willing to make a sacrifice to the gods to make it work.
I spent 5 hours trying to follow along with the tutorial to get it working only to realize that I hadn’t specified the correct PORT number for Azure. 5 hours. Next time you think you’re wasting your life on something simple, just think of me.
Nevertheless, I have something hosted on Azure. Granted it’s just a landing page, but hopefully as the project grows, I can incorporate more interesting things such as a bot, dashboards etc.
Thursday 14th December 2017
Changed the design of the website. I found a template on the Bootstrap website that I like and I think I’ll incorporate something similar into this project.
The big obstacle today was figuring out how to perform analysis on my data. My plan was to import the data into a Jupyter Notebook using pandas. This worked well enough, but I made a couple of mistakes that I think are worth sharing:
- The data that you extract from Fitbit includes commas with thousands numbers. This means that when you import the data into a DataFrame, the column becomes an object type rather than a int64 type that I needed to perform statistics on it. I tried using .astype() to change the column type, but in the end had to manually change it in the excel file before converting the file type to csv.
- This comma in the data caused duplicates of commas. Trying to remove these columns was a nightmare, as I forgot that removing columns in csv files is not the same as removing them in excel.
This whole episode has only further strengthened my resolve to learn how to use the Fitbit API within my application, but I learnt a bit about my data so it wasn’t a complete fail on my part. Running some initial statistical analysis on my data was fascinating and helped me understand it. I’m going over DataCamp tutorials that I’ve done and then applying that analysis to my DataFrame. Hopefully as I go on, this work will become more intuitive to me.
Friday 15th December 2017
Things start to fall apart. Not in terms of the project, but other crap starts to creep in. On Thursday, the band on my fitbit brakes so I ordered a new one through Amazon. I’ve been a Prime member since 2009 and I’ve always had reliable service. The driver comes up to the drive way, turns around and doesn’t even bother to get out his van. I’m livid!
I call customer service twice, who assure me that they’ve sent a message to the driver to get my package to me on Saturday. It’s only just the start.
In terms of work, I do some initial analysis on my exercise data. You can check out the progress here so far, but it looks like I need to do some more work in the gym. I may need to reevaluate the way I measure my data. I may need to create or find an app that allows me to track my exercises automatically instead of calculating rest days.
But I shut off my computer, not even realising what fresh hell awaits me the next morning.
Saturday 16th December 2017
I wake up at 5am to a blue screen. My SysAdmin skills suck but I think it had something to do with me installing Docker. I had to reset the PC and re-install all my programs back. Luckily I could reset the PC and keep my files, plus all the files for the project were safely put away on GitHub. It was just a waste of time that took 5 hours to resolve. Still, nothing was lost and I learn a painful lesson.
Amazon still suck. I get an update on my order saying that it’s left Swindon at 7:30 am and is out for delivery. I wait, I wait and I wait. Nothing. No one comes to the house and I get an update from them saying I need to rearrange delivery. I’m livid still!
I call customer service in a much harsher tone. The lady on the phone assures me that she’ll escalate the issue with logistics. This is complete horseshit so I do the only thing I can do in this situation. Complain on Twitter.
I’m not a fan of Twitter. TLDR, it brings out the worst in human civilisation and is the main cause of everything bad that’s happened over the past 2 years (maybe longer). Every time I complain, they suggest that I contact customer service. Which I have done now 3 times along with 1 email. At the end, my frustration boils over:
Not my finest hour. But without that band I can’t wear the device to do my work. At this point, I want Jeff Bezos to fly on that drone, kneel before me and beg for forgiveness.
I fear that my senses may have left me at this point.
Sunday 17th December 2017
To calm down from yesterdays fiasco, I go to the gym. I can’t record it, but it helps me work through some stress.
I also work on implementing the UI that will be implemented once the website is done. I found a cool theme on the StartBootstrap site that I like and that I think will serve as a good production starting point. I’m quietly sitting there, marvelling at the ease of implementing Express applications is.
Next week I need to get a strap. I contacted them for a fourth time and they’ve sent me another package that will hopefully be delivered tomorrow. Just to let you know what I’m dealing with here, here’s the email they sent me:
The 18th December is a Monday. I think it might be better if I go out and look for one myself instead of relying on these idiots. But the nephews arrive for lunch, so I turn off the PC and hang out with them.
- Data is sorted! I’ve done some inital work here and it’s enough for me to get a sense of what data I’m capturing, what data I should look to capture in the future and how I should capture it.
- Website is live. At the moment, it’s deployed to the cloud as a landing page, but I’ve started the work to build something really meaningful.
- Slight PC malfunction took a day out of development.
- Amazon continually failing me on a simple delivery. That Prime membership is not being renewed.
- No Fitbit, no data. This gives me a chance to look at other aspects of the project, but it does mean that my data collection for December will be lacking :(.
That’s a wrap for week 1! These are the things I’ll be focusing on in the coming week:
- Sorting out the band for my Fitbit so I can collect valuable data!!
- Continue to work on the web application with particular focus on connecting the API.
- Finishing up statistical analysis on my September-November data so I can continue to formulate my models.
See you next time!