Week 7: Starting the Final Project

I’ve realized there’s no reason to constrain everyone to having┬áboth a real user and a real data set right out of the gate. Once you settle on one constraint, we’ll find or simulate the other.

There are a few considerations about the data source:
1) It’s updated at least daily
2) It can be made completely publicly available, even in granular form
3) It can be used to make a decision with important consequences

Also about the user:
1) They’re able to answer questions by email or phone over the course of the next few weeks
2) They would be able to gather this sort of data
3) They are either the decision maker, or directly talking to the decision maker

Once you’ve nailed those down, you’ll step through the data pipeline that we’ve set up repeatedly in class. The first part, which we only touched on briefly, is the collection of data. We looked at web scraping options, but you can also use APIs, or more batch-process data extracts. Whichever intake method you choose, your first step will be writing it into an SQLite database in as most granular form (the operational database). The reason for this is that you will probably make changes to your data warehouse design as you iterate, and you’ll need to be able to recreate the whole thing from raw data.

Once you’ve built a data warehouse, you have several options for your data front end. In all, there should be three interfaces on your dashboard:
1) A weekly report
2) An ever-morning status update
3) An alert

Of course, this is going to be an iterative process, and what you’ll enter into the spreadsheet now is your best guess. But it pays to be bold and specific with them, because they are the platform on which your next, more refined, draft will be built.

In the spreadsheet below, add at least one data source and one potential user (someone that you know personally, and suspect would be open to trading emails with some students). It doesn’t have to be one that you’re interested in personally. Once we get to class, we’ll go through the list, each of you will pick an idea, and get we’ll get going!

Add ideas to the spreadsheet

Leave a Reply

Your email address will not be published.