Trademe Graphs

Lets Build Some Trademe Graphs

I’ll hopefully start of simple and keep output simple to begin with. My goal is not to create a tutorial, but give updates every few days of progress and the process I am going through. BEWARE. I will be taking  a lot of shortcuts to get to a minimum viable product in a test area first, which I will go back to again and again to continuously improve with what I hope are best practices. I am a beginner in a lot of these technologies so this is primarily a learning process for me.

Process

This may change, but will use nodejs to query trademe api and load selected data into MySQL database periodically. Then will use nodejs to create a server and send data to the client to display some simple graphs via a javascript graphing library.

Output

Trademe has interested me for a while and have chosen to show some graphs of residential property listings by location. Simple stuff like price, type and bedrooms. More importantly want to start using some best practices. So will:

  • Use Git for code and issue tracking. You can find that at mabznz/trademe
  • Use Ansible to up an AWS server with required software and code
  • Use some test suite that have not decided on yet

1. Start Using Trademe API’s

Maybe the best documented and comprehensive NZ based API for your average public joe like me, Trademe is a brilliant site so have started here.

A warning. First place I got bogged down was around the oauth authentication and authorization methods. Not knowing anything about oauth did not help and I am still a little unsure how this all works. But if you just want to use your own and site wide read-only data, you can just generate your own token and token secret for your application that you register at the authentication documentation. I have saved my failed attempts at requesting a token and authorizing to git (trademe_app2.js). Oauth would be great to explore and did find a useful page for people if they are looking at this area. http://emtek.co.nz/trademe-api-and-oauth/

2. Getting data into database

This is quite simple with nodejs and MySQL. The table structure I am going with is simple as well with just three tables of. TODO. Add diagram.

As an example the script for doing the one of loading of geographicalocalitties

Advertisements