Tuesday, September 27, 2016

My Project: Travel time bot with Google Maps API and Python - Part 2 - Making the travel data useful

In Part 1 of this series, I showed you how I found the travel time data I wanted and how I figured out how JSON data files function and how to pull out just the data I need. Now that I have this data, I needed to learn a bit of Python coding to extract is and use it in calculations and present it to the screen and other in other ways.

Learning Python

Stack overflow

As has been the case throughout my technology work history, I learn on-demand. For me, this means coming up against a problem and then researching how to solve that problem on the Internet, adapting sample code and finally getting it all to work. That was exactly what happened in the project. I was not very familiar with Python, so each time I needed to accomplish a task I would head over to Google Search (which almost always led to me Stack Overflow) to find the answer.

Let me walk you through my Python code and explain how it works and a little about how I developed it.

Learn more about Python with these books from Amazon.com

More Python Books

Python Modules

Let’s begin at the top. First, I needed to figure out what Python modules I would need to accomplish all I wanted with the Travel Time program.

#!/usr/bin/env python

import urllib
import simplejson
from os import system
import os
import time

The urllib module gave me the tools to access web URLs within the program and request web pages — or in this case — a JSON file served up from the Google Maps API server.

simplejson — as you might imagine — provided the code to easily manipulate JSON by converting it into easily accessible arrays of information. 

os and  system modules allow me to call on basic operating system utilities like curl and — since I am running on a Mac — say, which speaks the information out loud.

The time module provides me the ability to add wait times and get the current data and time. Looking back at the code now, I realize I don’t really use it in this version. I originally had the script set to run every 5 minutes or so, instead of on-demand, so that functionality is not longer needed.

Getting the JSON Data

Google maps api

Next, it was time to get the actual current JSON data from the Google Maps API. Each time this is called, it returns the current travel time based on distance and also traffic loading.

Note: I have removed my own personal API key from the URL. You’ll need to add that in in — along with your start and end coordinates — to make this useful to you.

# Google Maps API URL
api_url = 'https://maps.googleapis.com/maps/api/directions/json?origin=34.178026,-118.4528837&destination=34.057835,-117.8217123&departure_time=now&key=<INSERTKEYHERE>
# Get JSON data from Google Maps API
response = urllib.urlopen(api_url)
data = simplejson.load(response)

The api-url variable is simply an easily modifiable placeholder for the url you might use. This is then used to in the call to urllib to pull the actual data from that API url. Note that I have no error checking in whether this actually succeeded or not. This is something you would probably want to add.

The data returned by the urllib call is stuffed into a variable called response. This response variable is then used in the call to simplejson which creates an array called data which has all the various key-value pairs sorted out and made easily accessible. My initial version of the program used simple text commands to hack away at the data I needed and was much more difficult and brute force. Once I learned how simplejson worked, though. it open the doors to not only accessing API data for this program, but any API data in JSON format from any API. This was a big step in my learning, for sure.

Finding the data I needed in the simplejson array

# Get current duration of trip duration = (data['routes'][0]['legs'][0]['duration_in_traffic']['value']) # Get current summary name for trip desc = (data['routes'][0]['summary'])

You may remember from Part 1 how I used the JSONViewer to give me a clearer, hierarchical understanding of the returned JSON file. These 2 lines are exactly why. Sure, if you are good, you can look at the text version of the JSON file and figure out all the indentations and curly brackets, but for me, using JSONViewer made this much more clear. 

Gmaps json expanded

One confusion for me was the use of the [0] sections instead of — what I would consider more useful — named sections. Looking at the data in JSONViewer, though, it was easy to follow the structure and then see how to specify the exact data i wanted in the array. For this program I pull out value entitled “duration_in_traffic”, which I found to be a more pessimistic, but more accurate time measurement here in Los Angeles and the summary which provided me the suggested route for the trip. This is very important as there are 3 basic routes for this trip that are possible and traffic conditions can cause the Goole Maps API algorithms to switch between them almost minute-by-minute.

A Few Time Calculations

Now that I have the data I need, I can do some calculations on it for better presentation later.

# Perform Time Calculations
dur_min = duration/60 #convert seconds to minutes
dur_hour = dur_min/60 #convert minutes to decimal number of hours
dur_hm = dur_min % 60 #convert hours to hours and minutes

These lines should be fairly self-explanatory except perhaps for using the modulo operator (%) to give me the proper time output in the form of x hours and y minutes. I vaguely remembered that from a programming class in my ancient past and was able to research it again to use it here.

An OCD fix

# Use correct plural in output
if dur_hour == 1 :
    numhour="Hour"
else:
    numhour="Hours"

This little snippet was added later to get around the situation of using the wring plural when reporting the data. It’s funny how something small like this can annoy you so much. (LAUGH)

Printing the Information

Next comes some simple screen output of the data we received. The program started out with these print statements only until I had everything working the way I wanted. This could be dressed up in many more ways, but this is basically just debug data as the information gets reported in several other ways.

# Output API Duration Information
print "\n"
print "Via "+desc
print "Duration: "+str(dur_min)+" Minutes"
print str(int(dur_hour))+" "+numhour+" and "+str(dur_hm)+" Minutes"
print "\n"

Sample Screen Output from Python program
[DEW-Mini:~/Downloads/py] dewelch% python cpp2.py

Via CA-134 E and I-210 E
Duration: 58 Minutes
0 Hours and 58 Minutes

Congratulations! You've fired the cpp event
[DEW-Mini:~/Downloads/py] dewelch% 

Speaking the information

# Build text to speech strings
cpp_sum = "say Current Travel Time to Cal Poly Pomona Using... "+desc
if dur_hour == 0 :
    cpp_time = "say "+str(dur_hm)+" Minutes"
    cpp_time_email = str(dur_hm)+" Minutes"
else:
    cpp_time = "say "+str(int(dur_hour))+" "+numhour+" and "+str(dur_hm)+" Minutes"
    cpp_time_email = str(int(dur_hour))+" "+numhour+" and "+str(dur_hm)+" Minutes"

# Speak Output
system (cpp_sum)
system (cpp_time)

One of my first ideas was to have my Mac speak out the data several times over the course of the morning so that my wife and son would get regular, passive, updates on their potential travel time. This is done using the Mac say command. These lines simply build the appropriate strings which are then executed using the system call. This would need to be altered for other operating systems, as the say command is specific to Mac OS.

Note that, like the OCD Hour/Hours fix above, I make sure to omit the Hours segment if the duration is less than 1 hour. It made it much easier to understand when it was spoken.

Click to hear spoken output from this program

Sending out the data

Maker channel

# Send SMS via IFTTT Maker Channel system ("curl -X POST -H \"Content-Type: application/json\" -d \'{\"value1\":\"" + desc + " - " +cpp_time_email+"\"}\' https://maker.ifttt.com/trigger/cpp/with/key/<ENTERYOURKEYHERE")
print "\n"

Finally — and most usefully — I send the description and the duration values to IFTTT via the Maker channel. This then allows me to use any of the IFTTT channels to deliver the data however I wish. I can send it via email, text message, iOS notification and more. There are other online systems like iFTTT can be used here, as well. Most of them operate by making a REST call to the URL of the API like this, so it is typically easy to send the data using a relatively simple CURL command. I found that formatting the command, along with all the properly escaped quotation marks, was probably the most difficult parts of this entire program, Sometimes it is the simplest things that catch you up the most.

Future enhancements?

As I mentioned in Part 1, I really like the idea of passive data delivery and this project would lend itself well to that, I think. You could use this script, along with an Arduino or Raspberry Pi to present a “traffic light” interface where the lights would indicate the current travel time conditions. A simple RGB LED would probably suffice. I plan on during just this in the future using my Arduino YUN.

In fact, as a small step towards making this a reality, I ported this Python code to the Linux side of my Arduino YUN and — with the relatively easy additional installation of the simplejson module on the Arduino — I was able to run the code. These means I could then use the data to “bridge” to the Arduino side of the YUN, access any of the digital or analog pins and interface to nearly any device I might wish. My recent acquisition of a Raspberry Pi seems to indicate I could do the same with it.

Again, through the process of developing this simple program I have learned much that can server me in the future, which was exactly what I intended when I started out. I hope this has been both interesting and educational for you. There is surely a lot more that could be done with this program, but I think it was an excellent learning exercise and still today provides some useful data to me and my family.

 

Arduino Boards and Components via Amazon

Arduino Boards and Components via eBay

Monday, September 26, 2016

Event: FrankenPumpkin at Orchard Supply Hardware, Oct 1, 2016

I saw a flyer in this Sunday’s paper for the OSH FrankenPumpkin event for kids. OSH will provide the pumpkin, hardware and decorations for free.

This decorating idea is new to me and I think it is a very cool “maker” event for kids, too. Quite a neat idea to use hardware parts — of which OSH has many — as decorations

Check it out at your local store. There is a store locator on the web page.

160918 frankenpumpkin header

Want more ideas for your jack-o-lanterns this Halloween? Check out these books from Amazon.com

Wednesday, September 21, 2016

Project: Build an Arduino BLE-Enabled Indoor Air Quality Monitor via All About Circuits

I am very interested in projects which help me gain a better understanding of the world around me and monitoring my environment is certainly one of the more important tasks in that regard. I have been thinking about projects like this as builds for school-based maker groups so that they could monitor their own surroundings and perhaps find areas that need improvement, long before research projects through more official channels could even be mounted.

I would consider this an intermediate to advanced project with lots of components to be managed and coded for. I love the inclusion of Bluetooth LE (or any other wireless system) so that data can be read and monitored even at some distance. For me, all IoT devices need some way of sharing their data easily in real-time and Bluetooth is just one way to accomplish that. This would also allow for the creation of a dashboard to monitor data, in realtime, in some central, public, location, which would greatly help to make the data known to a wide variety and number of people.

This project is well documented and should prove to be an excellent source of new skills and knowledge for anyone who attempts it.

Build an Arduino BLE-Enabled Indoor Air Quality Monitor via All About Circuits

Build an Arduino BLE-Enabled Indoor Air Quality Monitor via All About Circuits

Like most folks, I sometimes wonder about the quality of the air I breathe. Naturally, I turned to the possibility of a project designed to measure the quality of the air in my own home.

Indoor air quality is not a simple concept to quantify. I assumed that too much CO2 (Carbon Dioxide) didn’t seem like a good idea. Furthermore, large concentrations of VOCs (volatile organic compounds) didn’t sound too pleasant either. So, I reasoned that these would be some important measures to have available.

IAQCore

After further investigation, I found only a few ICs available to build such a project around. I settled on the iAQ-Core module from AMS.
As I read more about measuring indoor air quality, I began to appreciate the complexity of the issue. The volatility of many compounds (the degree to which they vaporize into the air) depends, in part, on the temperature. Also, relative humidity can be a factor for some compounds. Thus, I decided to add temperature and relative humidity sensors to the project.

Of course, I wanted to use a microcomputer with a display to read sensor output. As an additional feature, I decided to include BLE connectivity to allow remote monitoring.

Read the entire article on All About Circuits

Air quality schematic

Get Arduino Parts and Books Here

Arduino/Genuino 101 via eBay

Arduino Boards and Components via Amazon Arduino Boards and Components via eBay

Sunday, September 18, 2016

Project: Victorian Ticket Dispenser via Hackster.io

Here is an old school project — using no electronics — but still a cool build in many ways. I suppose you could add some electronic bells and whistles to make it even cooler, but it is a great idea and a great build for when you want something retro, interesting and cool!

Victorian ticket machine

Victorian Ticket Dispenser

I was asked by my wife to create a "contraption" to help sell tickets [all proceeds to charity] for her book launch at the local independent bookseller. Since the novel is set in 1880 New York, I wanted to create a Victorian feel and opted for a true, gear-clanking coin mech over a electronic version. I Borrowed/stole heavily from this piece for the guts of the device. The overall feel had to be Victorian and inform the users about the launch. I had a great time doing this and learned a lot about gearing, brass and a method for affixing paper to wood that I will be employing from now on.

Friday, September 16, 2016

Help Create Maker Faire LA 2017! – Kick-off Meeting: Thu, September 22, 2016 @ 7pm

Help Create Maker Faire LA 2017! – Kick-off Meeting: Thu, September 22, 2016 @ 7pm

Help Create Maker Faire LA 2017! 

Please join Hackerspace LA for a meeting for all individuals and organizations interested in forming the leadership team for Maker Faire Los Angeles 2017! 

What is Maker Faire?

Maker Faire is a gathering of fascinating, curious people who enjoy learning and who love sharing what they can do. It’s a venue for makers to show examples of their work and interact with others about it. Many makers say they have no other place to share what they do. DIY (Do-It-Yourself) is often invisible in our communities, taking place in shops, garages and on kitchen tables. It’s typically out of the spotlight of traditional art or science or craft events. Maker Faire makes visible these projects and ideas that we don’t encounter every day

Time/Location

Thursday, September 22, 2016 at 7PM 
Marvin Braude San Fernando Valley Constituent Service Center
6262 Van Nuys Blvd. Van Nuys, CA
3rd floor conference room
RSVP Here

We will discuss the Maker Faire Los Angeles Draft Plan and how it might be developed into our final plan. As a guide, we’ll be working from the Maker Faire Playbook. Please review these PDF documents before the meeting.

Maker Faire Los Angeles Draft Plan 
Maker Faire Play Book

Please join with Hackerspace LA in creating this amazing Maker/STEAM/Creativity event right here in our own backyard!

Jorge Cornejo, Producer - hackerspacela@gmail.com
Douglas E. Welch, Co-Producer - douglas@hackerspacela.org

Thursday, September 15, 2016

My Project: Travel time bot with Google Maps API and Python - Part 1 - Finding the Data [Programming]

My son will be starting university in just under a week at a school where my wife is also a professor. This means they are going to be carpooling on many days. Since the school is about 60 miles from our house, it is important to keep tabs on the driving time as here in Los Angeles, these times can change dramatically over the course of the day. This need to track driving time gave me an excuse to dive back into programming — which I was never that fond of anyway — despite my 30+ years working in computer support and consulting.

Here is a step by step story of the project and where it stands today.

Get the Data

The first step, of course, is to figure out where I can get the real-time routing and drive time data. I immediately thought about Google Maps and found the Google Maps API pages.

Google maps api

It is free to sign up for a developer account, which gives you an API key for each request you make to the Google Maps API servers. For me, my usage of the API is well below the number of free API requests per day, so there is no cost to run my program and gather the data — even if I were to request it about once per minute throughout the day. If you re going to develop high-usage commercial application, you’d have to sign up for one of Google API payment plants, based on the number of requests you make.

The specific Google Maps API I am using is the Directions API, which allows you to calculate directions between locations and returns a JSON-encoded file with a wide variety of information. They provide a basic example URL to get you started and then there are a host of options you can fine tune for your specific needs.

https://maps.googleapis.com/maps/api/directions/json?origin=Disneyland&destination=Universal+Studios+Hollywood4&key=YOUR_API_KEY

After a bit of trial and error, I came up with an appropriate URL that I could request manually and see the results returned.

https://maps.googleapis.com/maps/api/directions/json?origin=34.178026,-118.4528837&destination=34.057835,-117.8217123&departure_time=now&key=YOUR_API_KEY

This URL returns this JSON file...

{
   "geocoded_waypoints" : [
      {
         "geocoder_status" : "OK",
         "place_id" : "EjE1OTE0LTU5MTggQ2Vkcm9zIEF2ZSwgU2hlcm1hbiBPYWtzLCBDQSA5MTQxMSwgVVNB",
         "types" : [ "street_address" ]
      },
      {
         "geocoder_status" : "OK",
         "place_id" : "ChIJLVP1K64uw4ARc3vu78f00B0",
         "types" : [ "street_address" ]
      }
   ],
   "routes" : [
      {
         "bounds" : {
            "northeast" : {
               "lat" : 34.1794199,
               "lng" : -117.8012388
            },
            "southwest" : {
               "lat" : 34.0452654,
               "lng" : -118.453076
            }
         },
         "copyrights" : "Map data ©2016 Google",
         "legs" : [
            {
               "distance" : {
                  "text" : "46.2 mi",
                  "value" : 74333
               },
               "duration" : {
                  "text" : "54 mins",
                  "value" : 3251
               },
               "duration_in_traffic" : {
                  "text" : "58 mins",
                  "value" : 3455
               },
               "end_address" : "9 Olive Lane Walk, Pomona, CA 91768, USA",
               "end_location" : {
                  "lat" : 34.0577836,
                  "lng" : -117.8221829
               },
               "start_address" : "5914-5918 Cedros Ave, Sherman Oaks, CA 91411, USA",
               "start_location" : {
                  "lat" : 34.1780256,
                  "lng" : -118.4530714
               },

[… more information snipped…]

So far, so good. 

No I needed to figure out how to get the data out of this raw JSON file and into a form I could use it. The fact is, the API returns a lot more data than I really need, so I am only pulling out 2 pieces of information that are important to me. 

Navigating JSON Data Structures

Of course, now that I have the raw JSON data — which is a series of dicts, keys and values — I had to figure out the structure of the data and which pieces I needed. This was one of the more difficult learning parts for me. I understood that the JSOn file was a nested database of data, but figuring out how to get an individual pieces of data took me some time. Once I figured it out, though, it opened the door to any API that returns data in JSON format — a huge leap in my programming knowledge.

Visualizing JSON Data

Looking at a JSON file, especially a large one, and trying to discern the structure of the data can be a bit difficult. Levels of indentation aren’t always lear and nested curly braces always give me a headache. (LAUGH) Thankfully I searched out a web page that would allow me to view the data in a more database-like fashion, clearly showing how each piece of data was nested in the file.

JSONViewer.stack.hu allows you to either load a JSON file by URL or copy and paste a JSON file into the viewer. You can then click the Viewer Tab to see the data in its hierarchical format — expanding and collapsing nodes as needed.

Jsonviewer

Finding the Data I Needed

After a short while poking around in the Google Maps API JSON file, I found the 2 pieces of information I was most interested in — the summary or route of travel and the duration_in_traffic field.

Gmaps json expanded

This information would allow me to tell my wife and son which route was suggested by Google Maps (which can change based on traffic, road work and incidents) and the travel time in both text format and seconds. I did some basic checking with the interactive version of Google Maps to make sure the travel times were similar and saw that the duration field was a bit optimistic in its estimates. The duration_in_traffic field seemed to present a more realistic estimate, especially based on our heavy traffic here in Los Angeles.

Ok, so now I had the data I needed, how was I going to manipulate and use this data to make something useful?

In Part 2 of this series, I’ll dive into my Python program and discuss how I learned to process JSON data in Python, how I learned enough Python to do this and more and show off my working, if extremely basic, program. 

Sample Screen Output from Python program
[DEW-Mini:~/Downloads/py] dewelch% python cpp2.py

Via CA-134 E and I-210 E
Duration: 58 Minutes
0 Hours and 58 Minutes

Congratulations! You've fired the cpp event
[DEW-Mini:~/Downloads/py] dewelch% 

In future parts, I’ll discuss how I presented the data via screen, text, email and voice and how I got the code running on an Arduino Yun in preparation for creating a passive information device that will visually show the current travel time via graphical icons, colored lights and more!

Tuesday, September 13, 2016

Project: Flickering Flame Lighting Effects with Arduino for Halloween and more!

Halloween is rapidly approaching, so I went looking for Halloween specific projects I could make and share with my fellow makers at Hackerspace LA.

Here are a couple of links, and a video, for creating realistic flickering flame effects using a few LEDS and an Arduino micro controller. I am thinking of putting together a couple of these for our jack-o-lanterns this year.

Realistic Flickering Flame Effect with Arduino and LED's by TheArduinoGuy 

Flickering led

In this project we will use 3 LED's and an Arduino to create a realistic fire effect that could be used in a diorama, model railway or a fake fireplace in your home or put one inside a frosted glass jar or tube and no-one would know it wasn't a real candle inside. This is a really simple project suited to beginners.

Check out the entire project

Here’s a video, too.

…and a large collection of similar projects from How To Do Tip.com...

Flame effects projects

Learn more about Arduino

Arduino Boards and Components via Amazon

Arduino Boards and Components via eBay

Wednesday, August 31, 2016

Project: Simple Arduino-Controlled, No-Pump Plant Watering via Make

Now here is a project I could really use here in my own greenhouse. It’s basic setup is great, but it also allows a lot of room for expansion and improvement, so it is a great learning project, too. I think it is always important to start with a simple project and experience some success so you are more inclined to fiddle with it and improve it in the future.

For me, I could see adding additional sensors for multiple pots, a solenoid value for controlling water from a hosepipe, alerts, alarms and more. What could you do with this project? Let me know in the comments!

NoPump 2

Simple Arduino-Controlled, No-Pump Plant Watering

I love this simple and clever design. It basically uses a microcontroller-powered servo motor to pinch a watering hose on/off on a gravity-fed plant watering system. No pump required. To control the system, the Norwegian maker, Eirik, used a SparkFun Arduino-compatible RedBoard. To tell when the plants are thirsty, he uses a $5 SparkFun Moisture Sensor.

Read the entire article

Build your own projects with these Arduino parts and more!


Arduino Boards and Components via Amazon

Arduino Boards and Components via eBay


Sunday, August 28, 2016

Software: Arduino IDE (Integrated Development Environment) Recently Updated to 1.6.11

On August 17, 2016, Arduino.cc upgraded their IDE (Integrated Development Environment) to version 1.6.11

Arduino ide

If you are working with Arduino microcontrolllers (or the host of Arduino compatible systems) you should check out the new software. There are no big feature additions in this version, but a lot of bug fixes and tweaks.

Here are some of the fixes made:

ARDUINO 1.6.11 - 2016.08.17

  • [ide]
  • Fixed a serious bug that prevented some 3rd party boards, installed through external index.json URL, to work correctly.
  • Fixed a bug in boards manager that, in some rare circumstances, could lead to remove bundled  tools when uninstalling a previously installed AVR core
  • builder: fixed regression about inclusion of files in subfolders of the sketch  (see https://github.com/arduino/Arduino/issues/5186 for details)
  • avrdude: reverted to version 6.0.1, until all discovered regressions are solved  (see https://github.com/arduino/Arduino/issues?q=is%3Aissue+is%3Aopen+label%3A%22Component%3A+Avrdude+6.3%22 for details)

Get the latest version of the Arduino IDE here

Learn more about Arduino with this books

Arduino Boards and Components via Amazon

Arduino Boards and Components via eBay