Enipedia Maps

From Enipedia
Jump to: navigation, search
Enipedia data in Google Maps

This page documents our efforts at creating a Google Maps interface to the power plant data contained on Enipedia. Feel free to add any feature requests, bugs, etc to either this page or the discussion page.

Contents

[edit] Background

While we've created a Google Earth Visualization for the data on Enipedia to allow people to navigate it more easily, this has limitations in terms of interactivity, such as being able to select different subsets of the data. The interface below shows some of the work in progress that we're doing on addressing this and create an interface that users can tune to there needs. The code heavily borrows from what Nono has done to visualize data from Enipedia and other sources.

[edit] Features

[edit] Layers

Selecting Map Layers

Main Layers:

  • Map - The standard Google Maps street map.
  • Satellite - The standard Google Maps satellite imagery.
  • OpenStreetMap - This shows data from OpenStreetMap, a map of the world that anyone can edit. In some parts of the world, OpenStreetMap is more detailed than Google Maps. You can compare them side-by-side here.
  • NASA Black Marble - This is a classic cloud-free image of the earth at night, based on map here. Aside from just showing cities, this is an important map for energy as one can in some areas very clearly see the flaring of natural gas.

Transparent Layers - these layers are drawn on top of the layers described above.

  • ITO Map Electricity Distribution - The data is derived from OpenStreetMap, with the features of the electricity distribution grid being highlighted.
  • ITO Map Electricity Generation - Also based on OpenStreetMap data, this uses much of the same data, but emphasizes aspects such as the fuel types of the power generation facilities.

Combining the NASA Black Marble map with one of the ITO Map layers is recommended as it very clearly shows the infrastructures that supply the power for the lights.

[edit] Functionality

All of the functions listed below will override the previous functions that have been chosen. For example, if you search for all of the power plants within a country and then search by a particular owner, you will get all of the power plants of that owner in all countries.

[edit] Search by Country

Selecting Power Plants by Country

You can load all of the power plants for a particular country by selecting it from the drop down list. The countries are based on the list at Category:Country.

[edit] Search by Company

Autocompleting on Company Name

As you start typing in the owner, it will try to autocomplete the name, based on entries contained in Enipedia. The owners can be found within Category:Energy_Company. Subsidiaries, Joint Partnerships, etc are not dealt with explicitly, although you can often find them via the autocompletion feature. Some of our current efforts at documenting the ownership structure of companies can be found here and will likely eventually be incorporated into the functionality of the map.

[edit] Filter by Fuel Type

Filtering by Fuel Types

Once you have data loaded, you can filter by the fuel types

  • We allow you to turn on or off specific fuel types, or complete categories of fuel types:
    • fossil fuel (thermal power plants)
    • nuclear (thermal power plants)
    • renewables (wind parks, solar, wave, biomass thermal, geothermal etc.)
      • Landfill Gas and Waste to Energy (WtE) installations are counted as renewable, although clearly they are composed of both fossil and biogenic components.
  • In the menu, the numbers under the icons show the number of power plants found for each fuel type for the particular country selected.
    • Please note some double counting may occur (across different fuel types) if a plant uses several fuel types.

[edit] Load All Power Plants on Current Map View

It is also possible to load all of the power plants within the current view. You need to first select the "Advanced Menu" and then on the text as shown.

Advanced Menu
Loading All Power Plants within Current View

[edit] Download Data

Downloading Data

Once you have data loaded via any of the functions listed above, you can click on the "Advanced Menu", and then on the "Download Data" link in order to download the data in CSV format which can be opened in Excel.


[edit] Map

(See map as whole webpage)

  • The data used for the map is based on the live data on the wiki. Any changes to the wiki data will show up on the map once you refresh the page.
  • You can click on the Advanced Menu to load all of the power plants in the current viewing window.
  • Whenever markers are shown on the map, the Advanced Menu also contains a link to download the data in an Excel format.

[edit] Source Code

The source code is now up on github at https://github.com/cbdavis/enipedia-maps. This will likely evolve and branch as we figure out what's feasible and experiment with different options. Feel free to experiment with it and list bugs or suggestions for improvements or new features.

The code runs as an HTML page, so people can run it on their own server, or locally through a browser. The main dependency is that the code talks to a SPARQL endpoint on our server, so you need to have an internet connection to get it to work. The URL for the endpoint will likely change as this is part of an effort on Upgrading Enipedia.

The code is interesting for open data projects as it is able to query a SPARQL endpoint and then display and filter the data. Further functionality to be integrated can be found here.

[edit] Sandbox

A sandbox for experimenting with the code is here since the map is not wiki-enabled. New functionality tested in the sandbox nearly always makes it into the github repository and onto the site here. The only reason we have a locked-down version here is that we want to maintain a functional version for demos, but in general, we heavily encourage experimentation and quite often incorporate feedback and new ideas.

[edit] Issues & areas for improvement

[edit] Data

  • View/edit links do not work for powerplants whose name contains & or ' characters (no page for the former, bogus page for the latter).

[edit] Code

  • Internet Explorer
    • Internet Explorer 7, 8 and 10 seem to function almost ok.
    • The code doesn't work with Internet Explorer 9. The top menu bar shows up (at the bottom), and the map layers are not loaded.
    • The ie7-js library ("A JavaScript library to make MSIE behave like a standards-compliant browser") may help with these issues.
      • Initial tests lead to errors with IE 7
      • [1] may be useful for fixing the issue with markers not showing up in IE.
      • (fixed based on suggestions here) A quick look into it seems to show that the issue is not related to marker display but to the ajax query not firing. Setting the query type to POST instead of (default) GET seems to fix it at least for IE 7 (IE 9 not tested). IE 7 applies an old limit on querystring size, but anyway it's still good practice to use POST for arbitrary size data.
  • The CSV export format is not ideal due to issues with Excel and the fact that different delimiters are expected in different countries. European windows uses the semicolon as the delimiter since the comma is used as the decimal symbol, while other locations would use the comma as a delimiter. Excel also doesn't always property interpret UTF-8 files and a Byte Order Mark needs to be added, which is a technique that is not recommended.
    • While it's possible for someone to re-import the data with the correct delimiters and encoding, this isn't exactly a transparent process for many users. It would be great to have just download the data and then directly open it in the correct format.
    • A fix for this may be to use XSLT to directly convert from the XML output of the query results into Excel format (Office Open XML, see also here and here).
  • The download link shows up even if there's no data found for the query.
  • References to external javascript libraries cause security warnings. On one version of Chrome, it was observed that nothing loaded, and it wasn't obvious that content was being blocked. For the version of the maps on Enipedia, we should include the libraries locally. I'm not sure if there would be a solution for the scraperwiki version.
  • The code for country selection doesn't load all the time. Sometimes the page needs to be refreshed.
  • Need to upgrade SMW so we can use the sfautoedit API call to update wiki data directly from the map. Easy first step is by setting markers to be draggable so that people can update the coordinates. But this can also be dangerous, especially with many markers cluttering the map space, and may at least require a confirmation. It would also be unwise to use it until we can achieve synchronisation with the wiki, since it would imply editing data you don't know the current state of.
    • There should be some sort of explicit editing mode/sequence that helps to prevent confusion. Need to think about how to make people remember to save changes, how can they undo, etc.
    • sfautoedit is working with the current software but it seems to be tied to forms which implies that you cannot automatically edit fields such as primary_fuel_type or latitude/longitude that are in the template but not in the form. Another downside is that if you don't specify the point property during the edit it gets lost during the edit, just as when the map fails to load.
      • latitude/longitude will probably still be derivative properties of point since (it seems) that the maps interface in Semantic Forms can only write to a single property. Conversion of point values to lat & lon is done automatically when the page is saved. The fuel_type and primary_fuel_type do need to be refactored to something more sane, perhaps like primary_fuel_type (1 value), secondary_fuel_type (1 value), other_fuel_types (multiple values). This would have the added benefit of getting rid of the permutations of results that sparql queries generate when one plant has multiple values for fuel_type specified.
        • (fixed, also slight rounding/truncation occurs, but this has not been seen to introduce noticeable differences) latitude/longitude do not actually get updated on save when point changes (see Malassis_Powerplant or Motz_Powerplant)
          • It's not clear yet what's causing this. Examining Malassis_Powerplant and Motz_Powerplant shows that the lat/lon seem to be truncated versions of the value set for the point. It's strange the point would be set, and not the lat/lon since these are all set from the point field in the template. Also some tweaks have been made to the cache settings to (hopefully) fix an issue where the point on the map was not updated when the coordinates were changed. The combination of these may be causing the problem, or there may be a deeper issue.
          • Special:ExportRDF/Malassis_Powerplant shows that the values for lat/lon are directly based on the value for the point. However, in virtuoso, these get truncated to what is shown in the factbox at the bottom of the page. This seems to indicate that Special:ExportRDF may not use exactly the same code as that which synchronizes the RDF on the wiki with the triplestore.
          • Is it possible to remove the latitude/longitude fields from the template if the RDF properties are definitively set from the Point field? Having them remain with their outdated values can be confusing.
            • The majority of pages have been cleaned up via a bot script. One more pass is needed to make sure all pages have been fixed (seems to be memory leak on RSemanticMediaWikiBot code).
        • (fixed) There are also a number of power plants that still do not have latitude/longitude at all, though they should have been updated during upgrade. Those just do not show in the latest sandbox version. For example, looking at Guadeloupe island in the French West Indies displays only 3 plants while there should be at least 17 if latitude/longitude were not mandatory.
      • We also need to see what happens when people edit a large number of points. If you edit 10 points, the code behind the scenes will save data to 10 pages. If this takes a long time to run, it would be possible to set up a queue system with a bot to process the edits.
  • A somewhat related concern is that there are instances of power plants occurring at the same exact coordinates due our use of geocoding for power plants that did not have any coordinates specified in the original data. We may want to consider using a bot to offset the points so that people can differentiate between them when they are all the way zoomed in. In these cases, we know they're incorrect anyway, but (hopefully) in the correct general region, so this shouldn't mess up things. Modifying data because of a display issue is however not the best practice. Using an advanced marker as in the sandbox may be a better solution, though it could be a bit smoother.
  • Values for data derived from the carma.org data set should be optional in the sparql queries that get data for the map. This way if we only know the name and location of a power plant it will show up on the map. Since the icons are scaled due to the output of the power plant, we need to think of some default value to use if nothing is known. Eventually the maps interface should evolve (split off?) into a tool that allows for highlighting data with missing property values.

[edit] Appearance & Enipedia

  • We need an extra icon to distinguish Fuel Oil from Diesel Oil (Gas Oil in English mostly...)
  • Width of control panel is too wide for embedding. Should look into moving the google maps controls out of the way.
  • Control panel gets in the way of some of the icons. Fix by setting slightly bigger bounds?

[edit] Functionality

  • The SlickGrid javascript library could be useful for integrating an interactive tabular view that could allow for another way of filtering the data. A very cool example is here. This could be adapted to allow people to explore other data sets, such as the EU-ETS. Functionality such as grouping would allow selection of plants owned by a set of companies.
    • We've got a proof-of-concept running - this needs to be placed on scraperwiki. With a test on EU-ETS data, it's quick for navigating data (once it's loaded), but grouping can be slow. For NL data, grouping by account holder takes a few seconds. Doing the same for the whole dataset takes considerably longer. See here for development notes.
  • For some of the ideas below, may want to split the functionality into two interfaces - one simple default (similar to what we have now), and another for more advanced types of filtering. See the code on scraperwiki for initial work on being able to switch between a single and advanced menu.
  • Add selection on capacity installed (for example > 1 MW, > 10 MW, > 100 MW > 1000 MW
  • Add selection for "status"
    • I don't think that we have that much data on power plant status currently.
  • Add a link to this view on country pages, of course viewing only that country data
  • Add intuitive clicks -- navigate the map, click on a country and get the power plants
    • An easier option is to only select by bounding box (requires prop:Point to be split into lat and long properties). Being able to click on a country requires that we have polygons defined for country borders and then have some way to calculate which polygon the mouse is in. I'm not sure if there's an easy way to do this within Google maps. The bounding box approach would be more robust as it would allow people to get data from different zoom levels.
  • Match Google's map functionality -- navigate to a place and show the power plants around it, depending on your own selection
    • or like Analytics functionality - clickable world map with statistics, with "power plant footprints" displayed, as shown on country pages
  • Show selected statistics (again -- set by user preference?) on the view
  • Click button to get top x powerplants within current viewing window.
    • Should limit this to a range of zoom levels so that people don't try to get everything at once and crash their browser (is it possible to load everything at once and still have the map function?).
    • The code on scraperwiki currently is set up to get all of the power plants within the current map bounding box.
  • Add ItoWorld map tiles, based on code here. The version on scraperwiki allows for this.
    • This functionality is currently commented out in the scraperwiki sandbox. The main issue is that we should use a background (OSM Mapnik?) that allows us to very clearly see this layer. It's hard to see with the Google Maps satellite tiles. As an advanced option, we should allow people to show tiles from different mapping services.
  • Something with D3.js and/or Crossfilter?
  • Use circles to indicate emissions/output similar to what we've done with the Google Earth visualization - does this work for 1000's of points or should we use clustering instead? Could also just scale icons into different bins for a cleaner interface by using MarkerImage properties.
    • The current version scales the icons quite nicely. As an advanced option, it would be nice to choose which property the icons are scaled to.
  • Allow for viewing external data sets separately from Enipedia data. Also allow for comparisons, such as shown here and here.
  • Information display in popup windows can be improved - may want to try running DESCRIBE queries to get all the data related to the plant. This all depends of neede use: when comparing two datasets, having a small set of equivalent properties is more effective. On the other hand, if you use the map as an alternative interface to explore data, you may indeed want to see a more complete summary.
  • Eventually the text field with autocompletion to query by owner company would be aware of which companies are owned by other companies, i.e. if you select Vattenfall, it would also show the power plants of Nuon. We don't have the data needed to do this yet, but if we did, a query using a property path would be able to traverse the ownership hierarchy.
    • The latest CARMA release (v3) seems to use top level company names (groups) instead of subsidiaries as previously. This could help in identifying companies from the same family. Perhaps set a parent company property?
Personal tools
Namespaces

Variants
Actions
Navigation
Portals
Advanced
Toolbox