Getting to the bottom of statistics: Software utilizes data from the Internet for interpreting statistics

The "Explain-a-LOD" tool that Paulheim developed accesses linked open data (LOD), i.e., enormous compilations of publicly available, semantically linked data accessible on the Internet, and, from that data, automatically formulates hypo­theses regarding the interpretation of arbitrary types of statistics. To start off, the statistics to be interpreted are read into Explain-a-LOD. Explain-a-LOD then automatically searches the pools of linked open data for associated records and adds them to the initial set. Paulheim explained that, "If, for example, the country "Germany" is listed in the corruption-index data, LOD‑records that contain information on Germany will be identified and further attributes, such as its population, its membership in the EU and OECD, or the total number of companies domiciled there, generated. Attributes that are unlikely to yield useful hypotheses will be automatically deleted in order to reduce the volumes of such enriched statistics.

Once that preprocessing has been concluded, Explain-a-LOD proceeds to the second stage and automatically formulates hypotheses, based on the enriched statistics. The methods employed include simple correlation analyses, as well as other methods for recognizing regularities in statistical data, in order to allow formulation of more-complex hypotheses covering more than just a single attribute. Users will then be presented with the resultant hypotheses, in the form of, e.g., phrases, such as "OECD-member countries have low corruption indices" if any positive correlation exists between the attribute "OECD‑member­ship" and the target attribute, "corruption index," regardless of whether the original statistics contained any references to countries' OECD‑membership, or lack of it. That background knowledge will be automat­ically taken into account by Explain-a-LOD.

Surprising and useful hypotheses

Paulheim and his colleagues have thoroughly tested their approach on various sorts of statistics, including Mercer's standard-of-living study and Trans­parency International's corruption index. Paulheim noted that, "What one obtains are mixtures of obvious and surprising hypotheses, such as "cities where tempera­tures do not exceed 21°C during the month of May have high stan­dards of living," "capital cities generally have lower standards of living than other cities," or "countries that have few schools and few radio stations have high cor­rup­tion indices." An evaluation of the results by test persons verified that impression. Paulheim added that, "The test persons perceived the resultant hypotheses as largely surprising, as well as nontrivial, and, very frequently, as useful." However, the test persons had serious doubts regarding the trustworth­i­ness of the resultant hypotheses, which, Paulheim noted, was also attributable to the unsatisfactory qualities of some of the data contained in the open-data cloud.

Explain-a-LOD has been presented at several international conferences over the past few months. The tool received the "Best In-Use Paper" and "Best Demo" awards at the Extended Semantic Web Conference 2012 held on Crete in late May. Several upgradings of Explain-a-LOD, among them implementation of further attribute-generation algorithms and facilities for accessing further data pools from the LOD‑cloud, are planned for the future.

Further information: http://www.ke.tu-darmstadt.de/resources/explain-a-lod