User Tools

Site Tools


chicoutimi_2016:enron_email_example

Visual Analytics Course

Instructor: Guy Melançon (email: Guy dot Melancon at labri dot fr)

Chicoutimi Summer 2016 roadmap / May 3, 3pm course

Visual Analytics by Example

Visual Analytics is (as defined by Cook and Thomas 2006) is the science of analytical reasoning facilitated by interactive visual interfaces. (See the InfoVis Wiki)

The Enron Email dataset / Step by step manipulations

The domain and questions

The Enron Email datasetcontains about 0.5 million email exchanges (messages) collected from 150 individual email account. The messages concern more than 30 thousand person involved in these exchanges. The motivation for inspecting such a dataset is in a sense similar to that of the actual Panama Papers affair.

There are a number of questions one can ask about this data, with most probably the intention to identify the most prominent actors – those by whom the troubles may have emerged, let's say.

The data

The data, the one found on the Stanford Network Analysis Project is organized as a network (a graph) where:

  • nodes correspond to email addresses appearing in the data;
  • edges links two adresses , address sent at least one email to address ;
  • edges are undirected;
  • non-Enron email addresses act as sinks (no outgoing edges) and sources (no incoming edges) in the network as the network only covers communications between Enron email addresses.

Remark Can you think of other types of networks, or any other type of dataset one can build from the raw material describing all email exchanges (the 0.5 milloin full messages)?

Remark How does the choice of building the network the way it is on SNAP limits the questions we can hope to answer?

  • Edit the file so as to replace the header lines by a single one, tab-separated fields “From To”
  • Launch the Tulip agent, run a new (void) instance of a graph
  • Import the downloaded txt file, creating a single edge for each , entry (one per line)

The visuals

What do you see? (The layout)

  • The imported data is instanciated as a graph, laid out randomly
  • Rename the graph “Enron email” (edit the name “graph_0” in the left panel
  • Re-layout the graph using a simple “Circular” layout (look for the Layout tab in the plugin list on the left)
    • The “hole” in the center of the view indicates some players have but a few connections; but the view does not allow to get much insight on how/why that happens
  • Run the FM3 layout (spring embedder)
    • The view reveals much more information about how exchanges take place
    • The view shows a larger components, with a bunch of satellite components we will discard
    • Select the larger connected components and creates a subgraph from it
    • Rename the subgraph “Larger connected component”

The (network) analytics

We may expect major actors as having exchanged more messages than others. A way to spot these players is to compute the degree of nodes (the number of edges connected to a node).

  • Compute the degree of actors
    • Run the “Degree” plugin (left panel)
    • The resulting value is (temporarily) stored in the viewMetric property
    • Store it into a separate metric called “Degree”
  • Create a histogram view of the degree on nodes
  • How strange! Most actors have but a few connections. Compute how many actors have degree at most
    • Use the command line python Tulip utility
    • 25756 nodes have degree at most 10
    • 11211 nodes have degree at most 2
    • 1347 nodes have degree at least 100
  • Select those nodes of degree at least 100
  • Go back to the node-link view and explore where these nodes “live”

More visuals (Color mapping)

  • Let's map degree onto node color so we can simply “see” how degree distributes in the network
  • Play with the different parameters offered by the plugin
    • Use the “RdPu_9_from_ColorBrewer.org.png” colorscale
    • Can you figure out what the “Uniform”
  • Customize background and edge color*
    • Turn the background view to black
    • Turn the edge color interpolation on
    • Turn off node labels

More analytics (Actor Centrality)

  • Compute the “PageRank” index for actors (email addresses), that is the same index Google uses to rank web pages.
    • One may expect actor with high page rank to be referred to (or to refer) more often in the network.
  • Select the single actor with maximum page rank
  • Go to the spreadsheet view and:
    • Make its size 10 times as big as other nodes (map it to (10,10,10))
    • Map its size to “2D circle”
    • Map its bordercolor to green
  • See where this actor sits in the network?
    • This suggests that something interesting may be taking place between the core part of the network (the “center”) and a denser component …

Make it simple and readable (Aggregating things)

  • We'll here try to have a simplified view of the connectivity structure in the network
  • Run the “Louvain” algorithm
    • This algorithm identifies subgroups in the network (more about this during the course)
    • The algorithm does nothing but assigns each node an integer (the subgroup number it belongs to)
  • Let's compute each of these group separately
    • Run the “Equal value” plugin: it does what it says, it forms subgroups by putting all node having a same value in a same subgroup (using the viewMetric property unless stated otherwise)
    • All subgroups end up begin subgraphs of the graph on which the plugin is run
  • Have a look at the subgraph hierarchy
    • The original Enron graph
    • The larger connected component
    • The bunch of subgroups we just computed
  • Run the Quotient Clustering plugin
    • Make sure the “Larger Connected Component” graph is selected
    • Make sure you select the “Layout quotient graph(s)” option
    • You get an extra graph in the hierarchy called “quotient of …”
  • Send the Quotient graph“ onto the node-link view (drag & drop)
    • This view provides even more insight than the plain node-link view did
    • It reveals how these subgroups interact with each other, some being peripheral, others being more “central” and somehow “bigger”
/net/html/perso/melancon/Visual_Analytics_Course/data/pages/chicoutimi_2016/enron_email_example.txt · Last modified: 2016/05/02 00:37 by melancon