Back to Visualizing Course Data!

Jamie Mahoney Data, Visualization , , ,

After having worked on creating a badge system for universities over the past few weeks, I’ve now gone back to looking at how the massive amount of course data that I currently have can be visualized in a meaningful and useful way.

My first bout of visualization resulted in a series of A0 posters showing the links between all of the modules currently being delivered at the university. Whilst these visualizations are very useful for showing the complexity of course structure and relationships, it becomes fairly difficult to extract any information that is particularly useful. For example, the edges in the network denote a connection between two modules in terms of the award that the combination is delivered on. A collection of edges of the same colour show a group of connections for the same award, i.e. a group of modules delivered as part of one particular award.

As the next step in my on-going quest to make sense of all of this course data (and related datasets), I’ve decided to look at a different abstraction of the same datasets, this time looking at the connections on an award level. This is one level of abstraction higher on the scale of University -> College -> Faculty -> School -> Award -> Module. By changing to this level of abstraction, it means that a) there are far fewer nodes on the graph, making it easier to see the information and b) it is easier for people to relate to an award (i.e. more easily recognizable what the node is referring to) than it is at a module level. At the moment, the visualizations are considering data for awards that are ‘Active’ i.e. have students on all levels and have a full-time, ‘traditional’ degree ‘feel’. I chose to do this as taking into account awards that are on their way in or way out, and part-time variations on a theme offered in a full-time course started to distort the data, flooding the networks with nodes and edges that are essentially replicas of other nodes and edges in the graph. Obviously the visualization exercise could be repeated for part-time or post-graduate courses, or to include them.

Narrowing the data down as described above, and running it through the trusted Gephi, this time using a circular layout algorithm, produces visualizations such as the following:

Links between awards for 2006-07

 

Links between awards for 2009 - 10
Links between awards for 2012 - 13

Each node around the edge of the graph represents an award that was active at the university for that particular year. With the university being relatively young in the grand scheme of things, the time-span between the first visualization (06-07) and the final (12-13) represent a substantial proportion of the university’s (in its current form) history. Award codes have been used as they are fairly short and remain similar in groups of awards offered by the same departments or schools. By doing so, the relative position of awards is more or less maintained in each visualization. For example. the pattern created between Computer Science awards and Media awards exists and can be easily spotted in each of the visualizations, even though the amount of awards in each visualization changes and the exact award codes of each award code may change. The full collection of visualizations can be accessed here: 2006 – 2007, 2007-2008, 2008-2009, 2009-2010, 2010-2011, 2011-2012, 2012-2013. These visualizations show three different sets of information: the amount of active awards for each year, the codes for the active awards and the relationships (where they exist) between the awards, i.e. where they share modules in common.

By taking into account the amount of modules shared between the awards and including this in the visualizations, we get a different view of the data. We can not only see where links exist, but also the strength of the links between the awards. Including the amount of modules shared between awards as the weight of each of the edges produces the following visualizations:

Weighted joins between awards 2007-08
Weighted joins between awards 2009-10
Weighted joins between awards 2012-13

The full collection of these visualizations can be found here: 2006-07, 2007-08, 2008-09, 2009-10, 2010-11, 2011-12, 2012-13

By introducing the weighted edges into the network, we can learn new pieces of information through the visualizations. Whilst the Computer Science – Media pattern exists across several years, we can see it move from being one of the more dominant links (2007-08 / 09-10) to being overshadowed by the amount of modules being shared by, for instance, Film and Television and Media Production and History & Social Science awards.

As well as making pretty pictures with the course data, the statistics associated with these networks can also be analyzed, but that will be the focus for my next blog post.

 

Release the Badges!

Jamie Mahoney Badges , ,

After working on the badges system that I outlined in a previous post, it has finally reached a point where it is functional enough to be ‘released’. It should be noted, though, that it is neither fully functional ‘out of the box’ and is by no means a shining example of development practices at their best. The mini-project of looking at how a system such as the Mozilla Open Badges platform could be used in higher education has suffered tremendously from scope creep and the underlying code (at the moment) reflects this. Over the past few weeks I’ve been through the following phases:

  • Consider how Open Badges (or similar) could be used in higher education.
  • Create a prototype system to be used in higher education.
  • Develop a more stable system that could be used in a trial run within our university.
  • Develop the system in such a way that it could be picked up and used in a variety of institutions or situations with minimal reconfiguration.

Initially, I considered creating a small database to hold ‘objectives’ that need to be met in order to be awarded badges, along with the bare minimum of APIs in order to interact with the database and the Open Badges framework. After starting out down this path, I started to realise that there was far more potential in a badging system within a higher education institution than I had originally thought and began to think of more features that would be useful.

At this point, I started to develop an application rather than a group of APIs that would provide a far more usable method of managing badges and associated data. After discussions with others around the university and on the open badges mailing lists it seemed that there was potential to use badges (or something similar) to recognise skills developed at university, as well as extra-curricular activities that may form part of the HEAR (Higher Education Achievement Report) or similar. This led to me considering the reuse of any system I developed in other institutions, which altered the way I implemented a few features in the system.

In its current state, the application allows you to login via 2 alternative methods: local registration or via oAuth (which we use at the University of Lincoln). Obviously the oAuth settings would need altering if you were to implement it in this manner in your own institution. Once logged in, admin users (who currently need the admin flag setting manually in the database) are able to create badges, objectives for badges, create a source of badges (could be different departments within an institution) and mark objectives as being complete for a given user or set of users (the application calculates if users are then eligible for the awarding of a badge). When logged in as a ‘standard’ user (although admins can earn badges, too) you are able to see your badge ‘profile’, which shows badges that have been earned and are ready to be claimed, badges that are partially complete (i.e. you have met 1 of 2 objectives for the badge) and badges that have been completed and claimed by you.

Badges that are ready to be claimed can be sent to a backpack (currently beta.openbadges.org) or downloaded as an image, so that they can then be uploaded to another backpack. Badges that have already been sent to a backpack can also be downloaded as images, thus allowing you to add the badge to another backpack if you so wish.

There are still a few features that need to be added to the application and known bugs that need fixing, such as:

  • Feature allowing password reset needs completing
  • The ability for admin users to grant other users administrative privileges
  • A ‘single use’, first-run admin login, thus allowing the initial user to grant admin privileges to somebody. When combined with the previous point this will remove the need for users to manually edit records in the database.
  • There is a known bug with marking objectives as complete for a user ID that has yet to login to the system. Assuming the unique ID of a user is known (currently envisaged as being the staff or student number), objectives can be marked as complete for that user. If the user is eligible for a badge due to the completion of the correct objectives before logging in, the badge currently gets stuck in limbo, as it cannot be awarded as the application does not have the users email address. A simple fix would be to have a “I think i’ve earned this badge” button on the partially completed badges section of the user’s profile.
  • Functions within the code itself need tidying up, because of the amount of scope creep there is likely to be a few functions that are no longer used etc.
  • The user interface needs working on. At the moment ‘it works’.

Here are a few screenshots of the application:

Homepage
Secure sign on using University of Lincoln oAuth
Badge Profile Screen
Claiming a Badge

The code for the application is available on Github and is released under the GNU Affero General Public License, documentation for the application (such as it is) is also available on Github – I’ll be working on the documentation over the next couple of days. At present there are two branches to the repository: master and develop. The master branch *should* remain stable, while the develop branch will be used to implement the features, and fix the bugs, discussed earlier.

Now that the application is reasonably stable, we’re starting to think of ways that we could trial the application. At the moment it looks like a couple of potential ways are to mimic what may appear in people’s HEAR reports for extra-curricular commitments etc, or perhaps recognizing skills that are developed and presented in non-assessed environments, such as workshops that help develop practical skills etc. However, with the end of the student year fast approaching, it is more likely that I will continue to tinker with and improve elements of the system before trialing it in some form or another from September onwards, with the return of the students.

APMS Progress Update

Allister Homes Case Study

Two servers, one for live use and one for testing, have been installed in our data centre and are being remotely configured by Worktribe. Worktribe provide the virtual server to us as an OVA file that is then installed and hosted in our existing virtual server environment. Almost all of the support and maintenance will be done by Worktribe remotely, which is a different approach to most of our systems though should be straightforward. Active Directory integration, which will allow users to log on to APMS using their standard University username and password, is also being configured.

We are starting to see parts of the system coming together now, and have been able to do some end-to-end testing of selected processes.

The vast majority of fields and workflows for creating short courses, standalone credit and ‘normal’ programmes have been created. This is where testing has been focussed. Worktribe still have a little more to do with preventing changes to programmes once they have been validated, but it is encouraging to see some significant progress and to be able to run through from proposing a programme to getting it validated.

Here’s a sneak preview of what you currently see when first proposing a new programme. (There may, of course, be some change by the time the system goes live.)

The project team are all very pleased that another instance of the system has been set up to be used specifically for populating our existing programme and module data. The population exercise has begun, which in itself is a major undertaking, and towards the end of the project the data will be moved across to what will be the live system. Starting population with real information at this point brings an added benefit that bugs are identified early, along with any suggestions for tweaks.

In addition to the programme proposal and validation work mentioned earlier, Worktribe have also put in place the mechanism to ‘roll over’ programmes from one academic year to the next. This is needed so that Agresso Students can be populated with instances of programmes for each academic year, and will also form the basis of how modifications are made to programmes.

After a few discussions between ICT’s server specialists and Worktribe, an appropriate way of backing up the system has been agreed. Although we hope that the worst will never happen, it’s essential we have backups in case of a major failure that means we lose the server or its data. The APMS server will be backed up to another server via a share, and from there backed up to an off-site facility. The backup routines will run every day, with Monday to Thursday backups kept for a week and Friday backups kept for a month.

ICT’s Application Support Team attended a training course about DTC (Data Translation Centre) – a tool in Agresso Students used to import information to the system in a support manner. This will be the way that programme and module information is imported from APMS to Agresso Students, and our Application Support Team are now able to configure and support the technical implementation.

The main areas Worktribe is now tackling are

  • how the marketing information about programmes is stored in the system and made available to be published on the Univerity’s website
  • how the system will handle programme modifications
  • how deletions/suspensions of programmes will be managed and the information archived
  • how records will be ‘locked down’ once validated but certain fields are still made available for editing.

I have started thinking about how to get information about the system out to a wider audience when we get nearer its launch. A lot of people, especially academics, will need to know about and use it so it is important to strart thinking about how to get the message out. To that end I have started talking to the Internal Comms team, and we have started work on a simple communication strategy.

In order to allow Worktribe more time to concentrate on core functionality at the moment, access to the API will come later but will still definitely be made available as part of the project.  We’ve seen the first XCRI-CAP outputs from the system which Jamie has had a look and provided feedback to Worktribe. At this stage the XML is produced by manually selecting a programme and choosing an option to export the XML file, but later will be available via a URL. Worktribe is making some refinements, with the aid of the XCRI-CAP validator, and we should see the next iteration soon.

Designing a Badge System for Universities

Jamie Mahoney Badges, Data

Over the past week or so we have started looking at the Mozilla Open Badge platform and how it could, possibly, be applied to use within HEIs (and beyond). It started to become apparent, through reading other peoples blog posts and reading through mailing lists, that there is already some work going on to award badges for completion of modules of study or achieving particular learning outcomes. After looking at what information is required to award a badge using the Open Badge framework, I created a design for a platform that can be picked up and used within another institution or context with minimal customisation required. The purpose of this blog post is to document the decision processes involved and to describe and show the resulting design.

The Basics

The basic premise of the system is fairly simple: create badges and award them to individuals who meet the criteria for each badge. These awarded badges can be ‘baked’ (to include the relevant information) and sent to individual’s ‘backpacks’ – be that the OpenBadge backpack or an internally hosted version.

This suggests that data pertaining to the following is required: Badges, Objectives and Earned Badges.

Adding A Few Extras

If the objectives required for badges are to be at least based on outcomes of programmes of study (be that at a full degree award level, or one constituent module within a degree), then it may be necessary, or at least preferred, to categorise the objectives within the badging system, or to add types of objectives to the system. Programme outcomes, if taken from our own programme specification documents can be split into at least three different types: Knowledge and understanding; Subject-specific skills and attributes; Transferrable skills and attributes. However, this categorisation of objectives may not hold true for other HEIs that may wish to use this code, and almost certainly would not be applicable if the code was used outside of the education sector. So objective types also have to be stored within the system.

When considering (still within a HE context) how the badging system might be expanded beyond learning outcomes of programmes or modules, you begin to open up the potential of including other data sources into the mix, such as library data, participation with extra curricular activities etc. For this reason, I’ve also included a badge source to the data, so that badges can be awarded from different sources and be differentiated as such through the data and its presentation to users. Again, another group of data to be stored: the source of a badge.

Designing the System

Data Requirements

Firstly, a list of the data required within the system:

  • Objectives for badges
  • Badges
  • Badges earned
  • Source of badges
  • Types of objectives

Doing the whole relational database thing on the data results in a structure something like:

  • Objectives
  • Objective Types
  • Badges
  • Badge-Objective Links
  • Badge Sources
  • Completed Objectives
  • Awarded Badges

A quick Entity Relationship Diagram (including suggested links to possible external data sources) would look something like this:

An Entity Relationship Diagram for a Badge System
ERD for a badge system

 

Obviously this data structure and suggested ERD is subject to change as I go through the development and prototyping stage.

With a data structure in place, mapping out the flows within the system and designing and implementing the APIs to do the data input and output are the next two steps in the process!

Data & Process Flows

There are several obvious functions that must exist within the system to allow the awarding of open badges to individuals. These functions include:

  1. Storing badge information
  2. Storing objective information
  3. Storing objective types
  4. Linking badges and objectives
  5. Storing various sources of badges
  6. Tracking objective completion for users
  7. Checking badge completion
  8. Storing completed badges for users
  9. Creating assertions for completed badges
  10. Presenting badges to a user through email
  11. Sending badges to a user’s backpack

In order to make the system compatible with existing data sources outside the bounds of the system shown above, the following API functions will need to be externally accessible:

  1. Add badges to the system
  2. Add a new badge source
  3. Add objectives to the system
  4. Add objective types to the system
  5. Uploading a badge image to the server
  6. Mark objectives as completed
  7. Search for existing objectives
  8. View completed badges for a user
  9. View completed objectives for a user
  10. View existing badges
  11. View existing objectives
It should be noted that these lists are only indicative of the functions to be offered by the system and will probably change while I’m developing it.

A quick data flow (ish) diagram shows how the data may flow internally within the system and the data that may be required from external sources, for one of the main series of functions – marking an objectives as being completed, through to awarding the badge to a user.

 What Next?

Firstly, I need to put together a prototype of a system that resembles the one described above. The code will be available on GitHub and licensed appropriately for reuse. Then we need to decide how we *may* go about prototyping the use of the system and any further development that is required. There is no agreement to go beyond the research and prototyping of badges at Lincoln yet and the most likely use case is for awarding badges for extra-curricula achievements, such as participation in workshops and those identified by the Higher Education Achievement Report (HEAR).

Academic Programme Management System Data and Outputs

Allister Homes Case Study

In my last post I gave an overview of the APMS project so far, and how the work with Worktribe is going.  This time I thought I would talk a bit more about some of the information that will be coming out of the system.

Diploma Supplements

One of the driving factors behind the project is diploma supplements.  In order to be able to automatically produce diploma supplements with meaningful information about the programme of study, rather than just a very basic outline, the system needs to store those details about the programme and combine it with student details to produce the document.

Since APMS will be the main source of programme information and approval processes, the  programme information needed for diploma supplements will naturally be stored in it.  Details about students and their results are stored in Agresso Students, so the technical gurus at Worktribe are working on routines to bring that data in to APMS and produce the diploma supplements in it.  The full details of exactly how all of this will work in practice are still being ironed out, but ultimately we will be able to produce diploma supplements en masse, and possibly give them to graduates at the same time as their certificates.

Key Information Set

The Key Information Set (KIS) is certainly causing a fair degree of consternation in the sector at the moment.  Despite HEFCE’s and HESA’s assertions that organisations should already know all of the information that the KIS summarises, and indeed some of it is already reported and those sources will be used for its production, the reality is that the information is not necessarily held in a way that means it can be returned electronically.  In fact, some of the technical detail about how the extra data will be returned have only just been released by HESA.  An article published by Guardian Professional talks about some of the issues facing institutions.

As you probably already know, a significant part of the KIS data set relates specifically to programmes (courses in KIS terminology).  Institutions must submit this data set to HESA in a defined XML format.  Since at Lincoln we will be collating programme information in the new system, and much of it until now has not been recorded in a way that lends itself easily to producing the submissions as required, it made sense from the outset to include recording the necessary information and outputting in an appropriate manner part of the system requirements.

Like Diploma Supplements, the full details of exactly how it will work have yet to be discussed in their full detail, but I will give updates as we go through the project.  Technical information about the KIS, for those interested, can be found here.

HEAR

The Higher Education Achievement Report (HEAR) is intended to be a method of reporting and making available information about student achievements during their time at the institution.  JISC’s description is that it is “intended to provide a single comprehensive record of a learner’s achievement at a higher education institution. It will be an electronic document, which will adhere to a common structure and be verified by the academic registrar or equivalent officer. Institutions may also choose to issue a paper document.”

From my perspective, the HEAR will take what is currently in Diploma Supplements and take it further in the following ways:

  • It will be an electronic ‘document’
  • It will provide more details about types and weightings of assessments
  • It will also includes information on activities carried out by the student which do not carry credit towards their award, but which can be verified by the institution.

A HEAR document will be submitted by institutions for each student in XML format (specification version 1.0c is the current version at the time of writing).

The technical details of the HEAR, and exactly how it will be submitted and delivered, are still being developed at the moment; as a result we are understandably not actively working on this yet.  However, with the Academic  Programme Management System becoming source of our Diploma Supplements, and the HEAR being the next extension of those, it obviously makes sense for APMS to also deliver HEAR documents.  HEAR production was therefore included in the specification and will be delivered in due course.

XCRI-CAP

XCRI-CAP production is, of course, a vital piece of the JISC Course Data Stage 2 programme.  As part of APMS, Worktribe will produce an XCRI-CAP feed from the system.  It’s likely that we will cache this somewhere on our main University website rather than have to arrange for public http access to APMS, and of course the public facing website is designed to handle loads that APMS is not.

Worktribe is working on producing this feed which will replace the existing mechanism we have for producing our current XCRI-CAP feed provided at http://www.lincoln.ac.uk/xcri.

Programme and Module Specifications

At certain points we need to be able to produce programme and module specifications in document format, for example Word (.doc) or PDF (.pdf) files.  Occasions when these might be needed include:

  • During validation panel meetings
  • For auditors to review
  • To include on the website with programme marketing information

In time we may be able to persuade people to view the information in APMS rather than in a document, and there might be some potential for including the full programme specification as another web page rather than as document downloadable from the marketing information page for a programme on our website.  In the meantime, however, Worktribe’s system give us the ability to produce these documents as needed.

Data for ON Course

In addition to the XCRI-CAP feed and KIS information, the ON Course project at Lincoln will make use of Wortribe’s APIs to extract programme and module information from APMS and do interesting things with it!  The APMS project is focussed on improvements to course data flows within the institution; ON Course seeks to find innovative ways of exposing that data publicly and, combined with other existing sources of open data, provide examples of how it can be used.

At the moment we’re accumulating test data, designing APIs and experimenting with tools and ideas.  We are still undecided about whether we will focus on an outward facing application for prospective students or develop an application that has greater benefits for curriculum design and business intelligence, but recent conversations and experiments have us leaning towards the latter.

The documentation about these APIs will be made available at http://data.lincoln.ac.uk.