PracticalWeb Ltd - websites that work for you

Data.gov.uk

sean — Fri, 26 Feb 2010 19:10:12 +0000

I've recently been working on a very exciting project - building the Drupal front end of http://data.gov.uk/

It's great to be working on another opendata project using Drupal - and again working with really smart people committed to opening up data.

This is big project with real ambition - and I go to meetings where I get to learn stuff :-)

I've been so excited by the project I've created some open source code inspired by this and the London Datastore work I did earlier - the great thing is that the management are pushing for future open source code to be officially released as part of the project.

Drupal Module for CKAN Integration

sean — Wed, 24 Feb 2010 22:55:47 +0000

Here's a basic Drupal Module that pulls CKAN data into Drupal.

It provides a simple search interface - when the user views a CKAN package for the first time a corresponding node is created.

When this node is then viewed Drupal loads the package details - I'm only displaying notes and title here as I've run out of time tonight - but the rest of the data is there.

You can then have user comments, voting, etc on these pages.

This is fairly quickly put together code and it's incomplete - but it should be enough to get people started - if there's interest I'll carry on with this so please add comments.

I've created a github repository for it and I'll start using the issue tracker there.

http://github.com/practicalweb/drupal-ckan

<?php
/**
 * @file
 * A Module to Integrate CKAN and Drupal.
 * 
 * This works by 
 * 
 *   1. providing a serach and listing interface to CKAN with links to this Drupal site
 *   2. using a wildcard menu handler to match these links in the first instance
 *   3. this loads the packages and creates matching nodes (storing the ckan name)
 *   4. the node has a url alias which overrides the wildcard match
 *   
 *   Now we have Drupal nodes which can have user commments associated with them, fivestar votes etc...
 * 
 * @author Sean Burlington <a href="http://www.practicalweb.co.uk
" title="www.practicalweb.co.uk
">www.practicalweb.co.uk
</a> * @copyright PracticalWeb Ltd
 * @license <a href="http://opensource.org/licenses/gpl-license.php" title="http://opensource.org/licenses/gpl-license.php">http://opensource.org/licenses/gpl-license.php</a> GNU Public License
 * 
 */

function ckan_menu(){
    $items = array();

    // link to the search form page
    $items['ckan_search'] = array(
    'title'           => t('Search CKAN'),
    'page callback'   => 'drupal_get_form',
    'page arguments'  => array('ckan_search_form'),
    'access callback' => TRUE,
    );

    $items['ckan/search/%'] = array(
    'page callback'     => 'ckan_search_results_page',
    'page arguments'    => array(2),
    'access arguments'  => array('access content'),
    'type'              => MENU_CALLBACK,
    );
    
    $items['ckan/data/%'] = array(
    'page callback'   => 'ckan_new',
    'page arguments'  => array(2),
    'access arguments' => array('access content'),
    'type'            => MENU_CALLBACK,
    );
    
       
    return $items;
}


/**
 * Gets the ckan Object
 * Initialises it in one central place reusing it if needed.
 *  
 * @return Ckan $ckan
 */
function ckan_ckan(){
    static $ckan=null;
    if (!$ckan){
        require_once(dirname(__FILE__) . '/ckan.php');
        $ckan=new Ckan(variable_get('ckan_site', 'http://www.ckan.net/'));
    }
    return $ckan;
}

/**
 * Implements hook_form
 * 
 * @param Array $form_state
 * @return Array $form
 */
function ckan_search_form(&$form_state){

    $form = array();
    $form['q'] = array(
                   '#type'          => 'textfield',
                   '#default_value' => '',
    );
    $form['submit'] = array(
                   '#type'          => 'submit',
                   '#value'         => t('Search CKAN'),
    );

    return $form;
}


function ckan_search_form_submit($form, &$form_state) {
      // The search form relies on control of the redirect destination for its
  // functionality, so we override any static destination set in the request,
  // for example by drupal_access_denied() or drupal_not_found()
  // (see <a href="http://drupal.org/node/292565" title="http://drupal.org/node/292565">http://drupal.org/node/292565</a>).
  if (isset($_REQUEST['destination'])) {
    unset($_REQUEST['destination']);
  }
  if (isset($_REQUEST['edit']['destination'])) {
    unset($_REQUEST['edit']['destination']);
  }

  $form_state['redirect'] = 'ckan/search/'. trim($form_state['values']['q']);
  
}

function ckan_search_results_page($search){
//    return $search;
    try {
     $ckan = ckan_ckan();

     $results = $ckan->search(check_plain($search));
    } catch (Exception $e){
        return $e->getMessage();
    }
    return theme('ckan_search_results', $results);
}

/**
 * Create the node the first time this package is seen.
 * 
 * This creates a specific URL which override the wildcard one subsequently
 * 
 * @param String $ckan_name
 */
function ckan_new($ckan_name){

    $ckan = ckan_ckan();
    try {
        $ckan_data = $ckan->getPackage($ckan_name);
    } catch (CkanException $e){
        drupal_set_message($e->getMessage(), 'error');
        drupal_set_title("Error");
        return "Error" ;
    }
    $node = ckan_create_node($ckan_data);
    drupal_goto('node/'. $node->nid);
}



function ckan_create_node($ckan_data){
    $node = array(
              'title'   => $ckan_data->title,
              'uid'     => 1,
              'body'    => $ckan_data->name,
              'promote' => 1,
              'path'    => 'ckan/data/' . $ckan_data->name,
              'type'    => 'ckan',
              'comment' => 2,

    );

    if ($node = node_submit($node)) {
        node_save($node);
    } else {
        drupal_set_message("Failed to create node for package.");
    }
    return $node;
}


function ckan_node_info() {

    return array(
    'ckan' => array(
      'name'           => t('CKAN Package'),
      'module'         => 'ckan',
      'description'    => t('A package of Open Data.'),
      'has_title'      => TRUE,
      'title_label'    => t('Title'),
      'has_body'       => TRUE,
      'body_label'     => t('Package Description'),
      'min_word_count' => 0,
      'locked'         => TRUE
    )
    );
}

function ckan_load($node){
    $ckan = ckan_ckan();
    try {
        $node->ckan = $ckan->getPackage($node->body);
    } catch(Exception $e){
        drupal_set_message($e->getMessage(), 'error');
    }
//    print_r($node->ckan);
    return $node;
}

function ckan_view($node, $teaser = FALSE, $page = FALSE) {
  

  $node = node_prepare($node, $teaser);
  $node->content['title']['#value']  = check_plain($node->ckan->title);
  $node->content['body']['#value'] = nl2br(check_plain($node->ckan->notes));

  return $node;
}






function ckan_theme() {
    return array(
    'ckan_search_results' => array('arguments' => array('results')),
    );
}




function theme_ckan_search_results($results) {
    $content = '';
    $content .= "<p>" . $results->count . " Results found</p>";
    foreach ($results->results as $package){
        $content .= '<h2>' . l($package->title, 'ckan/data/' . urlencode(check_plain($package->name))) . '</h2>';
        $content .= '<p>' . nl2br(check_plain($package->notes)) . '</p>';
        $content .= '<p>';
        foreach ($package->groups as $i => $group){
            $content .= '<b>' . check_plain($group) . '</b> ';
        }
        $content .= '</p>';
    }
    return $content;
    
}
?>

Accessing CKAN data from PHP

sean — Tue, 23 Feb 2010 21:22:46 +0000

I've been working on a couple of open data projects recently - which has been very rewarding.

A couple of people have asked for some code so I've knocked up a quick version of the sort of thing I've been working on.

This code requires PHP 5.1 for the JSON functions but doesn't need any extra libraries (pecl_http can give better error messages but is a bit of a pain to install).

The Notes field in packages returns output in Markdown format - you can convert this to HTMl with a parser available at http://michelf.com/projects/php-markdown

This isn't production code so please test before using.

<?php
class Ckan {
    private  $url = 'http://www.ckan.net/';
    private $errors = array(
                              '400'  =>   'Bad Request',
                              '403'  =>   'Not Authorized',
                              '404'  =>   'Not Found',
                              '500'  =>   'Internal Server Error', 
    );
    
    public function __construct($url=null){
        if ($url){
            $this->url=$url;
        }
    }
    
    private function transfer($url){

        $ch = curl_init($this->url . $url);


        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_HEADER, 0);
        curl_setopt($ch, CURLOPT_TIMEOUT, 30);
        $result = curl_exec($ch);
        $info = curl_getinfo($ch);
        curl_close($ch);
        if ($info['http_code'] != 200){
            throw new CkanException($this->error_codes["$info[http_code]"]);
        }
        if (!$result){
            throw new CkanException("No Result");
        }
        return json_decode($result);
    }

    public function search($keyword){
        $results = $this->transfer('api/search/package/?all_fields=1&q=' . urlencode($keyword));
        if (!$results->count){
            throw new CkanException("Search Error");
        }
        return $results;
    }

    public function getPackage($package){
        $package = $this->transfer('api/rest/package/' . urlencode($package));
        if (!$package->name){
            throw new CkanException("Package Load Error");
        }
        return $package;
    }


    public function getPackageList(){
        $list =  $this->transfer('api/rest/package/');
        if (!is_array($list)){
            throw new CkanException("Package List Error");
        }
        return $list;
    }

    public function getGroup($group){
        $group = $this->transfer('api/rest/group/' . urlencode($group) );
        if (!$group->name){
            throw new CkanException("Group Error");
        }
        return $group;
    }

    public function getGroupList(){
        $groupList = $this->transfer('api/rest/group/');
        if (!is_array($groupList)){
            throw new CkanException("Group List Error");
        }
        return $groupList;
    }
}

class CkanException extends Exception{}
?>

Test with

<?php
require_once("ckan.php");

$ckan = new Ckan();


$data = $ckan->getPackage('edmonton-data');

print_r($data);
?>

I'm keeping this file and a Drupal module that uses it on Github at:

http://github.com/practicalweb/drupal-ckan

Drupal - the missing API

sean — Tue, 23 Feb 2010 09:28:08 +0000

I was just reading this presentation on deploying Drupal

http://www.slideshare.net/eaton/drupal-deployment-presentation

and noticed

http://drupal.org/project/install_profile_api

Which seems to have a collection of API functions needed at install/upgrade time that are missing from Drupal (and contrib modules)

And even better this is great motivation to write more such functions myself instead of just using SQL hacks.

Drupal as URL shortener

sean — Mon, 22 Feb 2010 12:43:42 +0000

Drupal has a couple of modules which allow you to produce your own tinyurl.com type website - complete with your own tracking etc.

This has been extended for use by the US government at http://go.usa.com/ and the code re-released to the community.

This mainly adds statistics gathering and reporting

Unfortunately the module maintainer has chosen not to integrate these changes back into the module at drupal.org - issue here : http://drupal.org/node/520588

So now the patch doesn't apply cleanly unless you checkout an old version of the module from CVS like this

NB the password for anonymous access to CVS is "anonymous"

cvs -d :pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib login
cvs -d:pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib co contributions/modules/shorturl
cd contributions/modules/shorturl/
cvs update -r DRUPAL-6--1  -D 2009-12-01
wget http://drupal.org/files/issues/shorturl_1.patch
patch -p0 < shorturl_1.patch

Update:

I've managed to merge both branches of development and the bugfix at http://drupal.org/node/627668

Download an updated version of the module : shorturl_with_statistics.tar.gz

This module is very much a development version - I've incorporated several people work and the result seems to work OK - but test before using.

ssh port forwarding

sean — Fri, 19 Feb 2010 11:54:02 +0000

Every now and again I find I have ssh access to one server - which has access to somewhere I need to get to - but I want direct access (eg to forward an X session)

ssh -L 2222:otherserver:22 server

ssh -X -p 2222 localhost

an now I'm logged into othersever with forwarded X

Drush "tput: No value for $TERM and no -T specified"

sean — Thu, 18 Feb 2010 11:23:34 +0000

The drush script which provides comman line acceess to drupal functionality emits an error message when run as a cron job

tput: No value for $TERM and no -T specified

# If it is not exported determine and export the number of columns.
if [ -z $COLUMNS ]; then
  export COLUMNS=$(tput cols)
fi

I presume drush uses this information to calculate layout of output.

However when running as a cron job COLUMNS is not set and tput gives the above error.

To resolve this specify a COLUMNS value in your cron job, setting an appropriate size for viewing in email

cron now runs the following for me.

#!/bin/bash

cd /var/www/mysite/public_html

COLUMNS=80 /home/me/bin/drush cron

Drupal Cron getting stuck

sean — Fri, 05 Feb 2010 11:02:06 +0000

I've just run into the drupal cron problem again

Cron fails and the error log says

"Attempting
to re-run cron while it is already running."

delete from variable where name='cron_semaphore';
clear caches
run cron from the web page

That seems to do the trick.

Now cron runs OK fom drush or web.

Programatic Block Creation in Drupal

sean — Wed, 03 Feb 2010 17:57:34 +0000

To remove static blocks and replace with as custom block.

Implement hook_block specifying the default visibility and so on

<?php
$blocks['random'] = array(
      'info' => t('Random'),
      'weight' => 0,
      'status' => 1,
      'region' => 'sidebar',
      'pages'  => 'offers',
      'visibility' => 0, // all pages except
      'cache' => BLOCK_CACHE_PER_PAGE,
    );
?>

Create an update hook that deletes the old blocks and boxes (I've specified the ID as I know they won't change on my site).

<?php
function mymodule_update_6101(){
    
    $items = array();
    $items[] = update_sql('DELETE FROM {boxes} WHERE bid in (2, 12,13)');
    $items[] = update_sql("DELETE FROM {blocks} WHERE module = 'block' AND delta in (2, 12, 13)");

// Update the 'blocks' DB table with the blocks currently exported by modules.
// It's a "private" function called when you visit 
// the admin/build/block/list/ page
// calling it here causes the rehash which you otherwise have to visit the page to get
    _block_rehash();

// clear the cache
    cache_clear_all();
    return $items;
}
?>

When you run update.php your shiny new blocks should then be immediately available with no manual steps required.

This makes it so much easier to test your upgrades.

Bug in Trac on Ubuntu Jaunty

sean — Tue, 02 Feb 2010 09:49:21 +0000

There is a bug in Ubunty Jaunty which cause Trac to have errors.

Attachments do not work, the admin interface for milestones doesn't work properly - and there may be other issues.

The bug report is at https://bugs.launchpad.net/ubuntu/+source/trac/+bug/369792

And the solution is

"go to http://packages.ubuntu.com/karmic/trac
and scroll all the way down. There you'll find a download link for the
karmic package which you can download and install via

sudo dpkg -i trac_0.11.5-2ubuntu1_all.deb"