PracticalWeb Ltd

Websites that work for you.

Non-root Packet Capture in Wireshark

There are lots of blogs about this already - but I didn’t quite understand at first so here’s my take …

Wireshark is a large and somewhat vulnerable program - best not to run it as root (especially as you may be looking a suspicious traffic)

It uses dumpcap for the actual packet capture which is the only bit that needs elevated privileges.

Since linux kernel 2.2 we don’t even need root for this bit - we can set file permissions to allow non root user to capture packets.

If you don’t want to allow just any user on your system to do this first make sure only a chosen group can even run dumpcap

I decided to use the sudo group - but you could make a new one.

1
2
sudo chgrp sudo /usr/bin/dumpcap
sudo chmod 750 /usr/bin/dumpcap

Now only members of the sudo group (who are already the users I trust) can run this program.

1
sudo setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap

This allows the program to gain elevated network rights without being run by root.

Note that any chmod action seems to undo the setcap so perform the changes in this order.

Now I can run wireshark as my regular user and have full functionality.

Specifying MTU for a VPN Connection in NetworkManager

I need to use a VPN which was suffering packet fragmentation due to the overhead involved (actually I use a VPN to connect to a network which then has another VPN connection to a third network).

To determine the largest MTU size which didn’t lead to fragmentation I just ran variations on

1
ping -M do -s 1327 server.example.com

-M do prohibits fragmentation so the ping will fail if the packet size is too large

I tried larger/ smaller values till I found the largest number that worked.

Network Manager didn’t give me a GUI to set MTU on the VPN connection (it does for ethernet)

So I needed a script in /etc/network/if-up.d/

I called mine like myvpn-mtu (owned by root and executable)

1
2
3
4
5
6
7
8
9
10
11
#! /bin/sh
# set low MTU for my VPN otherwise I get packet fragmentation
set -e


if [ "$CONNECTION_ID" = myvpn ] && [ "$MODE" = start ]; then
  /sbin/ifconfig $IFACE mtu 1327
fi


exit 0

Substitute “myvpn” here for your connection name

The script env has the following variables (I’ve anonymised some variables)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
ADDRFAM=NetworkManager
CONNECTION_ID=xxxxxxxxxxxxxx
CONNECTION_UUID=xxxxxxx-xxxxx-xxxxx-xxxx-xxxxx
DEVICE_IFACE=wlan0
DEVICE_IP_IFACE=wlan0
DHCP4_BROADCAST_ADDRESS=192.168.0.255
DHCP4_DHCP_LEASE_TIME=86400
DHCP4_DHCP_MESSAGE_TYPE=5
DHCP4_DHCP_REBINDING_TIME=75600
DHCP4_DHCP_RENEWAL_TIME=43200
DHCP4_DHCP_SERVER_IDENTIFIER=192.168.0.1
DHCP4_DOMAIN_NAME=xxx
DHCP4_DOMAIN_NAME_SERVERS=192.168.0.1
DHCP4_EXPIRY=1417685250
DHCP4_HOST_NAME=xxx
DHCP4_IP_ADDRESS=192.168.0.209
DHCP4_NETWORK_NUMBER=192.168.0.0
DHCP4_ROUTERS=192.168.0.1
DHCP4_SUBNET_MASK=255.255.255.0
IFACE=tun0
IP4_ADDRESS_0=192.168.0.209/24 192.168.0.1
IP4_DOMAINS=xxx
IP4_NAMESERVERS=192.168.0.1
IP4_NUM_ADDRESSES=1
IP4_NUM_ROUTES=0
LOGICAL=tun0
METHOD=NetworkManager
MODE=start
PHASE=post-up
PWD=/
VERBOSITY=0
VPN_IP4_ADDRESS_0=192.168.1.48/24 x.x.x.x
VPN_IP4_NAMESERVERS=x.x.x.x
VPN_IP4_NUM_ADDRESSES=1
VPN_IP4_NUM_ROUTES=28
VPN_IP4_ROUTE_0=192.168.1.0/24 0.0.0.0 0
VPN_IP4_ROUTE_1=192.168.2.0/24 0.0.0.0 0
VPN_IP4_ROUTE_2=192.168.3.0/24 0.0.0.0 0
VPN_IP4_ROUTE_3=192.168.4.0/24 0.0.0.0 0
VPN_IP_IFACE=tun0

Import Data to Graphite From Munin

To convert data from munin/rrd to graphite/whisper you need to use the rrdwhisper tool which in centos is part of the python-whisper package.

If you want to change the data retention/resolution properties in graphite you will also need to resize the files as well.

Note that Graphites reads from /etc/carbon/storage-aggregation.conf and /etc/carbon/storage-schemas.conf to select the retention and aggregation policies for new data, but since these are set in the file at creation time editing the config file has no effect on existing files (or those created with whisper2rrd).

Recent data is stored at 10 second intervals, when it becomes older 6 lots of 10 second data get aggregated to make a one minute interval. I think this aggregation also happens on newer data if I select say data for the last 2 days since this time period include data at the lower resolution - all the data I get is at this lower resolution.

When this data is aggregated it can be averaged; in my case I’m measuring how long something takes - and average is right. But say I was counting something then I’d want to know the total over a minute - sum - not average, and if I was recording peaks then I’d want the max.

The xFilesFactor as I understand it tells graphite to ignore all the data for an interval if there are too many nulls, but since my original data is at a lower resolution I have loads of nulls and don’t want to ignore anything - so I’m setting this to zero and my import seems to work well at different scales and on both old and new data.

1
2
3
4
5
6
7
8
9
for rrd in *rrd ; do
    rrd2whisper --xFilesFactor=0.0 --aggregationMethod=average $rrd
    whisper-resize  --xFilesFactor=0.0 --aggregationMethod=average --nobackup ${rrd%.rrd}_42.wsp   10s:6h 1min:6d 10min:1800d
done


[ -d /var/lib/carbon/whisper/munin ] || sudo -u carbon mkdir -p /var/lib/carbon/whisper/munin
sudo mv *wsp /var/lib/carbon/whisper/munin
sudo chown -R carbon:carbon  /var/lib/carbon/whisper/munin

This script excerpt deals with the basic conversion - I also needed to convert the long filenames munin used to a nested directory structure used by graphite.

That conversion will depend on your use case - if you look at the files stored under /var/lib/carbon/whisper and /var/lib/munin you should be able to see the pattern.

How to Log From Jmeter to Statsd

In order to log from Jmeter to statsd you need to add a statsd library to the jmeter /lib path

Then have a beanshell on the thread group setup a statsd object for use by the threads

store this in the props variable - JMeterProperties (class java.util.Properties)

Note that the docs say

Because Properties inherits from Hashtable, the put and putAll methods can be applied to a Properties object. Their use is strongly discouraged as they allow the caller to insert entries whose keys or values are not Strings. The setProperty method should be used instead. If the store or save method is called on a “compromised” Properties object that contains a non-String key or value, the call will fail. Similarly, the call to the propertyNames or list method will fail if it is called on a “compromised” Properties object that contains a non-String key.

However I didn’t find a better place to put the statsd object yet since the ctx variable is thread specific

1
2
3
4
5
6
import com.timgroup.statsd.StatsDClient;
import com.timgroup.statsd.NonBlockingStatsDClient;

StatsDClient statsd = new NonBlockingStatsDClient("jmeter.test", "statsd.exapmle.com", 8125);

props.put("statsd", statsd);

Now on the Sampler you want to record data for add a beanshell post processor

You can retrieve the statsd connection from the props variable, and obtain the request time taken from the prev variable which is a org.apache.jmeter.samplers.SampleResult

Once we have the pieces together logging the request time to statsd is simple

1
2
3
4
5
import com.timgroup.statsd.StatsDClient;
import com.timgroup.statsd.NonBlockingStatsDClient;

StatsDClient statsd = (StatsDClient) props.get("statsd");
statsd.recordExecutionTime("loadtime", prev.getTime());

I’ve put an example test plan on github https://github.com/practicalweb/jmeter-statsd-demo

Debugging With Maven and Eclipse

To use eclipse with Maven you need to add the maven plugin to eclipse, it seems to be part of the Indigo update site so this should be straightforward.

It’s also at http://download.eclipse.org/technology/m2e/releases

More instructions at http://maven.apache.org/guides/mini/guide-ide-eclipse.html

To run the debugger you need to do two things: you need to tell maven to run surefire, and run a remote java debug listening to that surefire

Setup a new “Debug configuration” > “Remote Java Application”

It should look like

screenshot

Next you need to run your maven code with the -Dmaven.surefire.debug

You can do this by adding it to the debug config like this

screenshot

I’ve used the debug mode - but maybe a run configuration would work too

An alternative is to run the maven project on command line

1
mvn test -Dmaven.surefire.debug

Generally if I’m using eclipse I prefer to do everything in the IDE (I stands for integrated after all) and it’s useful to have the console output in the same place

Either way run the project to debug and maven will stop with output

1
Listening for transport dt_socket at address: 5005

and wait till your debugger is attached

Now you have to start the remote application debug in eclipse

Execution will now stop at breakpoints and you will be able top explore the stack and all variables set

screenshot

This is an incredibly powerful way to explore running code, especially if you are unfamiliar with it. Even if you are familiar a debugger session can often be very enlightening.

Graphite Statsd/bucky and Collectd

Graphite is a highly scalable real-time graphing system.

It took me a while to realise but the data input to graphite is incredibly simple

The message input format is just

1
2
3
metric_path value timestamp\n

So for example, "foo.bar.baz 42 74857843"

Wow: just a label the number you want to store and the timestamp it relates to.

That’s really basic, there is nothing there about what kind of data it is, no difference between something you want to count (say page hits) and a number you want to average (like load)

Configuring Graphite for StatsD is a very important doc to read if you want to use these tools together.

The default config in graphite keeps data at 1 minute resolution for 1 day and then discards it and if you are sending data to graphite every 10 seconds graphite has to figure out how to convert 6 data points (or some nulls and some data) to 1 data point for the minute.

You really need the statsd specific config to ensure counts are not averaged and infrequent data is not dropped.

I needed to install on Centos using just official and EPEL packages, I wanted to use collected to gather system metrics and while version 5 can log direct to graphite version 4 ships with Centos 6.5 and that can’t log direct.

In any case I wanted to use statsd to allow easy logging of custom data from production code, the original written by etsy is a node service and probably doesn’t run with stock packages on Centos,

Centos does provide a python-bucky package which acts both as a statsd server and as a bridge to collectd.

The full list of packages I needed is

  • graphite-web
  • graphite-web-selinux
  • python-carbon
  • python-whisper
  • collectd
  • python-bucky

Apache config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
<VirtualHost *:80>
  ServerName graphite.example.com

  ## Vhost docroot
  DocumentRoot "/var/www/html"
  ## Alias declarations for resources outside the DocumentRoot
  Alias /grafana/ "/var/www/grafana/"
  Alias /media/ "/usr/lib/python2.6/site-packages/django/contrib/admin/media/"



  ## Directories, there should at least be a declaration for /var/www/html


  <Directory "/var/www/html">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>

  ## Load additional static includes


  ## Logging
  ErrorLog "/var/log/httpd/graphite.example.com_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/graphite.example.com_access.log" combined



  WSGIImportScript /usr/share/graphite/graphite-web.wsgi application-group=%{GLOBAL} process-group=%{GLOBAL}
  WSGIScriptAlias / "/usr/share/graphite/graphite-web.wsgi"

  ## Custom fragment

    <Location "/content/">
        SetHandler None
    </Location>

    <Location "/media/">
        SetHandler None
    </Location>

</VirtualHost>

Graphite Config

Storage-schames.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[carbon]
pattern = ^carbon\.
retentions = 60:90d

[stats]
pattern = ^stats.*
retentions = 10s:6h,1min:6d,10min:1800d


# unsure if this is redundant - does .* match stats_counts ?
[stats_counts]
pattern = ^stats_counts.*
retentions = 10s:6h,1min:6d,10min:1800d


# collectd 
[com]
pattern = ^com.*
retentions = 10s:6h,1min:6d,10min:1800d

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

storage-aggregation.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[min]
pattern = \.lower$
xFilesFactor = 0.1
aggregationMethod = min

[max]
pattern = \.upper(_\d+)?$
xFilesFactor = 0.1
aggregationMethod = max

[sum]
pattern = \.sum$
xFilesFactor = 0
aggregationMethod = sum

[count]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum

[count_legacy]
pattern = ^stats_counts.*
xFilesFactor = 0
aggregationMethod = sum

[default_average]
pattern = .*
xFilesFactor = 0.3
aggregationMethod = average

Statsd caught me out with a debug mode that only prints to stdout and doesn’t actually log any data to graphite, I expected debug mode to print out data in addition to regular duties.

I did need to make statsd listen on all networks instead of just localhost, and still need to wrap it properly as a service.

Edit collectd.conf to enable the plugins you want, the only problem I had here was collectd silently dying if I had a config error.

You need to setup an initial graphite users database

More details at http://obfuscurity.com/2012/04/Unhelpful-Graphite-Tip-4

1
/usr/bin/python /usr/lib/python2.6/site-packages/graphite/manage.py syncdb

Writing this up now it all seems quite simple - but I haven’t really found it so, I think the main problem has been understanding what graphite is actually doing, and not taking the trouble to optimise that config as a first step.

Pages I found useful

Grafana

I’ve been playing with some new toys.

What motivated me was a desire to be better able to interpret load test data and performance metrics.

I love graphs, for me the visualisation they provide allows me to recognise patterns and figure out cause/effects massively faster than any other way.

In the past I’ve mainly relied on Munin which is very easy to setup, and I’ve even customised graphs. My most successful case was when two servers in a pool of four kept falling down, by graphing apache hits per second and severer load for all for servers on the same page - it provided convincing proof that the two servers which fell down did so under no extra load and at the exact same moment. It later turned out these two were VMs on the same host OS which was having previously unnoticed problems.

This visualisation transformed a conversation from one of what could or couldn’t be wrong to acceptance and how to fix it.

But with Munin I had to define the graph and then collect the data - I had to wait for another outage.

I wanted to be able to compose graphs on the fly to test hypotheses with existing data - grab the data now and define the graphs to visualise what I want to see after the fact.

Enter Grafana I just love this tool, it allows me to compose graphs entirely on the fly, mixing and matching any data I want on the same graph, or multiple graphs on the same page.

With the timeshift function I can even put say this weeks and last weeks data on the same graph for easy comparison.

I hope to work on some screenshots and maybe screencaps - the tool is so interactive it’s hard to explain - for now go an play with it.

Grafana works by calling the json API of graphite-web, one thing that threw me at first was the initial config example

1
2
3
4
5
6
datasources: {
  graphite: {
    type: 'graphite',
    url: "http://my.graphite.server.com:8080",
  },
},

I wasn’t sure which server component I was meant to be connecting to on 8080, this is just the regular graphite-web service from apache - in my case on port 80.

It’s also worth noting that https://github.com/graphite-project/graphite-web/issues/223 at least not for the version I am running.

Quick Steps to Secure Ubuntu Linux

Recently I’ve been reviewing security and realised I’ve been relying too much on my routers firewall - which isn’t even present if I work on an open wifi connection somewhere.

Steps so far

Reinstalled Ubuntu with full disk encryption, apart from the need to back and restore data this was a painless process. I don’t see a noticeable performance impact (though I have a fast system with SSD) the biggest drawback I can see is that if I mess around with a custom kernel and break the boot sequence - I don’t know if I can boot from a live CD to fix it.

Setup a restrictive local firewall

as root

1
2
3
4
5
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -I INPUT 1 -i lo -j ACCEPT
iptables -P INPUT DROP
iptables -S
apt-get install iptables-persistent

edit /etc/postfix/main.cf

set

inet_interfaces = 127.0.0.1

and /etc/init.d/postfix restart

I will review open ports each time I configure a new service and ensure I don’t have services I only need local accessible externally.

By both configuring firewall and limiting services I am applying the principle of defence in depth and even if there is a weakness (or I make a mistake) in one place I will still be protected.

Where I do need to share services between systems on my home/office network I have realised my old router is not sophisticated enough and am purchasing one that can separate secure and insecure networks.

All systems are now being configured for automatic updates, since I want patches as soon as possible, small frequent updates are easier to debug than problems with large updates, and generally I only delay updating out of inertia rather than any deliberate action. This way I don’t even have to think about it.

from http://www.richud.com/wiki/Ubuntu_Enable_Automatic_Updates_Unattended_Upgrades

my config files are now

1
2
3
4
5
6
7
8
9
10
cat /etc/apt/apt.conf.d/50unattended-upgrades  | grep -v ^//
Unattended-Upgrade::Allowed-Origins {
  "${distro_id}:${distro_codename}-security";
  "${distro_id}:${distro_codename}-updates";
};
Unattended-Upgrade::Package-Blacklist {
};
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::Remove-Unused-Dependencies "false";
1
2
3
4
5
6
7
cat /etc/apt/apt.conf.d/10periodic  | grep -v ^//
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::Verbose "2";
APT::Periodic::RandomSleep "1";

Also set /root/.forward to ensure I get root mail

I have also realised I rely too much on browser stored passwords, and while this is useful for low-security logins I will not be using it for any important site.

Funnily I’ve also found I needed to revisit my backup policy and actually delete more stuff not backing up code and documents which are business confidential and are already backed up centrally. In this case I realised my own backups were just causing a data management problem just increasing the risk that the data gets accidentally disclosed.

Drupageddon

Drupal has been in the new today for all the wrong reasons

Millions of websites hit by Drupal hack attack

Drupal sites had “hours” to patch before attacks started

Drupal users: Assume your site was hacked if you didn’t apply Oct. 15 patch immediately

Did Drupal Drop The Ball? Users Who Didn’t Update Within 7 Hours ‘Should Assume They’ve Been Hacked’

The biggest headache for me was the Public Service Announcement https://www.drupal.org/PSA-2014-003 which is clearly written to alarm users into updating.

We did update our clients site - within 24 hours which at the time seemed pretty good considering the timezones and chaneg control processes involved.

Fortunately we also reload our entire database and other content on each deploy - so even though we took longer than 7 hours to update we consider we are pretty safe (never say 100%) and beyond that we have a lot of other security in place.

However I find myself wondering what else we could have done.

I wish we had a member on the drupal security team - that seems to have helped some people.

I see now there were a couple of (low key) advance notices https://groups.drupal.org/node/445893 https://twitter.com/drupalsecurity/status/522128826101669888

I will try to pay better attention to those in the future, but we may also have to enhance our out of hours work, the security annoucment came in at the end of the day - perhaps we will ahve to work overnight another time to review, test, and help the client upgrade.

Hopefully good will come from the thread at Follow up Drupageddon responsibly

Drupal will need to learn and grow from this.

Now Hosted on Github Pages

Since I’ve moved to a static site I’m saving myself some hosting fees and moved to Github Pages

It’s free and fast, saves me from managing a server.