Interactive dashboards: using in-panel links to control Grafana variables

One of my colleagues recently had a question from a customer: is it possible to have one panel in Grafana show a set of all metrics, but the other panels on a dashboard filter to a single metric value from a click in the first panel?

It's an interesting way of working with data, and something that many people find intuitive.  If I have a menu of options displayed, I probably want to see only things relevant to one of those options if I click it.  Unfortunately, doing that's not something that is immediately obvious in Grafana.  For better or worse (and there are good reasons you might not want that behavior!) there's no automatic filtering inside of panels on a dashboard.

Usually people will use dashboard variables for this.  Variables provide powerful functionality for filtering queries in Grafana.  But they have a drawback: they aren't integrated directly into a dashboard, always appearing at the very top of the screen.

After playing around for a bit, it turns out that there is a way to use variables inside of a panel and have them act the same when you click a value as if you had selected that value from a dropdown at the top of the dashboard.  To do this, we can slightly abuse the data links functionality provided by panels.

I'll provide an example of this using Prometheus as a data source, but it should work with any data source you like.  We're going off the beaten path in Grafana here.  Buckle up, because this one gets a little bumpy!

Text panel tricks: drawing dividers in Grafana dashboards

Grafana has a lot of fantastic visualizations built in.  You can draw graphs of nearly every imaginable type, create beautiful maps, even build interactive flowcharts with multiple data elements that update in real time.

But my favorite (and definitely the most overlooked) panel is the simple text panel.

The text panel contains far more power than most people realize.  The fact that it can act as a container for almost any HTML means that you can do a lot of really cool things with it.

Today let's look at how to use the text panel to draw horizontal and vertical separators on a dashboard.

I've started by designing a simple dashboard showing some status information for three different manufacturing lines.  In a plant like this, the plant manager wants to see the status of all the lines at a glance but still easily differentiate them for troubleshooting.  With a small group of items, like the three here, it's useful to use a columnar layout.  But Grafana uses a row orientation for dashboards, meaning it is easy to accidentally move a panel around slightly and have it be unclear which column it belonged to.  And even if alignment isn't an issue, how can you let viewers know that you're lining up your data in columns rather than the default rows?

In the screenshot below, you'll see a possible solution: dividers between columns that make it perfectly clear that each graph "belongs" to a vertical group.  (In this case I've used a very bright violet to make the dividers stand out, but in a real dashboard you'd probably want something a bit more subtle!)

While there's no "line" option in the text panel, you can use some simple HTML and CSS tricks to create one.

Vertical dividers

To make a vertical line, start by setting a few options on the text panel:

• No panel title (just delete the text that's there)
• Transparent background turned on
  
• Panel mode set to "HTML"

This will make your text panel as unobtrusive as possible.  Getting rid of the title means that the full area of the panel will be available for your line with no space reserved for title text.  Enabling a transparent background means that you won't have a bright border around the panel itself, and we'll need to write HTML for the trick to work.

Once your panel is configured, add the following text:

<div

    style="margin: auto; border-left: 3px solid #f0f; width: 3px; height: 100%">

</div>

Once you're done, your panel configuration should look something like this:

Let's break this down a bit.

The <div> HTML tag lets us create an empty HTML container.  Usually this is used to hold text, images, or other content, but for our purposes we really just want a "thing" that sits on the page to draw a border around.

The style atrribute lets us attach CSS to this empty container.  We set the margin to auto which means that our div will be centered within its parent container, in this case the Grafana panel.

We want either the left or right border on this empty container to be turned on; the div will only be as wide as the full border so it doesn't really matter which one, and "left" is just slightly shorter to type than "right" so being lazy, I used that.  The properties of the border-left attribute control how the border will look.  Changing "3px" to "5px" will make the border two pixels wider, and changing "#f0f" to "black" will change the color from violet to black.  There are a number of ways to change this up, so check out the CSS border properties for more information.

Finally, we set the height of this container to 100%.  This makes it adapt as the panel is resized.  You can then easily make your divider longer or shorter just by resizing the Grafana panel.

Once this is done, you can shrink the panel down to the thinnest width and drag it out to be as tall as you need it to be.

Horizontal dividers

If you want a horizontal divider in Grafana, there are a couple of options available.  You could use a similar technique as above but with a border-top instead of border-left.  But HTML gives us a slightly easier way to make horizontal borders: the <hr> tag.

The <hr> tag creates a horizontal rule (i.e. line) in a page.  And like most other HTML elements, it can be styled with CSS.

To create a styled horizontal divider in Grafana, start by setting up a transparent HTML text panel as described above.  This time, add the following HTML:

<hr style="height: 3px; background-color: #f0f">

This will create a horizontal line on the panel with a height of 3 pixels and the same violet color we used before.  Feel free to change these values to match your dashboard style.

Note that this won't center the line vertically in your panel, but that's usually okay -- you'll be resizing the panel to be as short as possible anyway.  (If you really need some blank space above your line, remember that you can create a completely empty transparent text panel and resize it to whatever spacing you want.  That's often easier than trying to center things vertically in CSS!)

I hope this helps you make beautiful and logical divisions in your dashboards.  Go line everything up!

Working with noisy data in Prometheus

 I haven't written much here lately as I've been busy elsewhere.  But some of that includes working extensively with Prometheus for monitoring various services.  Along the way, I've written up a blog post on the Grafana Labs blog about working with noisy data.  The beginning of it is here but the full post is up on the Grafana blog.

Most of us have learned the hard way that it’s usually cheaper to fix something before it breaks and needs an expensive emergency repair. Because of that, I like to keep track of what’s happening in my house so I know as early as possible if something is wrong.

As part of that effort, I have a temperature sensor in my attic attached to a Raspberry Pi, which Prometheus scrapes every 15 seconds so I can view the data in Grafana. This way, I know how things look over time, and I can get alerts if my house is getting too hot or too cold.

Unfortunately, my temperature sensor is a bit flaky. It works most of the time, but occasionally it gives me wildly inaccurate readings. Here’s an example that looks at a few hours of data:

Even though the weather can be unpredictable here in New England, it’s pretty unlikely that the room temperature dropped from 20°C to -10° for 15 seconds before returning to normal!

Still, other than these occasional single readings that are obviously wrong, most of the data looks good. So my first thought when I saw this glitch was to replace the sensor with one that works more consistently. After all, having good data helps with everything.

But I started to think about the problem more . . .

What would I do if I had a sensor like this that couldn’t easily be replaced? If my sensor were on top of a mountain or deep under the ocean, it would be difficult and expensive to fix. And if it were on a satellite or in a rover on Mars, it would be impossible.

I couldn’t help but wonder: If most of the data is good, is there a way to keep the good bits and throw out the bad?

After speaking with some data scientist friends, it turns out that the answer is yes. And even better, Prometheus has a function to do exactly that!

Read the rest here!

Catching up on past projects: Magic Hate Ball

I've been doing things lately, but not actually blogging about them.  I should fix that, if for no other reason than to keep track for myself.  So I'll start with the most recent.

I have always enjoyed Magic 8 Balls.  They're so incredibly bizarre; I mean, a random fortune-teller, sure, but why as a floating blob in the bottom of an oversized billiard ball?  Nothing about it makes any sense, which is why it's so perfect!

Or almost perfect... if only it were rude to you when you picked it up.

So I ripped the guts out of a Magic 8 Ball, replaced them with an accelerometer (to detect when it turned over), a screen (to show the messages), and a microcontroller (to hate you).  I threw in a battery just for fire risk.

More details (including oh-so-hilarious project logs) at Hackaday.

Improving the state of security in the Internet of Things

I've spent the past few months as the Solutions Architect at resin.io, working to improve the security and functionality update process for Internet of Things (IoT) devices. As I've been doing this, I've been learning an enormous amount about the current state of the industry.

It's pretty clear that right now managing and updating IoT devices sucks. There are countless examples of devices being left vulnerable, exposed to the world, abandoned or even intentionally destroyed by their producers because they were too hard to keep updated.

There are a few factors that I see as contributing to this problem:

• a need for Free/Open tools for securing devices
• a need for Free/Open tools for updating and managing devices
• a need for education within the industry

Some more cynical people might replace "education" with "an attitude adjustment" in this list, but I really do think that most of the blame can be placed on ignorance rather than malice here. The tools are important, even crucial, so I'll expand on those points in the future. But even the best tools are worthless without the understanding of why they're so necessary.

A hardware mindset

Most people and companies involved with hardware manufacturing have what I call a "hardware mindset": the idea that a product is designed and manufactured exactly once and then identical units are distributed to the marketplace. This is understandable and a completely legitimate way of thinking when you're creating self-contained devices. If you've been building widgets for decades (or even longer!) then you've gotten very good at this sort of process. You spend a lot of time up front on design, getting it as perfect as you can, but then once you're done with that you start up the factory and move on to the next thing. As long as your widgets aren't catching on fire or falling to pieces under normal use, you don't really have a problem.

The problems do come in when you start talking about connected devices. When you were making widgets that came out of the factory perfectly safe, the only way they became dangerous is if someone intentionally opened them and tinkered with their parts.  And if someone gets hurt when they modify your product, that's really on them.  But once something is connected to nearby devices or the Internet, it's able to be opened up and modified by someone who might be thousands of miles away, and there's usually no external sign of tampering. Even without an actual attacker, it's simply impossible to test every permutation of interactions between your networked widget and all the other devices in the world. Your customers can be hurt by your product without any fault of their own! Suddenly you need a way to fix these issues as they arrive.  You must update your devices in the field to protect your customers.

Simply put, once the device comes out of the factory, you have to stop thinking like a hardware company and start thinking like a software company.

Moving to a software mindset

Software companies work very differently than hardware companies. There was a brief period where the idea was the same -- write some bytes to a disk or CD and ship them out and you're done -- but those days have been gone for decades. Now the goal isn't to produce a single piece of static software but to have a system that allows constant updating and refinement of the product. There can be no assumption that any program is ever perfect or complete or secure. The only way to protect yourself and your customer is to make the process of updating and fixing the software so fast and foolproof that keeping it updated becomes a matter of course.

IoT and hardware companies must begin thinking about their products primarily as software that happens to have a hardware component. This is the only way that the state of security in the Internet of Things will improve.

How to fix a USB audio device that interferes with mouse clicks in Linux

(This post includes an overview of how to solve similar problems to the one I encountered.  If you're looking for just the solution?  Skip to "The solution" below.)

I recently got a Jabra USB headset.  It's a very nice headset with active noise cancelling, a decent microphone, and a USB audio controller that includes a few buttons to control the audio volume.  These buttons work directly with the OS by presenting as a USB keyboard and sending the same keystrokes that the volume up/down buttons on a real keyboard would.

Unfortunately, it also presents itself as a USB mouse with 12 buttons and takes precedence over any actual pointing devices attached to the system.  Why a USB audio device needs to present itself to the system as a mouse is beyond me.  (The only theory I've heard that makes any sense at all is that there's some Windows-specific driver that would intercept mouse events and do some sort of deeper OS integration.)  Whatever the reason, it's a terrible "feature" and an abuse of the USB HID specification.

But rant over. Fortunately because X (the graphics engine of desktop Linux systems) allows the user fine control over the system, this is pretty easy to fix.  I'm including the steps I used to troubleshoot and fix this so that if anyone runs into similar issues in the future they know what to do.

Narrowing the issue down to X

As soon as I started using this headset I noticed something strange: I couldn't click the mouse anywhere outside of the window that I had on top when I plugged it in.  This persisted until I unplugged the USB cable, when suddenly mouse clicks worked again.  When I plugged the USB cable back in, mouse clicks stopped working again.  Weird, but at least consistent!

Knowing that it had something to do with the USB device I started looking at a few system logs.  I ran 'dmesg', which shows the device driver logs for the system, but didn't seem anything unusual.  So it wasn't a Linux issue, which meant it had to be in X.

Diagnosing in X

The first thing I did was to see what X saw when I had the device plugged in.  I ran 'xinput', which shows a list of all of the input devices attached to the system.  Before plugging in the device I saw:

⎡ Virtual core pointer                     id=2 [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer               id=4 [slave  pointer  (2)]
⎜   ↳ ELAN Touchscreen                         id=10 [slave  pointer  (2)]
⎜   ↳ DLL0665:01 06CB:76AD Touchpad           id=12 [slave  pointer  (2)]
⎣ Virtual core keyboard                   id=3 [master keyboard (2)]
    ↳ Virtual core XTEST keyboard             id=5 [slave  keyboard (3)]
    ↳ Power Button                             id=6 [slave  keyboard (3)]
    ↳ Video Bus                               id=7 [slave  keyboard (3)]
    ↳ Power Button                             id=8 [slave  keyboard (3)]
    ↳ Sleep Button                             id=9 [slave  keyboard (3)]
    ↳ Integrated_Webcam_HD                     id=11 [slave  keyboard (3)]
    ↳ AT Translated Set 2 keyboard             id=13 [slave  keyboard (3)]
    ↳ Dell WMI hotkeys                         id=14 [slave  keyboard (3)]

and afterwards I saw:

⎡ Virtual core pointer                     id=2 [master pointer  (3)]

⎜   ↳ Virtual core XTEST pointer               id=4 [slave  pointer  (2)]

⎜   ↳ ELAN Touchscreen                         id=10 [slave  pointer  (2)]

⎜   ↳ DLL0665:01 06CB:76AD Touchpad           id=12 [slave  pointer  (2)]

⎜   ↳ GN Netcom A/S Jabra EVOLVE LINK MS       id=15 [slave  pointer  (2)]

⎣ Virtual core keyboard                   id=3 [master keyboard (2)]

    ↳ Virtual core XTEST keyboard             id=5 [slave  keyboard (3)]

    ↳ Power Button                             id=6 [slave  keyboard (3)]

    ↳ Video Bus                               id=7 [slave  keyboard (3)]

    ↳ Power Button                             id=8 [slave  keyboard (3)]

    ↳ Sleep Button                             id=9 [slave  keyboard (3)]

    ↳ Integrated_Webcam_HD                     id=11 [slave  keyboard (3)]

    ↳ AT Translated Set 2 keyboard             id=13 [slave  keyboard (3)]

    ↳ Dell WMI hotkeys                         id=14 [slave  keyboard (3)]

The highlighted line shows the change: this was the device I had plugged in.  (It appears that GN Netcom A/S is the actual manufacturer and Jabra the brand.)

This told me I was on the right track: the USB headset was showing up as an input device.  This could be why it was interfering with the mouse!  But I needed more information.  Fortunately, xinput has a more verbose mode, 'xinput list --long':

⎜   ↳ GN Netcom A/S Jabra EVOLVE LINK MS       id=15 [slave  pointer  (2)]

Reporting 16 classes:

Class originated from: 15. Type: XIButtonClass

Buttons supported: 12

Button labels: "Button 0" "Button 1" "Button 2" "Button Wheel Up" "Button Wheel Down" "Button Horiz Wheel Left" "Button Horiz Wheel Right" "Button 3" "Button 4" "Button 5" "Button 6" "Button 7"

Button state:

Class originated from: 15. Type: XIKeyClass

Keycodes supported: 248

Class originated from: 15. Type: XIValuatorClass

Detail for Valuator 0:

 Label: Abs X

 Range: 0.000000 - 1000.000000

 Resolution: 0 units/m

 Mode: absolute

 Current value: 1600.000000

...

(I've trimmed this down quite a bit; 'xinput list --long' really is long.)

From this I saw that it was reporting itself as a mouse device.  (Why does a USB headset need to pretend to the OS to be a mouse?  I still don't know.)  Something in that list of mouse buttons was messing with my actual mouse.  So I needed to disable that functionality while leaving the rest unchanged.

Fine-grained device control in X

The X device control system allows you to make a lot of tweaks to how devices work.  You can change how a keyboard or mouse work, how quickly the pointer moves when you move the mouse, how it accelerates based on the direction you are moving... all sorts of things.  But for my purposes I needed to use the functionality to change the order of mouse buttons.  This is normally for doing things like making a mouse more comfortable for left-handed users (e.g. changing the button order on the mouse from "1 2 3" to "3 2 1" so the left-hand index finger is the primary button).  But you can also map physical buttons to "button 0", which means disabled.  So I just disabled all 12 (why 12‽) mouse buttons.

I did this with the help of the X device control manual here: www.x.org/archive/X11R7.5/doc/man/man4/evdev.4.html

The device control system looks in a special location for files that contain instructions  to change how devices work.  On Ubuntu this location is /usr/share/X11/xorg.conf.d/ but may be different on other Linux distributions.

The solultion

I created /usr/share/X11/xorg.conf.d/50-jabra.conf with the following contents:

Section "InputClass"

Identifier "Jabra"

        MatchProduct "GN Netcom A/S Jabra EVOLVE LINK MS"

Option "ButtonMapping" "0 0 0 0 0 0 0 0 0 0 0 0"

EndSection

I then restarted the X system to make sure the changes would take effect.  (A reboot is the simplest way to do this if you're unsure how.)

Once that was finished I could plug and unplug my USB headset with no issues!

Followup

I submitted a bug report to the Ubuntu bug tracking system with my fix attached.  Hopefully it will be picked up and added to the distribution.  In the meantime I'll leave this post up in case anyone else has similar issues.

Communicating internationally and being understood

English is the language of international business.  It's used as the default language of international conferences, technical papers, and contracts.  I've lost count of the number of conversations I've overheard where, say, a German and French speaker are communicating with each other in English rather than trying to use either of their native languages.

But unfortunately there's a group of people that often makes communication more difficult than it needs to be: native English speakers!  Too often a native speaker will forget that other people are working much harder than they are in a conversation and will slip into bad habits that make it even harder.

I've noticed some simple things that native speakers can do to fix this.  In my experience, these have a dramatic impact on how easily others are able to understand spoken English.  Think of these as a way to help your listener.  You're not meeting them halfway (as they are speaking your language, after all!) but at least you're removing some roadblocks in their path.

Keys to being more easily understood

• Speak slowly.
• Face your listener.
• Keep your mouth clear.
• Don't laugh while speaking.
• Keep your sentences clear and direct.

Speak slowly

This should almost go without saying, but is easily forgotten, especially if you are speaking with someone who speaks English nearly fluently.

Consciously make an effort to slow your speech down a bit more than normal.  Not so slowly that others will think you're making fun of them, but enough to ensure that you are not running words together or moving so fast that your listener can't keep up.  I usually mentally aim to speak at about 85-90% of my normal pace -- enough to remind me to take my time and not rush.

Face your listener

This too is common sense.  Partially this is to help you be heard clearly.  But more importantly, keeping your focus on your listener will help you see if they are getting lost or confused.  It's better to slow down and get everyone on the same page as soon as there's an issue rather than realizing that you haven't been understood for several minutes.

If you your listener starts to look confused or anxious, it might be a good idea to check in with them to make sure there aren't any communications issues.  It can be embarrassing to stop someone who is obviously on a roll to say "Slow down!" or "I'm having trouble understanding you."  Take responsibility for being understood yourself -- it really goes a long way.

Keep your mouth clear

We all know that it's rude to talk with your mouth full.  It also makes it harder to make out what you are saying, so don't do it.

But there's more to this.  Try not to cover your mouth when speaking.  You might have a habit of hiding your smile behind your hand or stroking your chin thoughtfully when speaking.  Be aware of these and make sure you avoid them!  Not only do they muffle your voice, but they hide your lips.  Visual information (in the form of mouth movement and shape) is extremely important in understanding speech.

Don't laugh while speaking

This goes along with the previous point, but is so easy to forget that I wanted to call it out specifically.  If you are laughing, stop speaking while doing so.  It's fine to laugh (or cough, or clear your throat, etc.) but don't mix it with speech.  If you need to wait a few extra seconds to keep going, that's fine -- much better to take a breath than to make your listener try to decipher something like "and then ha-heh-the-heh bar-heh-ten-ha-ha-der says 'why-he-he the-ha lo-ha-ong-heh fay-hay-hay-ce?'".

Keep your sentences clear and direct

I put this one last because I think it's the hardest technique in the list, so starting with the others will get you more effect for less effort.  But that doesn't mean it's not important!

What I mean is to try to use a consistent structure in your sentences.  Aim for the typical subject-verb-object word order.  Don't use too many dependent clauses.  Avoid dangling modifiers.  Basically, all the stuff you learned in grade school English classes and then promptly forgot.

Avoid run-on sentences and "filler" words.  Listen to your own speech.  Do you find yourself saying "ummmmmm" when looking for the right word?  Or things like "and then I saw the man and then he walked over to me and then we shook hands and then we started to talk and then..."?  This nonstop wall of sound is really cognitively taxing -- you're loading up a lot of extra work on your listener, forcing them to determine which words they need to mentally filter out and which words they actively need to translate.

So take pauses!  Breathe between sentences and let them sink in a bit.  If you need to think of a word, just think rather than filling the space.  You'll be much easier and more pleasant to listen to.

Be a better speaker

This advice applies to when speaking to native listeners too, by the way.  Listen to some TED talks or other really good speakers.  You'll note that they take pauses, speak at a moderate pace with clear syntax and diction, and don't "ummmm" or "uhhhh".  Learning and internalizing these techniques will make you easier to understand and a more compelling speaker.

(Of course, you could always try the opposite but I wouldn't recommend it.)