11 Mar How do you measure the impossible?

Measuring the Impossible.

Despite having “mountains” of data, the truth remains that surprisingly often we can’t directly measure what we really want too.

In fact, smart people can often put several balanced and convincing arguments to us about why getting that measured data is impossible.

It can be a tempting to accept that (they’re smart after all.)

It is nagging though. Sometimes more so.

For example:

Where do you armour a fighter plane so they don’t get shot down?

So the story goes that this was a real question the Navy was considering during WWII.

The challenge was how to better protect the planes so they didn’t get shot down.

The issue was that they had no data to go on.

Why?

Well, because they couldn’t inspect the shot down planes so they had no idea what happened.

Even when they could get to the planes, assessing the damage was hard.

The planes that got shot down were so badly damaged that any analysis of the wreckage was futile.

So what to do?

One solution would be to armour the whole plane.

The issue with this is that not only is it expensive but likely to be very heavy.

Planes don’t fly so well when they’re heavy. In this case making them more of a sitting duck.

What they really wanted then, was to be able to place the armour just where it was required and no more.

But where?

For these guys the issue was more than nagging, it was literally a matter of life and death.

“But what if … “

What if you could get that critical data?

What if you could have the information you needed?

What if you did have the knowledge that would help you answer your questions – about armouring planes or whatever is relevant in your world?

How about some solid data that would really inform your critical decisions?

What kind of difference would that make?

Quite a difference I’d bet.

Certainly in the case of these guys in the Navy the potential to save lives and have many more planes come home was huge.

I’ve been there like you may have been. Not in the military but stuck like that.

Stuck trying to get my head round a seemingly impossible measurement challenge.

It is the situation where we’ve worked out what we’d really like to know but there seems no way of measuring it.

It can be immensely frustrating but next time you’re in that “impossible” situation, don’t give up.

What I’m going to put to you is not to argue with those smart people – they’re normally right – but instead consider the “inferred metric.”

The inferred metric

The “inferred metric” is an approach to get data on something you can’t measure directly (like where those planes were getting shot down) by taking what you can measure and inferring what you want to know from that.

There are many different types of inferred metrics.

Here are 7 (with examples) for you to consider:

1. Proportional change

This can be something moving up or down in proportion to what we really want to know.

In sport we might want to know our blood lactate levels. To measure lactate requires a blood sample. Sadly this can be impossible for many reasons (i.e. lack of equipment at venue, moving too much to take sample etc.)

What we can measure quite easily is heart rate.

Whilst not an ideal match there is a strong relationship between heart rate and lactate levels.

Therefore by measuring our heard rate we can infer what our lactate levels might be.

2. Experience of others

By seeing what others have done in a similar situation, we can sometimes infer what someone else would do too.

A good example would be online services like Spotify or Netflix or Amazon where they watching what other people do (who they think are similar to us) and then suggest something for us that we might also like to try/do/buy.

That’s at a mass scale. It can also be relevant at a smaller scale.

An example could be in motorsports. If you’re trying to setup the racing car to have balance (balanced cars tend to go faster) then you can send out two or three different drivers.

If they all come back with the same comments then you can infer what you need to do to make the car more balanced.

3. Controlled Experiment

Measuring what you want directly but not in a representative situation.

For example, going back to our lactate verse heart rate. If you run on a treadmill in a laboratory, we can control the exact speed you run and the duration. You are also right their (and not 3 miles from the testing equipment).

We can get you to run at different speeds, taking your lactate periodically, and measure your heart rate at the same time.

A treadmill is not representative of your real running environment but it is close and enables us to get direct measurements.

When you’re back out running on the road you can then infer your lactate from your heart rate, from this lab data.

4. Simulation

A simulation is like a special type of controlled experiment.

They can take many forms.

The main two are statistical and physical simulation.

The “data science” community and all their machine learning-ai-magic are typically statistical simulations.

They use (really) clever maths to determine something of value (i.e. a preference or best decision) from actual or derived relationships between things.

My background is in physics simulation, where we are using our understanding of physics and materials to recreate something on a computer we can play with as if it we in real life.

With both, what we do is poke and prod the simulation model in a realistic way.

You see what the model thinks would happen.

You then infer from that what you think would actually happen.

They are always inaccurate (to some degree) but can still be useful.

5. Signal in the noise

Where you extract an underlying trend to infer something more valuable.

The obvious example here are stock prices. So they are up and down every second. What you may want to know is whether they are improving or not over time.

By applying a moving average or a filter to your stock prices you can infer whether in fact they are moving up or down over various periods of time.

There is a whole subject field on “signal processing” and “frequency analysis” largely focused on the world of music and sound engineering.

I find many people think a moving average is a filter. It isn’t.

Step outside the moving average …

6. Environmental context

Where there is a change in your environmental conditions, you can infer (how that will affect) what you’re interested in.

Context is a strangely complicated thing to define – as it is linked at a granular level to your specific circumstances.

However, typical environmental context could include:

• The weather (i.e. “cold”, “hot”, sunny)
• Seasonality (i.e. Christmas, Mondays, Mornings)
• Health (i.e. illness, pregnancy, fitness)
• Events (i.e. local events, national, international)
• Situation (i.e. indoors/outdoors, home/office, travelling, holiday …)

The thought process is: “Given this context, what can we infer?”

For example, what is the buying preference for paracetamol of a 34 year old Mrs Jones at 10.45am on a Saturday, 3 weeks before Christmas, when the weather is 10 deg and raining and her office had its Christmas party the night before …

If we know this environmental data for others we can infer the effect those factors will have on Mrs Jones’ paracetamol buying preference.

So does the 10 deg weather make her more or less likely to buy? What about the rain? What about the combination of the rain and the temperature? How important is the seasonality? Winter v summer. How significant is the Christmas party? What about the 10.45am time of day?

Finding the right context is often really challenging.

Data people talk about the importance of “domain knowledge” and this is why.

In my example about Mrs Jones, your domain knowledge might be that Christmas parties involve a few drinks. Therefore the morning after she is (intuitively) more likely to want to buy paracetamol (something that can help a hang-over).

Then you find out Mrs Jones is tee-total … etc

You get the picture.

7. What is missing

This is where data that could be there isn’t and means that we can use knowledge to infer what that means.

This links back to the plane.

Have a look at this picture of that plane the Navy were looking to armour.

Each dot is a bullet hole recorded where the planes were shot.

Great! You might be thinking. They did get the data.

Samir – Just put the armour where the bullet holes are? Right?

Not so fast.

They did get this data but it wasn’t from the planes that got shot down.

This data is from the planes that made it back.

That is (critical) context.

If you haven’t worked it out already have another look at that picture.

Where are you going to armour your planes?

Clearly your going to put the armour where the bullet holes ARE NOT.

You are inferring that by the fact that the planes that made it back could have bullet holes in all these areas.

In fact, you have also inferred that where the planes actually needed the armour is where there aren’t any bullet holes.

You have concluded that you’d actually recommend putting armour where there there is no evidence of bullet holes!

Just think about how that would sound …

Therefore, inferred metrics are great but, because it’s not direct measure, you really should to do one final step to put yours (and your bosses!) minds at rest.

The sanity check

Does this make sense?

Look again at the plane picture.

Do we have enough data?

• We could always do with more data, but in this case, the coverage is consistent enough across the whole plane to draw at least broad conclusions. Tick

What is in the areas which have no bullet holes? Are these important for flight ?

First the two easy ones:

• Engines? Tick
• Cockpit? Tick

Now two more that require (some minimal) domain knowledge:

• Rear fuselage? – this holds the tail on. Tail is required. Tick
• Mid wings? – this area holds the fuel tanks. Fuel is needed. It is also explosive. Tick Tick

Does this pass the sanity check then?

Yes.

Inspiration

When you can’t measure something directly it can feel impossible.

Don’t give up.

Use these suggestions to help you consider what inferred metrics you can devise for your situation.

You’ll be making assumptions taking this route. Just be aware of that and be straight with the people you’re working with – let them know that this is imprecise but potentially valuable.

In my experience, imprecise is often better than nothing at all.

See where you can take it.