1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Risk Assessment - discussion

Discussion in 'CFI Corner' started by killbilly, Sep 11, 2017.

  1. killbilly

    killbilly Vocals, Lyrics, Triangle, Washboard, Kittens

    Joined:
    Jul 3, 2006
    Messages:
    9,968
    Likes Received:
    3,624
    While looking something else up, I ran across what the PHAK calls "a more comprehensive risk assessment form." ) It's figure 2-6 in the book.

    I've seen a couple of these before - you answer the questions, there are scored values and the higher the score, the greater the risk to assess. I'm curious about the subjectivity of some of the questions, though. For example - one of the questions is about planning the flight.

    One of the options is "Used Charts And Computer to Assist." - It has a zero score for least risk.
    Another is "Used computer program for all planning." - This has a three-point score.

    There are some pretty sophisticated programs out there which provide some significant SA around flight planning. Hell - ForeFlight alone is pretty comprehensive, and I know there is some pro-grade stuff out there that some of you use which goes beyond that.

    So why - according to the PHAK - is using charts and computer to 'assist' less risky than running the numbers in a computer? What does going with a pure software solution do to increase risk?
     
  2. Mike Wise

    Mike Wise #NewSchool

    Joined:
    Dec 18, 2012
    Messages:
    914
    Likes Received:
    577
    I'm just a CFI Applicant (been waiting months for the checkride...irma messed things up for the third time)...BUT

    I'm going to say maybe just that technology has evolved faster then they could edit the text (if they even cared to)?
     
  3. Roger Roger

    Roger Roger Navajo Whisperer

    Joined:
    Sep 4, 2009
    Messages:
    7,798
    Likes Received:
    9,657
    I would guess that they are assuming that the risk of an un-caught GIGO error is greater when you're punching the numbers into a computer vs getting into the nitty gritty of planning.

    Also maybe that review of what the computer spits out doesn't breed the mental Situation Awareness picture that digging thought the nuts and bolts of the wx and notams does. Just my guess.
     
    Screaming_Emu likes this.
  4. killbilly

    killbilly Vocals, Lyrics, Triangle, Washboard, Kittens

    Joined:
    Jul 3, 2006
    Messages:
    9,968
    Likes Received:
    3,624
    I've considered this possibility - you might be right. The FAA seems oddly dogmatic about certain things and slow to change, yet very practical about others.

    There might be some truth to this - concomitant with the whole "automation creates a different set of new problems." Charts and Mk1-eyeballing are obviously a slower process, which take longer to assimilate info, but how much more SA do you really get? You can look at a radar chart, and you can also log on to Foreflight web and get the same information, but animated. You can work the legs/fuel burn/ETE with an E6B and a Nav Log, or you can punch the knowns into a spreadsheet once and get the same info.

    And you can just as easily make errors with an E6B and charts as you can with software; pilots have been making analog mistakes long before they made them digitally.

    So I get that there is inherent risk in flight planning that needs to be managed, but I'm questioning the dogmatic position that says full software is riskier - at least, without any additional context, it doesn't make sense to me.

    This is one of those things that I'd be a little afraid to argue with a Fed on my CFI oral, too.
     
  5. Jordan93

    Jordan93 Well-Known Member

    Joined:
    Oct 27, 2015
    Messages:
    636
    Likes Received:
    833
    I don't really care what my students use as long as they can still navigate and fly the plane when the technology craps out. Technology will fail. I'll usually make them use paper and an E6B at first, then when they show proficiency, they are free to use whatever they like. If they use the iPad, the location is turned off and they can bet that their iPad will "fail" at least once during the lesson. Same thing goes for W&B. We start off with pen and paper. We learn how to do a manual W&B, effects of forward and aft CG, etc. If you have an app, you don't really know what you're doing. You're just plugging numbers in and the dot on the app is showing if you are in the envelope. After they show proficiency doing a manual W&B and understand the mechanics of it, I don't care what they use.
     
    Bamaaviator likes this.
  6. ppragman

    ppragman Direct BOOKE

    Joined:
    Sep 15, 2007
    Messages:
    9,497
    Likes Received:
    3,418
    Because I doubt there is any science in why each of these were picked.

    This is a huge problem in the world of "risk assessments."
     
    z987k and tcco94 like this.
  7. tcco94

    tcco94 Professional GTA V Pilot

    Joined:
    Feb 16, 2015
    Messages:
    1,937
    Likes Received:
    1,306
    Oh the logic behind the PHAK. Nobody understands it.

    I think using charts can be fatiguing when you compare it to a device like ForeFlight. My students still learned the material just as good, if not better with ForeFlight. The fact examiners nationwide are accepting it (AND encouraging it) shows how valuable it is.
     
  8. Screaming_Emu

    Screaming_Emu You people

    Joined:
    May 29, 2002
    Messages:
    14,806
    Likes Received:
    16,838
    I could definitely see it both ways. When I go flying for work we get a 54 page stack of paper. While most of us try to familiarize ourselves with as much as we can, it would be incredibly easy to simply cherry pick the pieces of info you need off it and do an entire flight with the minimum in situational awareness.

    On the contrary there is a lot of good info in there as well as other flight planning tools that if used correctly can absolutely put you way ahead of the game in terms of SA.

    Like anything else, it really comes down to how the tools are used by the pilot. Hand an artist a chainsaw and a block of wood and they can do something amazing, hand it to an amateur and they might very well kill themselves.
     
  9. MidlifeFlyer

    MidlifeFlyer Well-Known Member

    Joined:
    May 17, 2003
    Messages:
    6,525
    Likes Received:
    1,301
    The guy who wrote it is afraid of computers. We all know folks like that - reasonably intelligent people who just don't get them. They're the ones who never figured out all the menus in Windows programs started with File, Edit.

    But yeah, risk assessment tools are inherently subjective. In simplest terms, the risk of a night flight by a person who has not flown at night in 2 years is different for the pilot who has 500 hours of night experience than it is for the pilot whose only night experience was to meet certificate requirements.

    The goal to me is more about getting you to think about it and, to the extent scoring is helpful, to think about how to score for yourself or your company.
     
  10. ppragman

    ppragman Direct BOOKE

    Joined:
    Sep 15, 2007
    Messages:
    9,497
    Likes Received:
    3,418
    This is a huge problem - the lack of objective risk assessment programs means that there is no actual usable data on how effective or how are useful risk management techniques and training are in preventing aircraft accidents. If it's only purpose is to "get you to think about the risks you're about to take" then it is more likely that we are failing in the training pipeline somewhere else because pilots don't know how to think about "what is dangerous/risky" without the help of a chart.

    I am a huge proponent of risk management tools and techniques, I really like scientific tools. That said, it's been my experience in the field that the tools that are available are not scientific, nor do they effectively manage risk. Consider this, we have terms like, "probable, occasional, remote, improbable" that are inadequately defined in the risk management handbook:

    "An event will occur several times?" What? Per flight, per hour, per day? Per minute? Per second? What is the actual probability threshold? Is it "the even has a 75% probability of occurring on each revenue flight?" Do we know? Is this even knowable? What does a "high score" on a risk assessment really fundamentally mean?

    If we score up a hypothetical freight flying flight that I used to do nightly here's what I get using the sample FRAT on 4-3 of the Risk Management handbook.

    Sleep - I did not sleep well (back side of the clock) +2
    How do I feel - Feel a bit off (it's 3:30am, and I'm about to go fly of couse I feel crummy). +2

    Weather at Termination - IMC Conditions +4

    How is the day going? Great day (night really) +0

    Is the flight - Night? Yup. +3

    Planning: Rush to get off the ground (always flying UPS packages, gotta start moving so the center can show you blocked out!) +3

    Used computer program for all planning - iPad ftw +3

    Did you verify weight and balance - theoretically, they told you what the weights were, that doesn't mean they're accurate. If the tailstand isn't touching you're probably OK +0

    Did you evaluate performance? - not specifically for this turn, but it's a cold winter day, and I do this flight 8 times a week on 10,000' runways, and I know I need 3800' to make accelerate stop...so kinda +0

    Do you brief your passengers on the ground and in flight. N/A +0

    I was at 17 basically EVERY night. If I have a cold, or if I am having, "one of those days" suddenly I'm in the "area of concern" - this FRAT isn't useful to me for that mission, and doesn't tell me much of anything.

    Now let's say I score a recent flight that resulted in a fatal accident with this thing. There was a bonanza that crashed in Ogden, UT in good day, VMC weather, shortly after departure.

    If that guy was having a "good day" and did everything right before takeoff, he would have scored a 2. Even if he had a cold and didn't sleep, he would have scored an 8. Granted, this is kind of an exercise in cherry-picking, because I could pick any accident on a good weather day and screw with the data, but if we are going to have tools, they need to be tools that use real, quantifiable metrics.

    In mathematical modelling, in order for a model to be a valid model, the model has to be testable on real world data, and should fit the data, or at least be able to have some predictive quality to it. I'm not sure that numerical flight risk assessments do that.

    Even the CFIT Risk Assessment in that book isn't all that great. Where do these numbers come from? I've tried to find out, I find a document from the FSF, but beyond that, nothing. The Flight Safety Foundation says, "studies" but won't give me access to them. In short, there's little that leads me to believe that they weren't just made up by someone because they seemed "risky."

    Why does "No ATC Service" score at -30 - the same score as "Limited lighting system?" Why does an NDB approach in Kansas where there's literally nothing to run into score the same as the NDB in Dutch Harbor? This begs a further question - do the minimums of the approach as defined by TERPS provide equal levels of risk at all airports (something tells me the answer is "no").

    Regardless - I love this stuff, but it needs a serious overhaul, and the numbers you get from these risk assessment charts are by and large meaningless. They don't "quantify" anything. We need to do some serious model building to see what exactly a "risky flight" actually looks like - as it stands right now, what we're doing is groping in the dark.
     
  11. killbilly

    killbilly Vocals, Lyrics, Triangle, Washboard, Kittens

    Joined:
    Jul 3, 2006
    Messages:
    9,968
    Likes Received:
    3,624
    Excellent post and points well-reasoned.

    Funny thing - I'm pretty sure if I articulated this argument to the FAA Inspector conducting my CFI initial he'd flunk me as a heretic. :)

    This is good stuff to think about, though. I'm going to bounce the idea off an instructor I know who is an NTSB guy here in DC. See what he says.
     
  12. ppragman

    ppragman Direct BOOKE

    Joined:
    Sep 15, 2007
    Messages:
    9,497
    Likes Received:
    3,418
    Thanks!

    Have him call me if he does! Seriously. The risk assessment tools are basically like codified old wives tales in their current form, and we don't know why, or in many cases "if" they even work.

    I really truly believe we can build an effective model of risk, then use that actively reduce accidents in aviation. We actually have plenty of data - it's just a matter of doing the statistical modeling now.
     
    z987k likes this.
  13. Nark

    Nark Well-Known Member

    Joined:
    Jan 27, 2008
    Messages:
    3,904
    Likes Received:
    3,109
    Pat,

    I haven't looked into the new PHAK, so I'm taking everyone's word for it.

    We do risk assessments before every "mission." Low, moderate, high, extremely high. Each mission has to be approved by the relative authority, company commander up to general officer.

    Like you stated, what is the measurement of the subjective terms?

    The enemy situation in my area puts us at an automatic "moderate."
    Okay, no arguments.
    Landing to a dusty, unimproved surface, also "moderate." Add in night (NVG's of course) and low illumination, again, 3 compounding factors, yup still only "moderate."

    I can tell you from experience, I'd much rather be shot at, than do a 0 illumination, dust landing. (Beer thirty time, I'll tell you about that night) It's extremely nerve racking. I'd rather land to a boat, in pitching seas, kind of nerve racking.

    So again, it's the process design to get you thinking about the potential risk, or does it quantify the risk?
    Does the decision change, based upon the matrix outcome? Or do you still go?
     
    ppragman likes this.
  14. ppragman

    ppragman Direct BOOKE

    Joined:
    Sep 15, 2007
    Messages:
    9,497
    Likes Received:
    3,418
    See this is the issue - with no actual science to back up what "moderate" actually means...well, it's all just make believe. At the vast majority of places I've worked at, when the risk assessment was produced and I ended up in the "caution" range where I had to ask for CP or DO approval, here's what happens:

    DO: do you feel comfortable with the flight?

    Me: uhh, yeah, otherwise I wouldn't have even brought it up to you.

    DO: did you think about X?

    Me: yup, this is how I'm going to mitigate it.

    DO: okay, go ahead and go.

    So ultimately it ends up being a judgment call every time it's questionable anyway. Most of the FRATs I've used have cutoff points for flights with too much risk - but generally they're so high that you'd have to be a total dumbass to try to fly when you're above them anyway:

    "Let's see, its 2:30am, thundersnow, with moderate to severe icing reported in clouds by multiple Alaska Airlines 737s...I don't think im going in the Caravan."

    And again, where does the terminology come from. One guy's high risk is another guy's meh. Humans are notoriously terrible at assessing risk - this is why people are more terrified of home invasions than they are of global warming and heart disease.

    This inability to judge risk is really good for surviving when the biggest threat to your life is a saber tooth tiger or another tribe's warrior that is planning on cracking your skull with a caribou bone, but it does t translate into flying all that well. This is probably why even though the leading cause of accidents is still loss of control some businesses won't let their executives ride in turbine singles, but are totally fine with them riding in Piper Navajos.

    This is also probably why your system puts zero illumination dust landings in the same category as enemy action. I could be wrong, but I would bet money that there are far more accidents due to Brown out than enemy fire - but people's fight or flight gets tripped the second you say "there are armed men who want to kill you" in the area. That doesn't mean there's not risk - rather, I suspect you're right, you're probably much more likely to be killed or injured doing blacked out landings to dusty LZs under NVGs than you are to be shot down.

    So where does the "moderate" come from?

    One of my favorite things in this world is to challenge people's assumptions about risk then watch them squirm a bit. What's amusing is that generally speaking people are very very sure of their risk models even though most of them were flawed.

    For instance I used to go to a "Day VFR" only airport in the Navajo. It was day VFR only because the owner of the airplane (who used to fly tomcats on carriers) said it reminded him too much of night carrier landings at night. We had another airport 45 miles away that was (at least to me) way more terrifying at night. Personally I thought we should avoid both of them at night, but I didn't sign the pay checks and it could be done at night, I just didn't like it.

    How do you score these using the coarse tools people develop? I don't think you can. You need finer tools. To get finer tools you need research, but that costs money. I suspect that most of these FRATs are designed to limit liability to the operator, prevent inflight diversions, and otherwise save money - not to actually reduce risk.
     
  15. MidlifeFlyer

    MidlifeFlyer Well-Known Member

    Joined:
    May 17, 2003
    Messages:
    6,525
    Likes Received:
    1,301
    The FRATs I've seen combine both objective numbers and subjective assessments. This is one that is pretty simple. I've seen more detailed and less detailed but this is pretty illustrative. Most of the items are at least attempts to be relatively objective. Scientific? Probably not. The numbers most likely reflect company or personal minimums rather than massaged data on exactly how much sleep the night before makes a difference. Besides, that can vary from pilot to pilot. Could they be far better? What couldn't? Do they cover everything? No. I notice most don't have a category, "I just feel like crap today," which is completely subjective but probably accounts for a lot of accidents. And, to cover everything, filling out the form would take longer than the flight.

    I guess the danger is thinking things are hunky dory because an inadequate tool gave a low risk score but I'm not willing to go with, "Don't bother with any" just because they are not good enough. I certainly don't have a usable concept for something better. Do you have a sample of one, from any industry or profession, you think makes the grade?
    [​IMG]
     
  16. Roger Roger

    Roger Roger Navajo Whisperer

    Joined:
    Sep 4, 2009
    Messages:
    7,798
    Likes Received:
    9,657
    The only way to get a real world risk assessment tool is to do a massive data dive of various types of accidents and look at what in aggregate objectively increases accident risk, then use that to create a tool. The data is out there. The tough part is getting people to accept it, because often the reality of what is risky collides with people's perceptions (right @Murdoughnut?).
     
    ppragman likes this.
  17. ppragman

    ppragman Direct BOOKE

    Joined:
    Sep 15, 2007
    Messages:
    9,497
    Likes Received:
    3,418
    The problem is this: a subjective method of risk assessment passed off as an objective one may be worse at predicting actual risk than an actual subjective one. We don't know, and that in and of itself is a special kind of problem.

    See, but here's the thing. Why is "destination visibility less than 1 mile" inherently a +20 on the risk. Where does "+20" come from? Typically, well, it comes out of someone's ass. Why does. If the scale only goes to "15" why does there even exist things on the assessment that put you up to 30? You can only go to like -11 or so anyway... Where does the "high risk" designation come from? What I'm saying is that without explanation, these numbers are worthless - they don't come from anywhere.

    Do these numbers ACTUALLY quantify risk. I do not think that they do. For instance, in the "aircraft category" weight within 10% of max gross is a "risk factor." Looking at accident data up here in Alaska shows us that despite being wildly unsafe and illegal, flights being loaded overgross is actually seldom the primary cause of fatal accidents. So if the flight is "within 10% of max gross" is it really that much safer than if it's lighter? I suspect that someone just said, "it's riskier if you're heavier" without doing any actual research on accident data to support that. If you actually "run the numbers" you find that being within 10% of gross weight doesn't really make that much difference in the percentage change in the stall speed:

    [​IMG]

    The redline runs from 50% loaded to 80% loaded, the blue line is 80% loaded to 90%, and the green line is 90% load to 100%. The y-axis represents the percentage difference in stall speed.

    Or, to make this a little bit more clear, the difference in stall speed of an airplane that stalls at 61kts (which is the part 23 requirement to not have to meet the other crashworthiness requirements).

    90% of gross weight means the airplane stall 3.1kts slower than at MGTOW, 80% means the airplane stalls about 6.4 kts slower than max gross.

    For me anyway, this leads me to believe that in most GA airplanes this isn't really the issue we're making it out to be; it's silly to me that we'd quantify a difference of 3kts with as much risk as flying a messed up NDB circling approach, or landing on an icy runway.


    The FAA (and many in industry) give us the false choice of "well, it's better to have this than nothing at all," and I'm not even sure that is correct. Having continuous and better training, doing thorough preflight preparation (not just blasting direct into foreflight and looking at the "standard brief option"), and having a strong mentoring culture in aviation is probably a "better" model than any of this, but those things cost time and money that's not necessarily available. In 10 years of professional flying, I've seen a ton of different philosophies. I've seen zero risk assessment tools on the job, several poor risk assessment tools - but only ONE good one, and it wasn't really a FRAT in the same vein as the ones the FAA promulgates - and even then it was only "ok." The points based ones in my experience tend to be a "mother may I" form rather than a real or objective assessment of risk.

    The one I liked broke the flight down into several parts (because it's stupid to say the entire flight is riskier if you're only exposed to risk for 5 minutes during departure in the terminal environment).

    It had a self-assessment (which was the only part that was somewhat "hokey" at the top of the form). Then it broke each part of the flight (ground ops, departure and climb, enroute, descend approach, landing, ground ops, alternate airport), then had "arcs" of acceptable weather and conditions. It was custom tailored to our flight department, we constantly adjusted it (for instance, in the transition seasons, runway conditions are substantially more critical to monitor than during the summer and winter up north), and any time something didn't make sense we fixed it. We didn't apply any numerical scores to phases of flights - rather, a flight segment was simply green, yellow (which required consultation with a manager), or red - in which case we weren't going. We were regularly tweaking and adjusting the forms, and several airports had their own custom risk assessments when they had special risks that we were aware of. It was dynamic, it was up to date, and it made you think about the conditions in route.

    It didn't purport to suggest that the morning flight after a sleepless night in a hospital arguing with your cancer stricken wife (which is only +5 on the form above) is the same risk as a non-precision approach in mountainous terrain. These aren't even in the same category - we're comparing apples to oranges using the same numbers, and that's not really a good thing. What if you do the approach into the mountainous airport 20 times a day? Does the risk go down? I'd argue that it does, but we'd need serious data to back up these assertions.

    TL;DR - I think most risk assessment forms are a good start, but they are spurious. We need better tools to assess risk in aviation, and no one is doing the research.
     
  18. MidlifeFlyer

    MidlifeFlyer Well-Known Member

    Joined:
    May 17, 2003
    Messages:
    6,525
    Likes Received:
    1,301
    As you can see from my post, no disagreement on that.
     
    ppragman likes this.

Share This Page