Monday, March 05, 2007

Autopilot From Hell

My first job in the Air Force was Automatic Flight Controls Systems Specialist, which is fancy for Autopilot Repairman. After I got out of the Marines that was the job the Air Force felt I could best be useful. I didn’t care; I just wanted to get back into uniform after the unexpectedly long 7 months that it took me to transfer over from the Marines. As it turned out I didn’t mind at all—autopilot systems can be challengingly fun to work on. My last aircraft technical story I entitled “…Hey, it’s technical,” and at the end of it I said I’d eventually write a story about a particularly thorny C-130 autopilot, so here it is.

It was the mid-80s, in hindsight some of the best years of my life. At Yokota Air Base I was happy to get to work on a host of different Air Force aircraft such as the C-5, UH-1, WC-135, C-9, C-141, T-39 and my favorite—the C-130, or the C-130E “Hercules” to be exact.

One of our “Herc’s” was giving us fits. For almost a year it continued to cause great concern to aircrews when for no apparent reason the airplane’s autopilot would cause it to dramatically pitch up or down. It didn’t happen every flight—the problem would come and go. There were times when it would fly perfectly for two or three weeks before it would startle the hell out of some aircrew again.

Over the course of months we changed every component that could conceivably cause the malfunction. We could never duplicate it on the ground, so all we could do was change a likely component and perform a comprehensive operational check and then let it fly again. Of course after doing that we briefed every aircrew to watch out for a possible “pitching autopilot.” Sure enough we would be out there yet again for the umpteenth time trying figure it out. Like every other time though it always seemed to check out perfectly.


After months of beating ourselves to death on this apparently unfixable automatic flight controls system, out of desperation the branch chief instructed us to swap out the complete autopilot system with another airplane that had a perfectly working autopilot. Towards the end we were made to do this at least three times, but to no avail. We spent another few hundred manhours changing out perfectly good connectors, but what else could we do? We had tried everything, and had done so two, three, four, five times already.

The cost in parts alone ran up into the hundreds of thousands of dollars. As we neared the million-dollar mark the “powers that be” decided to ask the “experts” at PACAF headquarters to get us some outside help. A high-level decision was made to bring in an engineer to help us out from Lockheed, the company that had designed and built the C-130 back in the early 50s and continues to do so even to the present.

The man they sent us was an avionics engineer who went by the name of “Big Al.” I wish I could remember his last name, but if I recall right it might have had an Eastern European sound to it. Regardless, Big Al was cool. I thoroughly enjoyed working with the big fellow.

My shopchief, MSgt Jim Friend, assigned me full-time to assist Al during the week he was with us. Being the second highest ranking guy in the shop meant that it was usually me that got these kinds of interesting out of the ordinary projects. That was just fine by me, especially getting the chance to work with Big Al.

Al was a fascinating storyteller. I wish I had written down the airplane development tales he told us, because they would have been great now as Blog entries. When he came to us he must have been in his late 50s and more probably in his 60s. Thirty years before he was an engineer developing the E4 autopilot for use on the at-that-time as yet un-fielded brand new C-130A transport. Now here he was three decades later called out of retirement trying to help the Air Force figure out why one of “his” well-worn autopilots no longer wanted to operate correctly.

Retrorocket C-130 Story

I do hazily remember one of his stories. He said that the particulars were classified for awhile, but I doubt they were by the time he told us of it. He was in on the retro-rocket modification of a C-130 that was supposed to be able to land and take off from a soccer stadium in downtown Tehran as part of a planned hostage rescue plan. The idea was scrubbed he said when during one of the test landings the retro-rocket fired while the airplane was still too high. It came to a halt in midair some 15 or 20 feet above the ground, and when the rockets stopped thrusting it fell earthward in a catastrophic crash. It caused some pretty serious injuries to the test pilots’ backs and wrecked the airplane. The story went that it hit the ground so hard that the wings snapped off at the fuselage. He did say that the rockets worked for taking off, just like a giant harrier jump-jet. Anyway, so much for that idea. But who knows, if they had gotten that retro-rocket C-130 to work maybe we would have successfully rescued our hostages instead of the fiasco that happened at Desert One.

Al brought with him a paper-drum chart recorder; it looked exactly like a polygraph to me only instead of recording bodily function activity Al intended to use it to record a multitude of voltage signals from the autopilot system on our misbehaving C-130. We spent much of a day building a wire harness for it, cutting lengths of 20 or so feet of small gauge blue insulated wire and crimping on terminal lugs. He was thrilled to be able to do most of the work himself saying that for most of his engineering career with Lockheed he was not allowed to do any of that stuff. He gave the orders and technicians did the wire cutting, splicing, crimping and soldering. Sounded like a bunch of union rules nonsense to me—glad I’ve never had to deal with that kind of civilian crap.

We finished the harness and rolled the long length of wires into a large loop ready to take out to the aircraft. Afterwards Al wanted to look at the entire past year’s worth of aircraft forms. We went down to where the records were archived and he spent an hour or two poring through them while occasionally making a few notes as he continued to comb through the hundreds of pencil written pages. When he finished, he got up, stretched and grinned at me saying, “Let’s go. I’ve seen everything I need to.”

Kidding him I said, “So, what do you think? Got it figured out yet?”

Mysteriously, he continued to smile and said noncommittally, “Could be, but I didn’t come all this way without at least flying on it and checking it out.”

“Well Al, we got that pig for the next five days.”

“Okay,” he said, “We’ll see what happens.”

That afternoon we headed out to the plane for the first time and I helped him connect all the wires and power source to his signal charting recorder. Using the C-130E autopilot wiring diagram we connected the ten or so signal input wires from his machine to terminal connections in the autopilot junction box located just inside the avionics aisle, also nicknamed the “hell hole,” immediately to the left past the removed flight deck ladder. Once again he was happy that I “let” him do much of the work. I figured why not—he seemed to enjoy it so much.

All the autopilot voltage taps he installed had one thing in common—they all had something to do with pitch voltages, or signals within the autopilot that would cause the aircraft to nose up or down—like voltages from the altitude hold, pitch hold, automatic elevator trim, and so forth. We found a place on the cargo floor to securely strap the recorder and we were ready to fly for the next five days.

Zero Grav

Our first day in the air was the most notable. Before the crew came out we were there first to make sure the recorder would power up, that the paper drum would properly spool and that each recording stylus would make its tell-tale ink line. Big Al briefed the pilots on what he needed them to do up in the air, which was to basically fly around in a continuous circuit while taking the aircraft through its paces using the autopilot.

We took off with the cavernous cargo bay completely barren; in fact the whole airplane was empty except for the two pilots, a flight engineer, a loadmaster, Big Al and me. Big Al’s first task was to calibrate each signal around a “null” or centered zero voltage. For instance, once at cruise altitude he asked the pilot to make sure the aircraft was in a completely neutral pitch attitude. In other words, to make sure the nose was neither pointing up or down, but absolutely “trimmed” and level with the ground. This was the spot he set each of the recording pens so that a “nose up signal” would record above this “null line” and a “nose down signal” recorded below it. He did this for each signal until all of them were perfectly centered at a zero voltage null. It was ingenious in its simplicity.

It took two or three flying hours for Al to calibrate at null all his chosen pitch voltages, and at first I stayed close, seated next to him on a canvas jump bench seat, intently watching everything he did. Everyone on the plane was on headsets so I could listen to everything as well, especially the interchange between the pilots and Big Al.

After a time I decided to go sit for a while up on the flight deck at the navigator’s table with the seat rotated and locked facing forward. On my way up to the flight deck I passed the loadmaster doing what all loadmasters do so well—he was sound asleep on the canvas seat against the left cargo bulkhead directly across from Al. I used to joke around with my loadmaster buddies that their specialty badge should be crossed pillows.

After taking a seat at the Nav’s table I loosely buckled in and plugged my headset into a jack. I heard Al instructing the pilot that he wanted to calibrate the output from the autopilot pitch thumbwheel and exactly how he wanted the aircraft flown to do it. I wasn’t paying much attention; instead I took great interest at the beauty of the Japanese mountains on the horizon as far as I could see out the front windows.

Big Al told the pilot to give him pitch up on the thumb wheel and the pilot responded to the affirmative with “Roger that, doing so in 3, 2, 1…”

I watched the pilot’s gloved right hand spin the large thumb wheel quickly to the rear and suddenly the mountains disappeared as the nose pitched sharply and alarmingly up. I had never been in a C-130 maneuvered so violently before and it was disconcerting to say the least. Big Al was alarmed too because I heard him say in a slightly urgent voice, “Bring it back, bring it back please!”

This caused the pilot, whose hand was still on the thumbwheel, to spin it almost as quickly back to the forward position. This resulted in a flight maneuver that before that moment I had only read and heard about. The airplane was in zero gravity!

That five or so seconds of weightlessness is locked in my mind’s eye forever. I felt my body float upward away from the earth as if the airplane was trying to fly down and away from me. It is the exact same feeling you get on a rollercoaster as it barrels earthward. My stomach went straight up into my throat as my butt was no longer even touching my seat, being a good 3 or 4 inches above it, restrained only by my loose seatbelt.

What really fills my memory though is what I saw. In a split second the air was filled with stuff! By stuff I mean everything imaginable. Three of us were drinking from Styrofoam cups filled with coffee, and out of the blue those little white cups and the brown liquid within them floated freely about us in the air. The coffee wandered its way out of the containers, broke up into little droplets and dispersed in a brown liquidy cloud. I was stunned by all the dirt and dust that also suddenly filled almost every square inch of air around me as if it just appeared from nowhere. Amazing also was all the bits of hardware now at eyelevel such as tiny screws and bits of wire. ‘Where did all this stuff come from?’ I wondered. I remember the flightdeck being almost spotless and well ordered, yet here was all this trash suddenly appearing from nowhere. Also in the air was everything else we hadn’t tied down—uniform items, hats, coats, aircraft manuals, paper, tools, you name it—it was in the air in a jumbled drifting clutter.

Then, as if by magic the moment of zero grav ended and did so literally with a THUMP! I heard all the stuff hit the deck at exactly the same time and it was a sound that resounded about the aircraft. The loadmaster and Big Al had suddenly found themselves against the ceiling of the cargo bay—a good eight or nine feet up. Al was unhurt being able to grab hold of the canvas seat back, but the poor blissfully sleeping “loady” was badly hurt by his quick “trip” to the top of the airplane and nasty return voyage back to the cargo deck. His body made the loudest thump of all followed by an anguished howl as he gave voice to unexpected pain and shocked surprise. Poor guy, but he should have known better.

It took me a good 15 minutes to physically recover. As soon as my butt became replanted to my seat where it belonged I felt my mouth fill with saliva as I turned green around the gills with nausea. I took some deep breaths and struggled against vomiting. Big Al never missed a beat and after he and the flight engineer saw to the “wounded” loadmaster he and the pilot got back to work nulling out his signal voltages.

I suppose the pilot just wasn’t thinking when he made that zero gravity maneuver, otherwise I’m sure he would have warned us or just wouldn’t have done it. Surely he must have known what was going to happen when he quickly spun that pitch wheel backwards and forwards like that. Still, I’m glad he did it—just for the experience.

The unfortunate loadmaster ended up with a small cut on the side of his face, some bruised ribs, a wrenched shoulder and a twisted knee. Nothing was broken and the bleeding wasn’t serious so he asked that we continue with our mission. I think he was a little embarrassed that he hadn’t strapped in knowing full well that we were going to do some out of the ordinary flying.

For the next five days nothing else of note happened. In truth, I spent most of the five plus hours of air time per day just as sound asleep as the loadmaster. After the fourth day I was having lunch with Big Al and he commented to me that the autopilot appeared to be working as good as or better than any he had ever seen. I chuckled in response, “That makes sense… since over the last year we replaced every part and half its connectors, …not to mention we’ve tweaked the hell out of the damned thing like you wouldn’t believe. …So what do you think Al, is it fixed or what? Fess up man!”

He just grinned and chuckled back, “Well, truthfully… I don’t think there’s a thing wrong with that airplane. But, we’ll continue our checks just to make sure. Only one more day of flying and I’ll fill you in when I’m absolutely sure.”

I knew then that he knew something about that airplane that he must have discovered during his search through the aircraft forms, but he still wasn’t telling.

*** Part Two – Al Figures it Out, or Does He? ***

Our last day of “troubleshoot flying” was a doozy. Big Al had the pilot fly the hell out of that autopilot from hell. We climbed and dived, and swooped and soared, all while using every possible combination of autopilot commands and modes. After the first day I started taking Dramamine and I was happy that I had, especially for that last day. The final result, after five hours of putting the aircraft through its paces—the autopilot worked like a champ, just as Al had inferred that it would the day before.

Al told me thoughtfully, “Come on PJ lets go back to the forms. I want to show you something.”

After thanking and bidding a so-long to the aircrew we climbed onto the waiting avionics stepvan and headed back to the autopilot shop. From there I drove us in my little Japanese sedan back to the office where the archived aircraft forms were stored. Al and I sat down at a table and he began flipping through the pages sorted by month. “Take a look at this,” he said.

The first few pages he showed me were of entries from just a few days before his arrival. These entries showed the removal and replacement of a trim servo motor flex drive. Pointing at it with his big index finger he announced, “THIS is what fixed the autopilot PJ. There was never anything wrong with it. This flex drive was the culprit all along.”

I shrugged, “Al, what the heck is a flex drive? I still don’t get what you’re saying. What does it have to do with the autopilot?”

He explained putting on his “engineer hat,” “Okay, so you understand how trim tabs work, right? Airflow over a small surface called a trim tab located along the trailing edge of a primary flight control surface causes that larger primary surface to move, or to trim.”

“Okay. Yes, go on. I’m following you,” I nodded.

He continued “The C-130 elevator trim tab servo motor is not mounted in the exact center of the aircraft; it is offset—to the side. This motor causes the tabs to move up or down by spinning the flex drives attached to either side of it, one going to the left trim tab and the other to the right. The flex drives are flexible and act just like a speedometer cable where an internal flexible shaft spins within a stiff yet flexible protective housing.”

The first light went on in my head. “Okay, I can picture that.”

“Because the trim tab motor is offset, there is a long flex drive and a short one. They have two different part numbers. Up until this date—he pointed to the date of the forms entry—this aircraft had TWO LONG flex drives installed.”

“Okay, I gotya. Go on…”

“The long one mistakenly in the short side eventually got kinked so bad that it became harder and harder for the internal shaft to spin freely. Here look…”

He began to thumb through the reams of forms, and every few months he found entries where the trim tab motor was replaced time and time again. “You see? The motor had to work way too hard and would eventually burn up after a few weeks. I’m surprised your people didn’t figure it out months ago that the wrong flex drive was in that airplane. They kept changing the motor without trying to figure out what was causing it to burn up. When they fixed this flex drive problem they not only stopped burning up trim tab motors, they fixed your autopilot problem as well.”

“Okay, that sounds feasible, but I remember several times that we asked the AR shop (Aircraft rigging) to check out the elevator trim and they always said it operated perfectly fine. So why was the elevator trim operating fine for them and NOT for us?”

Big Al continued, “When the autopilot trim tab adapter sends its signal to correct a longstanding pitch error to the elevator trim motor it does so incrementally. Inside the adapter is a spinning cam that turns at just 23 times per minute that puts out small pulses of voltage to the servo motor. It works to make the elevator trim tabs move very very slowly. On the other hand, when the trim motor is operated manually it applies as much voltage as necessary to get it to move. The operator simply holds the switch for as long as required to get the tab to move where it’s needed.”

About then, another light went on for me.

Al was on a roll. He went on, “The autopilot’s small pulses of voltage simply could not overcome the friction of that kinked flexshaft until finally there was enough torque built up to make it break free. When it did, the elevator tabs would snap into their new position and THAT is what caused the aircraft to sharply pitch up or down, which probably made the aircrew crap in their flightsuits!” He chuckled.

I grinned and slapped Big Al on the shoulder. “Man, its all so obvious now. Why didn’t WE see it all this time? You knew right away from the first day you looked at these forms, didn’t you?” I accused him good-naturedly.

He just smiled. I still remember his big balding head, large nose and glasses.

****** “Unlikely?!” *******

We went back to my shop and briefed Al’s findings to my shopchief, Sgt. Friend, as well as to the branch chief, Chief Fountaine. The chief asked me to take Al down to the AR shop to brief the shopchief there on what we found. I drove Big Al down to the hangar where the Aircraft Rigging Shop was located. The man in charge of the shop was an older technical sergeant, an angular black fellow with a clipped moustache. I introduced Big Al to the guy and he didn’t even ask us to sit. He wasn’t the least bit impressed with the big engineer’s credentials, and to me it seemed that the guy was already pre-armed against what we were about to tell him. In a phrase, he was chillily defensive.

Big Al took ten or fifteen minutes to fully explain to the unhappy glowering fellow what had been causing the autopilot problems with me chirping in on occasion. When Al finished there was a few seconds of uncomfortable silence which ended with a snort from the quietly angry shopchief, “Unlikely!” was all he had to say to Al’s perfectly reasonable elucidation.

Jerking forward toward this obvious dunderhead I exploded, “WHAT!? UNLIKELY? You have GOT to be kidding me!”

Al grabbed hold of my arm and gently restrained me saying, “Never mind Sergeant Spear. It doesn’t matter…” He turned his attention to the disgruntled Tech Sgt and dismissed himself saying politely, “Thanks for your time. It was nice meeting you.” After which Big Al cheerfully offered his hand to the man who only grudgingly shook it. Al then pulled me out of that tense little office by pulling on my elbow.

I left with him but I shook my head in disgust on the way out. That wasn’t the first time and it certainly wasn’t the last time that I had to deal with arrogance from another shop member over “whose system was actually causing an aircraft malfunction.”

To conclude this longwinded little novel I have to say again that working with “Big Al” was one of the highlights of my aircraft maintenance life. To me it was like working with a legend, and he was certainly a gentleman. Often while we worked on our troubleshooting project he’d tell stories of how the C-130 autopilot turned out the way it did. For instance, I’d always wondered about the big torque limiting resistors permanently mounted against the right bulkhead all the way in the rear of the “hell hole.” Turns out those were his “babies,” devices he came up with that until he did so, the autopilot would not work properly.

I know it all sounds unremarkable, even boring to most non-maintenance types, but to me it was fascinating stuff. Big Al was like a time-machine; talking to him enabled me the chance to glimpse the murky past from someone who had been there. If anyone else out there reading this ever knew or knows Big Al, please let me know. One of the things I’d love to know is HIS LAST NAME! Big Al was one cool big old engineer. It would be great if he was still around somewhere out there.



Anonymous said...

Really interesting and I'm waiting for the next installment. I'm a former AF avionics guy, many years on 130 gunships out of Ubon, Thailand. We had a similar problem on an F4, turned out to be associated with an unshielded UHF radio cable which ran parallel to the autopilot cable. Every time the pilot keyed the mike the plane would pitch violently.

PhilippinesPhil said...

Ahhh, F4s were renowned for their problems resulting from unshielded RF. I don't know how many times I've heard that. I worked 'em for a while and never became a fan of the Phantom.

Speaking of "stray RF," on the C130, after the new HFs were installed back in the early 90s, we began to experience engine rollbacks when the HFs were keyed, always a big concern as you can imagine. The new HF radios were multitudes more powerful than the old ones, and believe it or not there was an unshielded short run of cable at the top of the cargo bay where the antennas entered the fuselage. The old HFs were weak and didn't cause a problem, but the new more powerful transmissions sprayed RF energy directly into the inside of the plane at that spot. Not more than a foot or two away from it was the wiring runs from the engines. This wiring was shielded, but in most cases was 30 odd years old and faulty.

So here's what happened...The RF would "fool" the tach system into thinking the engine was over speeding and so would pull it back. I and a few other fellows made a few bucks off that one under the suggestion program. I was part of the team that figured it out, and then put into effect a fix by providing a large oversized shield.

So anyway, stay tuned and I'll finish this autopilot story soon. I promise.

Kevin said...

Vomit Comet!

I always wanted to do that. I read that you can pay somebody down in Florida (maybe it's NASA) $3500.00 to go for a weightless ride. You got to do it for free. (sorta)

I'm pretty sure I would've barfed up my underwear, though.

PhilippinesPhil said...

Ahhhh. Finally, it's done.

I didn't get sick Kev, but it was close. Dramamine is a Godsend!

Anonymous said...

Thanks Phil,

I loved this account of the E4 auto-pilot from hell, and will try to use it in abbreviated form for a Diploma course scenario. Since we are using the E4 system as a generic example of a 3-axis AFCS, and then doing practical fault-finding on a bench mounted system, I will probably wrap this story up as a fault scenario without the answer and see if the engineers can come up with a solution.
Not the most modern system anymore, but the good ole E4 does teach you about the various signal loops and interlocks.
For those who remember it with fondness, still going strong on the bench, and providing brilliant fault-finding training. I have lots of spare amps, but those valves seem to give virtually no trouble, they are all still running fine, just as well, as us current generation techs are not quite as comfortable trouble shooting triodes, pentodes etc...

PhilippinesPhil said...

Awesome Kevin. I can't tell you how gratifying it knowing that my little anecdotal E4 Autopilot troubleshooting post will be used for something so real world practical. Thanks!