Share and discuss this blog

Tuesday, February 14, 2017

Oh Accenture, you spent all that money and you learned nothing

Learning technology has its fads. One by one they are adopted by the big corporations who have one real goal: don’t spend a great deal of time on training people. 

In 1989, I was hired by Andersen Consulting (now Accenture) as a consultant to help improve their training which was mostly lecture based, at a training facility in St Charles, Illinois. Simultaneously, they gave Northwestern University a great deal of money enabling me to found the Institute for the Learning Sciences that tried to invent new ways of learning on the computer, and also took on ten Andersen people for two years master’s degree programs with the aim that they would bring back new ways of thinking about training to Andersen.

I happened to look at Accenture’s training site the other day:

I was curious what they were doing these days and wondering if I had had any effect on them. It was easy to draw two conclusions:

1- Accenture is now obsessed with the idea that all courses should last an hour and should be online. They are closing (or have already closed) the St Charles facility.

2. They learned exactly one thing from me. They learned that learning objectives for courses should be about doing rather than knowing. It doesn’t matter what people know (typically have memorized.) What matters is what people can do that they could not do before.

There seem to be hundreds of courses available. If you hit Risk Management, for example, about 100 course titles are listed. All of them seem to be one hour long (I didn’t look at every single one) and all of them have Learning Objectives that read like this one for Overcoming Challenges in Asset Management.

After completing this course, you should be able to:
Identify the top pressures and risks for asset management.
Recognize how top companies are addressing the growing talent gap.
Describe how to use remote asset connectivity for an effective asset management strategy.
Define how predictive analytics can strengthen your asset management strategy.

This uses the language of doing but objectives that start with “define how” are not really doing objectives.

And what does this one hour course involve the trainee in doing? It involves them listening to a one hour “webinar.” (The person giving this speech is working on a masters degree in statistics.)

Let’s look at another: Analyzing the Core Elements of the Strategic Plan. This one is one hour of online self study (which I suppose means one hour of reading). Its learning objectives are: 

After completing this course, you should be able to:
Determine how to create an effective mission statement at both the corporate and the divisional levels.
Recognize the role of objectives in the strategic planning process.
Identify the characteristics of an effective strategy.
Explain the function of tactics in a strategic plan.

I am impressed. I didn't know you could learn all that from an hour of reading. (The author of this course has an M.S. in Finance.)

So, clearly, I failed. I tried to change Accenture’s approach to training but failed. I already knew this because I hear from my former Andersen students from time to time and almost none of them are still at Accenture. And, ironically, my major clients are Accenture’s direct competitors. So, while I appreciate Andersen's help to me, so do their competitors. Accenture itself seems not to have learned much from me.

But, my real issue here, is trying to understand what you can learn and how you can learn, in an hour, since it this is now a fad that is driving demand for more and more one hour courses.

With this in mind, I asked myself what I might ever have learned in an hour (in a lifetime of trying and failing at many things.)

The first class (part of a graduate AI course) I ever taught (at Stanford in 1969) taught me a great deal in one hour. I learned how students at Stanford thought, what they paid attention to, and how Computer Science graduate students differed from me on what was important to think about. So I did learn a lot in hour. How?

I performed in front of people, tried to convey information by talking, and reflected on the kinds of responses I was getting back. Of course, I did this many times since I taught more than one hour of that course.

So, did I learn in an hour? Yes. Did I learn a lot? Yes. What motivated me to learn? I had to evaluate my own effectiveness.

In that same year, I learned something else in an hour. I taught one hour of a very different kind of course that was meant to encourage first year graduate students to sign up for a more intensive second semester course with the various faculty members who had that course. I was teamed with a guy named Ken Colby. He made people laugh when he talked. I didn’t. I had a lot to say in my hour. In the end our team signed up a large number of students. I was very proud of myself. I soon found out that they had all signed up because of him, not me. I asked him what I had done wrong. He said “you told them everything you know in one hour. If you can do that you don’t know much.” As you can see, I never forgot that. 

So, in effect I learned how to speak in an hour. One hour of failure and one comment from someone respected. I resolved to become a better speaker.  

My father once told me a story about how he had mistreated another lawyer when he was clerking after graduating from law school because that guy was dumb and he had graduated from a law school inferior to the one my father had attended. The punch line of his story was that that guy became the Chief Justice of the Supreme Court of New York. (My father hadn’t done much of note in his life.)

That was a lesson that was taught in ten minutes. But I never forgot it. In fact I learned two important things from that story. 

The first was not to be such a big shot because of what school you attended or how smart you are. I am sure that my father was unimpressed with how I had acted at some point which prompted the telling of the story.

The second thing I learned, after I reflected on this for some time, was the importance of just-in-time the story telling, which has served as the basis of my work in AI and in education.

The moral here is clear: stories are powerful and you can learn from them in a lot less than an hour, if, and only if, they are told by the right person at the right time.

So, I believe you can learn something in an hour. Here are some other things I learned in an hour. Each has a story associated with it, but in the interest of brevity I will omit the context of each:

1. Listen to what people tell you about themselves, they mean it.

2. In the academic world, be careful whom you attack.

3. Don’t assume you know the reason why things are the way they are. Dig into it and find out for yourself.

4. Politicians love to talk about education, but they really don’t give a damn.

5. Changing school is harder than simply making suggestions about what they should do.

6. You are not really encouraged to have your own point of view in college.

7. Life has its way of evening the score. 

8. Nobel Prizes aren’t awarded to revolutionaries. 

9. Genetics is powerful stuff. Most people don’t realize the extent to which their likes and dislikes and their ability to thrive under various conditions stems from thousands of years of evolution.

10. Smart is easy, but you are more likely to get a job by being smart and appearing to be cool. 

11. You know when you have won in sports. In real life, victory is never so clear cut.  

12. Education needs to be personalized and local at just the time when the country is trying to make it into one size fits all.

13. You can learn more by thinking about something and trying stuff out than you can by asking an authority for advice.

14. People rarely listen to the advice you give them

15. To produce great students help them to frame their questions and encourage hard answers. Asking a good question is much harder than answering one.  

16. Doctors seem to diagnose what they know, so find out what they know before you ask them whats wrong with you.   

17. Children are awful judges of their own childhood experiences, even as adults.

18. In the end, everyone just wants someone to pay attention to them. Good parenting is about paying attention while not overburdening the child with that attention.

19. Universities do not want to be “training school.” If you want to learn to do something practical, universities are probably not the place.  

20. You don’t really know what freedom is until you lose it.  

21. The goal of investors is to sell. The goal of inventors is to create. This always leads to conflicts.

All of these heave personal stories attached to them — experiences that taught me an important lesson in an hour.

So, can we build effective one hour courses? Yes. But they would have to not try simply to tell you something. They would need to put you in a situation you were in from which you could discover things about yourself and about the world.  Talking at people and telling them they will be learning to do things as a result of listening is simply wrong. We learn from actual experience and from reflection on that experience.

Here is a hint for Accenture. Instead of listing courses by their are and title, try cataloging the problems that you employees have on the job and create courses that help you resolve issues and problems that you have encountered. These shouldn't be listed alphabetically either. People think in terms of goals, and plans to achieve those goals, and problems they have encountered along the way.

Tuesday, January 31, 2017

My cousin Imre had to go to Canada when the U.S. kept him out. Worked out well for him. But, maybe that wasn't and isn't a great plan.

I don’t usually write political pieces, but today’s news reminds me of my cousin Imre. He grew up in what is now Slovakia, escaped to London as part of the government during the war and returned home to help after the war. He was forced to leave Czechoslovakia after the Russians arrived. He tried to emigrate to the U.S. He was denied entry. His uncle, my grandfather, had lived in the U.S. for over 40 years, but there was nothing my grandfather could do to get him. He visited him in New York in 1946 when he was concerned about staying in Europe  but was sent back. Canada let him in. He was always “Imre from Canada” to me. (There was also an “Imre from Vienna” in our family.)
As someone who now lives in Canada in the summer, I realize that nearly every Jew that I meet there has the same story. Their family was denied entrance to the U.S. in the 30’s and 40’s, so they went to Canada.
We have had presidents who stopped immigration for reasons best left to their own reasoning. I am no fan of Franklin Roosevelt, but I note that he never gets vilified for this. Today, we are very worried about Mr. Trump’s actions. I can say in his defense that we ought to be used to Presidential decisions that affect people badly. (Would we have ISIS today if Woodrow Wilson had kept us out WW1?) Maybe Mr. Trump is right. Who knows? Mr Roosevelt wasn't right. Many members of my family died because of his decisions. Keeping my cousin Imre out of the U.S. was Canada’s gain. 

Here is a link to a site about him:

Here is an excerpt about his life taken from this: 

lmrich Yitzhak Rosenberg. Bom in NoveMesto,Slovaki& May22,1913. Active in national Jewish youth movement. Doctorate in government and law, Bratislava University, 1939. After two interruptions to work underground against Hitler. While attending the Academy for International Law in The Hague, he helped organize the escapes of Jews from Czechoslovakia and Berlin in 1939. "I landed in England the day before the war broke out, to buy a boat to move refugees," he recalls.  "Every decent person who was safe in London wanted to join a unit fighting Hitler." He joined the Czech army and helped build the London-based resistance movement, led by Edvard Benes, which was eventually recognized as the legitimate democratic government- in-exile in Czechoslovakia. He headed for Czechoslovakia as the war ended, travelling with the Soviet Army as it liberated concentration camps in Poland, Germany, Austria and Czechoslovakia. "The world that had been closed in by the Nazis was being opened up. I was one of the first people to visit Theresienstadt camp, north of Prague, and I selected 301 orphaned children for transfer to England. The British Home Office sent 16 planes to pick them up. I also chose adults, most of them already with relatives in England, to accompany the children - about one for every ten children." There was also a steady stream of Jews arriving in Czechoslovakia from camps in Germany's Ruhr valley. "I don't think there is another living person who has seen as much as I in terms of broken people who survived the camps." He says now in a tone of wonder, "I was a young man when I was doing these things -today I think it was impossible. You're dealing with 180,000 people moving across the border. We gave them medical help, money, tickets. I heard about (Foreign Minister Jan) Masaryk's death from a street cleaner at seven in the morning, though the government didn't announce it until the afternoon. He was pushed from a window; there is proof. I slipped out of the country but my first wife made a mistake and she was caught. She was in jail, terrible jail, for twelve years." Having learned he'd been sentenced in absentia to life imprisonment, Rosenberg spent a year in England waiting for a Canadian visa.   When he arrived in Ottawa as a landed immigrant he was turned down by the Civil Service Commission and instead worked as a laborer, carrying vegetables in Byward Market. Eventually he started lecturing at Ottawa University and selling houses, becoming a partner in a successful real estate company. Along the way he donated $12,000 to establish a home for international students and public servants ("so others would not be left out in the cold in Ottawa, as I was”.)  

I believe in vetting potential immigrants. But there are many potential Imre Rosenberg’s out there. Let’s give them the chance that FDR didn’t give my cousin Imre.

Monday, January 23, 2017

To fight a Cyber War we need to train more people

I am of the age where the major preoccupation in my youth was avoiding having to go fight in the Viet Nam War. Ironically, during those years I was supported in my work by the U.S. Defense Department (DOD). I justified doing this because I wasn’t helping anyone kill anyone. If my work was to be useful to DOD at all, it would be helping us defend ourselves.

But now I am working with DOD on offense. What happened?

In the last year I have become more involved with cyber security. Why? I, and the people who work for me, primarily build online learn by doing courses (using live mentors to help when students are confused and to provide feedback on their work. When the DOD began talking to me about building a cyber operations course for them, I was interested. Developing new ways to learn is my business after all. So, I listened. I interviewed hackers employed by DOD and other federal agencies,  attended DefCon (a hacker convention held in Las Vegas) and over time I recognized what plenty of other people already knew. 

Here are some recent news stories I found about cyber attacks:

Today: Lloyds cyber-attack details emerge

Today: As attacks grow, EU mulls banking stress tests for cyber risks

Two weeks ago: Ukraine power cut 'was cyber-attack’

Two weeks ago: London NHS hospital trust hit by cyber-attack

Two weeks ago: Indian banks are waking up to a new kind of cyber attack

Three Weeks ago: U.S. Grid in ‘Imminent Danger’ From Cyber-Attack, Study Says

This is a serious issue and I want to help. We are, right now, building a course in cyber security. The Pentagon has a serious problem. Here is what Frank DiGiovanni, the Director of Force Training in the office of the Assistant Secretary of Defense for Readiness has to say: "The security of our nation is at stake. I think it’s imperative for DoD to embrace the hacker community because of the unique skills they bring to the table. They want to serve and contribute, and the nation needs them.”

This is from

DiGiovanni built an instructor led course, but there are limits to how many people can be taught face to face. Lecturing is not a really effective method for learning how to do something. We learn how to do things with one on one mentoring and we learn from trying and failing. 

DiGiovanni knows this:
“We infused the course with sociology, ethnography and anthropology.… You don’t conduct an assault on the enemy if you don’t know the terrain they’re in, what surrounds them.”

The social science disciplines help students better understand who they’re up against and why. Those facts can then be aligned with what we know of adversary’s signature techniques, tactics, and procedures.
“Techniques give clues about who they are and could also tip off what you’re after,” DiGiovanni says. This includes the way adversaries might seek to cover their tracks. For example, Russia adapted the concept of maskirovka – literally, masking –from conventional battlefield usage and applied it to the cyber arena. Students learn to identify the tactics of different adversaries, as well as the techniques that can be employed to cover one’s tracks. They have to become adept at identifying what the adversary is doing as well as executing their own cyber missions without leaving digital fingerprints in their wake.

“The biggest complaint about journeyman-apprentice is: It doesn’t scale,” DiGiovanni says. That makes it more costly and slower, compared to traditional teaching methods. Journeyman-apprentice is another core concept built into this course.

DiGiovanni doesn’t want to ditch the approach, just find a way to make it more efficient.

So, DOD contracted with my company to build a course to train tens of thousands.

We will help the military fill its large need for hackers by creating hackers. DOD wants to teach offense and defense. We can’t just simply sit back and defend, we have to frighten the enemy to stop as well as get into their systems. This is a lot like building missiles to defend against missiles.  

It crossed my mind that I could re-employ the defense part of the course and use it to train people who work for companies that may be subject to attack. I discussed this with one of the hackers whom we rely on as an subject matter experts in our course. I was told that I had it wrong. In fact, I had a lot wrong (after all why would I know?) 

Some stuff I (and most people) had wrong.

  1. Students would need to be people who can program (not true)
  2. Companies can hire the people they need. (They can’t be found)
  3. There must be some existing courses to train more (There are but they are short, or lecture based, or generally like most courses that try to teach complex skills quickly without using learning by doing with lots of practice and help.)  
  4. Businesses need defenders not attackers (This is completely wrong because some of the best cyber people are penetration testers who break into their own company’s systems to find out where they are vulnerable.)

We have developed just enough right now to be able to try it out on people who want to help. We are finding the oddest of people who want to do this (a massage therapist, an acupuncturist (OK, she was a computer scientist before she retired), a recent H.S graduate taking a gap year, and the former head of research at a big consulting firm. They are getting good at this and love it. (You need to be someone who gets into complex puzzles and generally thinks breaking into things is fun.)

We have a public website (which is changing very day) if you want to see more.

Below is a something from the first page of the course which lists what students will learn to do:

I am excited about this because I think it matters. Personally, I like having electricity and knowing that my money is secure when I use a bank.

Tuesday, January 10, 2017

Ten Questions about AI

I have had it with the stream of articles about what an “AI” can do. Yes, machine learning works. It is possible to analyze key words, correlate them with other key words, do a massive amount of statistics, and find out some stuff. People cannot do that and computers can. Is this AI? It sure isn't anything people can do, and it also doesn't correspond to anything I understand about what it means to be intelligent.

So, to make clear what AI is really about I propose the ten question test. Here are  ten questions that any person could easily answer and that no computer can answer well. Since I consider myself an AI person I would not say that no computer will ever be able to answer these. I hope we can figure it out. But AI isn’t here yet, despite what venture capitalists, giant companies, and the media keep saying. 

These questions can help explain why.

1. What would be the first question you would ask Bob Dylan if you were to meet him?

I am using this one because IBM’s Watson “met” Bob Dylan and told him that his songs were about love fading. First, I might point out that that conversation is insane. If you were to meet Bob Dylan you might have some things you'd want to know. I’d like to know if he feels that his songs are “literature." I’d also like to know if he thinks he helped a generation feel stronger about protesting injustice and war. I would not count all the words he used and tell him which word appears most often. Watson does not behave as intelligent entities do. Intelligent entities are curious.  They have things they want to know and can recognize who can answer questions that come to their minds about different arenas of life.

Here is another: 

2. Your friend told you, after you invited him for dinner, that he had just ordered pizza. What will he eat? Will he use a knife and fork. Why won’t he change his plans?

You will notice that eating is not mentioned in question 2. Neither are utensils. So how could an “AI” understand these questions.  It would have to know about how people function in daily life. It would have to know that we eat what we order, and that when we say we ordered food it means that we intend to eat it, and it also means that we don't want to waste it. It would also have to know that pizza is typically eaten with one’s hands. It might also know that Donald Trump famously eats pizza with a knife and fork and might mention that when asked.

3. I am thinking of driving to New York from my home in Florida next week. What do you think?

In order to answer the above question, one would need a model of why people ask questions like that one. It is hard to answer if you don’t know the person who is asking. If you do know that person you would also know something about what he is really asking. Does he have a car that is too old to make the trip? Maybe he has a brand new car and he is asking your advice about whether a long trip is a good way to break it in. Maybe he knows you live in New York and might have an idea whether the roads are icy there. Real conversation involves people who make assessments about each other and know what to say to whom based on their previous relationship and what they know about each other. Maybe the asker is really asking about a place to stay along the way (if the person being asked lives in Virginia say.) Sorry, but no “AI” is anywhere near being able to have such a conversation because modern AI is not building complex models of what we know about each other.

4. Who do you love more, your parents, your spouse, or your dog?

What does this question mean and why would anyone ask it? Maybe the person being asked is hugging their dog all the time. Maybe the person being asked is constantly talking about his or her parents. People ask questions as well as answer them. Is there an  “AI” that is observing the world and getting curious enough to ask a question about the inner feelings of someone with whom it is interacting. People do this all the time. “AI’s” do not.

5. My friend’s son wants to drop out of high school and learn car repair. I told her to send him over. What advice do you think I gave him?

If you know me, you would know how I feel about kids being able to follow their own interests despite what school wants to teach. So an intelligent entity that I told this to would probably be able to guess what I said. Can you? No “AI” could.

6. I just saw an ad for IBM’s Watson. It says it can help me make smarter decisions. Can it?

My guess is that this is something Watson can do. It can analyze data, and with more information a person can make better decisions. Could Watson make the decision? Of course not. Decision making involves prioritizing goals and being able to anticipate the consequences of actions. Watson can do none of that.

7. Suppose you wanted to write a novel and you met Stephen King. What would you ask him?

I have no idea what IBM is trying to say to the general public here. Apparently IBM  is very proud that it can count how many times an author says the word “love.” If I wanted advice on writing a novel I doubt I would ask Stephen King, but here is one thing that is sure. Watson wouldn't understand  anything he said about writing a novel and Watson won’t be writing any novels any time soon. Now as it happens my AI group frequently worked on getting computers to write stories of one sort or a another. We learned a lot from doing that. I am quite sure that IBM hasn’t even thought about what is involved in getting a computer to write novels. Having something the computer wants to say? Having had an experience that the computer is bursting to describe to people? That would be AI.

8. Is there anything else I need to know?

When might yir ask such a question? You might have had a conversation with a chat bot and found out how to get somewhere you were trying go. Then you might (if you were talking to a person) ask if there is anything else you needed to know. Answering that question involves knowing whom you are talking to. (Oh, yeah, there is great Ethiopian Restaurant nearby and watch out for speed traps.) Let’s see the chat bot that can answer that question.


9. I can’t figure out how to grow my business. Got any ideas?

It is obvious why this is a difficult question. But, in business, people have conversations like that all the time. They use their prior experiences to predict future experiences. They make suggestions based on stuff they have themselves have done. They give advice based on cases in their own lives and they usually tell personal stories to illustrate their points. That is what intelligent conversation sounds like. Can AI do that? Not today, but it is possible. Unfortunately there is no one that I know of who is working on that. Instead they are working on counting words and matching syntactic phrases.

They are also working on AI document checkers that will help Word with spell check, or grammar check. “NeuroGrammar™ uses its advanced neural-network artificial intelligence algorithms in order to analyse every noun phrase and verb phrase in every sentence for syntactic and semantic errors.”

How marvelous. So here is my last question:

10. Does what I am writing make sense?

Amazingly, this is hard. Why? Because in order to understand my points you need to match them to things you already think and see if I have helped you think about things better or decide that you disagree with what I am saying here based on your own beliefs.  You already have an opinion on whether my writing style was comprehensible and whether the points I made made sense to you. You can do that. AI cannot.   Do I think we could do that someday in AI? Maybe. We would have to have a complete model of the world and an understanding of what kinds of ideas people argue for and what counterarguments are reasonable. Intelligent people all do this. “AI’s” do not. An “AI” that understood documents would  not be a grammar checker.

It would be nice if people stopped pushing AI that is based on statistics and word counts and “AI people” tried to do the hard work that making AI happen would require.