I Am Finished

Is what my computer said after finding the prices of toasters listed on Froogle.

As you may know, I am a researcher at the Florida Institute for Human and Machine Cognition. For the last year, I’ve been working on a project called PLOW (pdf) for doing Procedure Learning on the Web.

If you call recording macros learning by demonstration, you might call our approach learning by discussion. You don’t just go through the motions, you also say what you’re doing. By “say” I mean speak, talk, vocalize. You tell PLOW what you’re doing, and it listens. From what you say, it’s able to learn what parts of a web page are relevant to the task and why (pdf).

PLOW doesn’t just listen and watch. It can talk too, ask questions, perform actions, and diagram what it’s thinking. Here’s the transcript of my Froogle teaching experience:

Will: Let me show you how to find the cheapest product.

PLOW: All right.

Will: Go to Froogle. I type froogle.com in the address bar of the browser and hit return.

PLOW: Okay.

Will: Put the product info here.

PLOW: Okay.

I type “rice cooker” into Froogle’s search field.

PLOW: Okay.

Will: Click “Search Froogle”. I click on the button labeled “Search Froogle”.

PLOW: All right.

Will: Select sort by price.

PLOW: All right.

The page has a “Sort by relevance” popup button. I click on it revealing the options. I select “Sort by price: low to high” by clicking on it.

PLOW: All right.

Will: Here is the list of products.

PLOW: All right.

I option+click on the list until it’s highlighted. (I added this option+click gesture to the browser. It lets you point to specific elements on a page.)

PLOW: Okay. Plow opens a new window with individual list items delimited. Whatever I do to one item is repeated for the rest. PLOW: Can you check that I understand the data?

Will: Here is the price.

PLOW: All right.

I select the price for one item by clicking and dragging: normal text selection.

PLOW: All right. PLOW highlights the price for the rest of the times displaying the value in a new column. PLOW labels the new column with “the buck” which shows you something about how PLOW understands language. If you tell me story, I won’t be able to repeat it word for word to someone else. I’ll remember what happened but not the exact words. PLOW is similar except it often says funny things since it doesn’t know what sounds most natural.

Will: We’re done.

PLOW: Okay.

Will: Let’s practice.

PLOW: What is the product information? Plow opens a window with a text field where I can make corrections to what it hears before it continues. I type in “toast” and hit return. PLOW goes through the steps it just learned. It starts at froogle.com. It puts “toast” in the search field, clicks the search button, waits for the new page to load, selects “Sort by price: low to high”, waits for that page to load, and finds the list of items. Finally for each item, it remembers the value of “the buck”. PLOW: I am finished.