Chris Boese's Weblog

A significant trend? Or a two-year anomaly?

Link: New Media Luring Money Away From Networks - New York Times.

I don't want to read too much into this, as economic factors run the gamut these days, so I don't think there's a great sucking sound of advertising money going from old media to interactive media just yet. I mean, yes, it is a massive shift in the money river. W ith the old media companies, very little of it is reinvested into the actual media products, and instead gets dumped into that gaping maw called "shareholder value" anyway. Whether those companies lose ad revenue or gain ad revenue, the way the money gets spread around in that world is highly suspect, from the standpoint of good business practices and making products of value.

But the late 1990s bubble makes me inclined to distrust the whims and crap shoot wisdom of these markets and ad buyers, and a war/deficit/evaporating middle class economy I think means all bets are off. But this is interesting to think about just the same.

Link: New Media Luring Money Away From Networks - New York Times.

June 28, 2006

Advertising

New Media Luring Money Away From Networks

By STUART ELLIOTT

ADVANCE advertising sales for the fall television season will probably be down for a second year in a row, reflecting the growing impact of new media outlets on broadcasters.

Estimates are that the five networks will sell $8.9 billion to $9 billion in commercial time ahead of the 2006-7 season, in what is known as the upfront market.

That compares with an estimated $9.1 billion sold in the upfront market last year, before the start of the 2005-6 season, and an estimated $9.3 billion in the upfront market two years ago, ahead of the 2004-5 season.

"I'm surprised it's two years in a row," said Wayne Friedman, West Coast editor at MediaPost Communications, which covers the media business through online (mediapost.com) and offline publications. "Most of the time, it's the networks holding all the cards."

Along with TV, newspapers and other traditional media are scrambling to adjust as marketers step up efforts to reach consumers in newer ways that include Web sites, podcasts and video-on-demand.

"The broadcast environment remains tough," Debra Schwartz, a media analyst for Credit Suisse, wrote in a report.

[...]

For decades, the broadcasters took in more during each upfront market than the one before, except when recessions dampened advertiser demand. That reflected a relentless focus on using mass media to reach consumers quickly.

Now, marketers' desire to explore alternatives, particularly digital media, is leading them to divert ad dollars from television or to add money to their budgets specifically for new media.

That significant change has not escaped the attention of the broadcast networks. They are offering marketers opportunities to buy commercial time or ad space in places away from the TV set.

NBC, for instance, has booked an estimated $50 million in digital revenue separate from the broadcast airwaves, said Randy Falco, president and chief operating officer at the NBC Universal Television Group in New York.

An additional $200 million in commercial time that NBC sold during the upfront market was attached to digital deals, Mr. Falco said.

[...]

During the upfront market, networks sell anywhere from 65 percent to 90 percent of the commercial time in the coming season. They hold back the remainder to sell during the season in what is called the scatter market.

"The big question now is really what the scatter market is going to be like," Mr. Friedman said. "Many media buyers are saying they want to wait for the scatter market to evaluate the new-media opportunities the networks are offering."

Given the challenging environment the broadcasters are facing, they may consider it a victory that the upfront market is not much lower than last year's.

"We believe that when the dust settles on the upfront market, total broadcast upfront-dollars results will be somewhere in the middle of bullish and bearish," William Drewry, another analyst for Credit Suisse, wrote in a report.

Or as Mr. Friedman put it: "Considering all the alternatives, that the upfront still has the power to do what it does is not so bad. If I were the networks, I'd take the money and run."

 

 

The technorati and the literati lock horns

What fun! John Updike takes on Kevin Kelly's NYTimes magazine piece "Scan This Book!" I've written about Kelly's piece here and here. I'll probably throw some of my own snarky commentary in below. Can't resist taking to jibjabs at his venerable Updikeyness.

Link: Washington Post: Explosive Words.

Explosive Words

At BookExpo America, Publishing's Digital Wave Crashes Against a Literary Pillar

By Bob Thompson
Washington Post Staff Writer
Monday, May 22, 2006; C01

[...]

The "Buzz Forum," as this exercise in competitive salesmanship is called, represents business as usual for BookExpo America -- the annual publishing convention that brought roughly 25,000 book people to Washington over the weekend.

If you listened, though, you could hear another kind of buzz at BEA this year. It was an angrier one, generated by a clash of cultures in an industry frazzled by technological change.

The clash is between what you might call the technorati and the literati. The technorati are thrilled at the way computers and the Internet are revolutionizing the world of books. The literati fear that, amid the revolutionary fervor, crucial institutions and core values will be guillotined.

[...]

When John Updike approached the lectern in the Convention Center ballroom Saturday morning, most of his bleary-eyed, coffee-swilling audience expected him to talk about his latest novel, "Terrorist." But Updike, the much-honored 74-year-old author of dozens of volumes of fiction, poetry, essays and criticism, said that would be "immodest." Instead, he praised the assembled booksellers as "the salt of the book world" and reminisced for a while about bookstores he had loved in his youth.

Then, without warning, he opened fire on the technorati.

"I read last Sunday, and maybe some of you did too, a quite long article by a man called Kevin Kelly," he began. He proposed to read a few paragraphs so that listeners who hadn't seen the article might "have a sense of your future."

The reference was to a piece called "Scan This Book!" in the previous week's New York Times Magazine. (The title echoes activist Abbie Hoffman's 1970 provocation, "Steal This Book.") In it, Kelly described -- in the messianic/hyperbolic style favored by Wired, the magazine with which he has long been associated -- the inexorable march toward an "Eden" in which the totality of human knowledge will be downloadable onto a single iPod-size device.

[...]

Reading further, Updike noted Kelly's assertion that "copy-protection schemes" are helpless to hold back the technological tide. "Schemes," he repeated sarcastically, drawing a laugh. As his audience well knew, the Association of American Publishers filed suit last year on behalf of five major publishers alleging that Google's library scanning project is a massive and flagrant violation of copyright law.

[Now, I'm no technological determinist, but even casual literary observers have to know how much this sounds like Plato in The Phaedrus bemoaning that TERRIBLE new technological invention that would be the death of intellects throughout Greece, causing their great Greek memories to simply atrophy (as they most certainly did) before the "marching dragon's teeth" that became syllabic letters on the page, alphabetic literacy. The invention? WRITING. Mr Updike, do you know whose side you're really on here?]

Updike went on at some length, heaping scorn on Kelly's notion that authors who no longer got paid for copies of their work could profit from it by selling "performances" or "access to the creator." ("Now as I read it, this is a pretty grisly scenario.")

Unlike the commingled, unedited, frequently inaccurate mass of "information" on the Web, he said, "books traditionally have edges." But "the book revolution, which from the Renaissance on taught men and women to cherish and cultivate their individuality, threatens to end in a sparkling pod of snippets.

[The part of this that is the most peculiar to me is the invoking of the Renaissance. I'd characterize that period as a time of explosive artistic and intellectual growth unleashed largely by social unrest due to structural and technological changes. Basically, the established "powers that be" found themselves overthrown, not so much be a deliberate and plotted revolution, but by something Marx would probably chalk up to historical determinism. I wouldn't call it that either, however.

Regardless of what actually swung the tipping point against the entrenched power arteries of the Church and Aristocracy, toward the rising merchant class and new ways of thinking, learning, and making, the end result was that the "fruit basket upset" of turning the known world's power structures upside down opened the way to new kinds of art and literature and science.

So I believe we are (or were) in a similar entrenched period like that now. Except that there is a similar revolution underway. It unsettles many people. Many are brittle and want to fight it. I'm no determinist. I don't see it as an inevitability. It looks to me more like a shift in the prevailing winds. The wind does not deterministically affect all who are buffeted the same way. Some resist, some bend, some spread their wings and fly off to wherever the wind will take them, for good or ill.

Normally, I'd hope the leading edge of our best artists and writers would understand such a shift, would be excited to be present at the birth of a new Renaissance. So it puzzles me that John Updike is sounding so much like those entrenched powers of the First and Second Estate who faced the Enlightenment and wondered why anyone would want a mass-printed book when clearly monk-copied manuscripts from the scriptoria are so much better?!]

"So, booksellers," he concluded, "defend your lonely forts. Keep your edges dry. Your edges are our edges. For some of us, books are intrinsic to our human identity."

[Updike wants edges to define the form, as if the form "book" exists in all time, on some Metaphysical Plane, a unit into perpetuity, and not something that was invented, evolved, and has had all different kinds of shifting edges throughout all of time, other than periods of oral cultures like Plato's, when even the existence of a book, a scroll, was a deeply-resisted threat to the intelligentsia of that time.]

[...]

Google has been coming to BEA for three years now, but it seemed a bit more visible this year. Maybe it was the fleet of subcompacts with "Google Book Search Mobile" painted on them that cruised the streets around the convention center offering free rides to people with BEA badges. Maybe it was the cookies bearing the phrase "Just a taste -- Google Book Search" thrust at conventioneers near the main entrance, or the lavish party the company threw Friday night at the old City Museum.

Turvey's business card gives his title as "Head, Google Book Search, Partnerships, Content." He has a PR problem and he knows it. His company's controversial library scanning program is not the same as Google Book Search, he said, but media coverage of the former has been such that most people confuse the two. [emphasis mine]

He explained: Book Search is a program supported by publishers, including the ones suing over the library program. It's a partnership with publishers in which Google digitizes their books with their permission, then refers any Google users who encounter the books while searching to the publishers' Web sites and to online vendors like Amazon.com, where the books can be purchased.

Not that he was disavowing the notion of library scanning. Libraries, he said, have been with us a long time and "all the books in them are free to end users."

Turvey liked Kelly's article. "I think he nailed it," he said. Told of Updike's criticism, he suggested that there's a bit of an "apples and oranges" thing going on.

"For novelists and trade publishers that publish books to be read sequentially," he said, the utility of searching within a book's content is harder to understand. But this kind of book is a minority, and a lot of publishers know that they can increase their sales by allowing searches that lead potential customers to texts they otherwise might never have found.

[that says it all in a nutshell, doesn't it?]

[...]

At the W.W. Norton booth, a crowd gathered around a familiar, furry face. It was Steve Wozniak, co-founder of Apple, founding member of the technorati and author of a fall book titled "iWoz: Computer Geek to Cult Icon: Getting to the Core of Apple's Inventor."

Had he by chance read the Kevin Kelly article on the future of the book?

He'd "glanced at it, at least," Wozniak said. "It's like everybody's scrambling to figure out how it falls out," he added, "and I don't know how it falls out.

"But I do know that heck, I wrote a book, and you know I did it for Norton -- and I wouldn't want to get copies somehow spread around the Internet.

[Generally I'd defer to the venerable Woz as well (all hail the Apple co-founder), but he's woefully behind the curve here. While he's stuck in unquestioned assumptions of a model of intellectual property that's been increasingly and restrictively defined by corporations, he seems to be unaware that quite a number of authors are releasing both print and electronic books at the same time, and not suffering on their sales (and I don't mean strict pay-per-electronic copy models either). Quite a few times already, I've read books online (like Dan Gillmor's "We the Media") and then went out and bought the HARDBACK edition for my library.

Looks to me like Woz is thinking electronic copies of books should be treated like software updates, with strict serial numbers just to open them or something bizarre like that (I'm torqued off these days because I lost a LEGAL copy of software I PAID FOR, because I can't find the &*(%$^%##@$#@ serial number! Like how many little slips of paper am I supposed to keep track of for little $49 pieces of software?!).]

'Who Needs Megahits?'

Score one for the literati. But the technorati had their innings as well.

[...]

At another BEA talk, Wired editor Chris Anderson presented a concept called "The Long Tail," about which he wrote a famous article in 2004 and which he has now turned into a book. It contains both hopeful and troubling news for publishers.

The basic concept is fairly simple, though it helps to have an illustration in front of you to understand it. Picture a graph showing sales numbers for, say, books. At the left, way up high, are the numbers for blockbusters like "The Da Vinci Code." The curve then dips sharply down and bends to the right, flattening out and stretching at some length (hence the name "long tail") as the sales per book get smaller and smaller.

It used to be that it simply wasn't worth keeping anything in print below a certain sales-per-year figure. But with the Internet's ability to reach niche markets, combined with online booksellers' infinite shelf space, this has changed.

The good news? More titles can be sold profitably over a longer time. The bad news -- for an industry highly focused on the care and feeding of bestsellers -- was summed up on a screen to Anderson's right as he began his talk.

"The Long Tail: Who Needs Megahits?" it read.

Steve Rubin, president of Doubleday -- who publishes Dan Brown and has reaped the benefit of Brown's astonishing sales -- scoffed at the notion that the long-tail phenomenon means blockbusters will become less important.

[Does saying so make it so? Cuz last time I checked, an argumentative point like this would need SUPPORT or else it should be roundly dismissed as a hasty generalization, a logical fallacy. If the president of Doubleday would like to send me the missing support for this argumentative point, I'd appreciate it, as perhaps a careless Washington Post editor cut that bit out for space or something (oh wait, in electronic texts, you don't HAVE to edit down for space. Silly me.)

Meanwhile, I have long lauded loudly Chris Anderson's Long Tail theories, which he works hard at supporting with, oh, things like numbers, and revenue figures, and so on.]

Jane Friedman, president and CEO of HarperCollins, said she embraces digital change. HarperCollins is "friends with Google," she said, because "we like the searchability of our books -- it's like taking the shrink wrap off books in bookstores to allow a consumer to see a page."

But Friedman doesn't want to give up those digital files to anyone. She was chair of the American Association of Publishers when it sued Google over the library scanning project. "I'm very bullish on everything digital," she said, but "w e are going to control the destiny of our digital files," no matter how much Google cites HarperCollins as an ally in its PR.

[...]

The programmer as journalist: a Q&A with Adrian Holovaty

Here's a great interview that ought to show TV people that "tech journalism" does NOT have to be constantly slotted as advertiser-sponsored segments of "Gee isn't this new gadget cool?!"

While I can massage databases fairly decently, I sure wish my dyslexia didn't prevent me from getting the kind of hard code programming chops Adrian Holovaty has.

Still, I do plan to teach myself to mess with some APIs one of these days... That would be a hell of a lot more fun than trying to get C++ into my impenetrable brain.

The bigger gulf here, is the vast cultural divide in newsrooms, with journalists who insist that anything involving technology HAS to be a black box that makes no sense. That's a social thing, technophobia, and has very little to do with the actual technology or processes themselves. I never met so many technophobic people as when I returned to the newsroom after many years away.

The world exploded with technology in the 1990s, and so many newsrooms stayed in a tight little anti-technology cocoon, not even up on the SOCIAL issues being raised in cyberspace, or how interfaces are actively re-shaping people's lives. Noooo, that's a job for that geeky "tech reporter" over in the corner, who goes to the big gee-whiz conventions every year, gets his geeky swag, and does a ton of gee-whiz stories mined straight from the gee-whiz product introduction press releases.

Link: USC Annenberg: Online Journalism Review: The programmer as journalist: a Q&A with Adrian Holovaty.

The programmer as journalist: a Q&A with Adrian Holovaty

Washingtonpost.com's Web tech guru answers questions about programming's role in news reporting and presentation.

By Robert Niles

Posted: 2006-06-05

[The universe of journalists who program is, well, pretty small. Which is why I welcome the chance to talk with Adrian Holovaty, an award-winning journalist/programmer whose work, both for WashingtonPost.com and for his own sites, expands this profession's capabilities. Adrian graciously agreed to answer a few of my questions via e-mail for OJR. -- Robert]

OJR: I think one can safely assume that everyone in the news business understands how one "does journalism" through writing or photography. But how does one "do journalism" through computer programming?

Holovaty: The way I see it, there are three basic tasks that journalists do:

1. Gathering information. This involves talking to sources, examining documents, taking photographs, etc. It's reporting.

2. Distilling information. This involves applying editorial judgment to decide what parts of the gathered information are important and relevant.

3. Presenting information. This involves shaping the distilled information into a format that is accessible to the readership. Some examples: writing style (inverted pyramid, etc.), photo color-correction, newspaper page design.

"Doing journalism through computer programming" is just a different way of accomplishing these goals. Namely, the technique favors automation wherever possible.

For example, it's possible to automate that first step, the gathering of information. That's how my chicagocrime.org site works. Each weekday, my computer program goes to the Chicago Police Department's website and gathers all crimes reported in Chicago. Similarly, the U.S. Congress votes database I helped put together at washingtonpost.com works the same way: Several times a day, an automated program checks several government websites for roll-call votes. If it finds any, it gathers the data and saves it into a database.

The second step, distilling information, can also be automated. Just as an editor can apply editorial judgment to decide which facts in a news story are most important, a programmer-journalist (we really do need a better name than that!) decides which *queries* should be made of data. For instance, on chicagocrime.org I decided it would be useful if site users could browse by crime type, ZIP code and city ward. On the votes database site, we decided it would be useful to browse a list of all the votes that happen late at night and a list of members of Congress who've missed the most votes. Once we made that decision of which information to display, it was just a matter of writing the programming code that automated it.

In the "journalism through computer programming" realm, the third step, presentation, is also automated. This is particularly complex, because in creating websites, it's necessary to account for all possible permutations of data. For example, on chicagocrime.org I had to account for missing data: How should the site display crimes whose data has changed? What should happen in the case where a crime's longitude/latitude coordinates aren't available? What should happen when a crime's time is listed as "Not available"?

Also, I should point out that the two example sites I've given are entirely automated, but often it's not possible to automate an entire project. In most cases, information gathering is done by humans rather than computers, and the computer programming comes into play in automating the distillation and display of the data.

A good example of this is washingtonpost.com's Faces of the Fallen site, which lists all known U.S. service members who have died in Iraq and Afghanistan. That information is collected by the Post's fantastic newsroom research team, not by automated scripts.

[...]

OJR: What should journalism schools be doing to prepare future journalists to work in a mash-up publishing universe?

Holovaty: J-schools need to get way more technical. A graduate of a journalism school should be a master of collecting data -- whether the old-fashioned way (by talking to humans) or through automated means.

The closest thing journalism schools currently have (to my knowledge) is computer-assisted reporting classes. Those classes should be required, in my opinion, and even better would be for j-schools to partner with computer-science departments so that journalism students would get some experience coding.

OJR: What types of information are newsrooms collecting right now, but most under-utilizing on their websites?

Holovaty: Much of the information that journalists collect, day to day, is structured. Information such as crime reports, obituaries and event listings always follow a certain pattern, which can be richly exploited by databases.

[...]

The very act of distilling information into a news story -- which is essentially a big blob of text -- removes any sort of structure. Information is exponentially more valuable if it's structured.

So I urge news companies to retain as much structure in their information as possible. These days, it's easier and cheaper than ever to set up a database server. Just do it.

A few specific examples? Any sorts of public records are structured, really. Crime reports are an obvious one. Fire-station reports, local school data, transportation data. There's a ton of this stuff.

[...]

OJR: What ought news organizations do to encourage tech innovation from their staffs?

Holovaty: Hire programmers! It all starts with the people, really. If you want innovation, hire people who are capable of it. Hire people who know what's possible.

And once you hire the programmers, give them an environment in which they can be creative. Treat them as bona fide members of the journalism team -- not as IT robots who just do what you tell them to do.

OJR: Do you think most news managers are afraid of technology? If so, how do  tech-savvy journalists overcome that?

Holovaty: I've met both types of managers -- those that are scared and those that aren't. (For the news managers who *are* afraid of technology, you can't blame 'em. It's only natural. Technology is completely changing their industry, whose rules haven't changed drastically in a long time.)

[...]

OJR: Other that the stuff you're working on, what technology you've looked at recently has grabbed your attention?

Holovaty: Generally I get excited by new APIs that various websites are launching. The Flickr APIs are a classic example: They let any programmer query the Flickr photo database via programs.

OJR: Journalism's always been a competitive business. But what technical initiatives should news organizations be cooperating on? What opportunities, if any, are the industry missing when companies don't work together?

Holovaty: I think news organizations should cooperate on removing mandatory Web-site registration walls, which are severely reader-unfriendly. It's embarrassing to be associated with an industry that treats its customers with such disdain.

[HEAR HEAR!!!!!!! But WILL anyone running those obtuse news media content management systems HEAR HEAR?!]

OJR: What online news projects have you seen recently, if any, that you thought were especially well done? (Not counting the Washington Post and other sites you've worked on....)

Holovaty: Off the top of my head --

* Just the other day I saw the great weather/hurricane tracking app at http://www.ibiseye.com.

* I'm consistently impressed by the stuff coming out of mySociety .

* Faneuil Media does some great work.

OJR: What tech sites do you check to keep up with the latest in mash-ups,  programming and Web development?

Holovaty: Every day I check delicious popular a couple of times. That's a good indicator of what people are talking about and the new things happening on the Web.

 

Google's Big Daddy, Sandbox, and Black Hole?

Link: Matt Cutts: Gadgets, Google, and SEO  Indexing timeline.

OK, so I came late to the party because I've been focusing in the past few years on content instead of technology. I usually manage to stay up enough on things to keep up, and noticing my pages falling out of Google indexes in these two previous posts, I started doing some research on the topic tonight. Found a ton of great information in the tech blogs, where I hadn't been lurking in quite a while.

Some of the links I've been reading are over in my Intriguing list on the side menu, and I may add more in here later, but here's what I gleaned in a nutshell:

Google launched a big new data hoover called "Big Daddy." Big Daddy went out slowly at the first of the year, but was fully launched by March 2006. Big Daddy plays a bit different with the Web than the old daddy did, I guess.

Many folks who live in this odd netherworld called "Search Engine Optimization" or SEO do nothing but obsess on making Google like them. I do understand this obsession, although I do not share it. I've always seen people who promise SEO stuff as rip-off artists who try to sell unnecessary services to gullible business owners wanting to hang out a web shingle. I usually counsel those folks to forego SEO prices and just do a few things consistently, like build a good site and have things of value to say, and keep their money for something worthwhile. If SEO is rocket science, then the tail is wagging the dog, and something is very wrong in the world.

However, the SEO-obsessors (like those who aspired to be the most popular kids in high school) did teach me something valuable tonight. I too have seen the Google sandbox in action, and all this time I've been puzzled by the odd behavior of Google. Sandbox is a good term. Now I know what to call it. It's a holding place where baby sites, new sites, are put until Google can decide if they are legit, have staying power, or are link farms and spam.

I've been building web sites since 1993, and the rules have changed so many times since then, I like to focus on the constants, the values that remain even as the game changes. So yeah, I played my own personal SEO game. I've never made a junk site, but I'm well-read enough to know there are a lot of things that make viable web sites that those chasing the momentary fad are unaware of. I try to focus on those things, put them up on sites, and observe what the search engines do. Like my dissertation. Like the Xenaverse itself. Like the blogosphere, as it shifted from tech blogs to security blogs to political blogs and citizen journalism efforts.

Which doesn't mean I'm not playing with interactive poetry, creative hyper-non-fiction, image-driven sites, or developing all manner of things behind my firewalls.

So with the latest sites I've launched, I noticed the Google Sandbox. Where Google picked up my sites right off in the past, suddenly new sites on the same domain are being virtually ignored, despite being timely, pulling comments from some big name bloggers, and containing substantive content with specific proper noun keywords. Because I've got better things to do than obsess on SEO, I just sort of said, "Hmph," and went on my way. Now it's starting to make sense. Not that I agree with the Sandbox, but at least I have a name for what I've been seeing.

But that still doesn't explain why back archive sites that have inbound links from all kinds of interesting neighborhoods, comments that enlighten, correct, and debate, and traffic, are disappearing into a Google Black Hole.

Matt Cutts is a pretty cool Google guy, and he really tries to explain all this stuff, but one of the commenters on this blog made the point so well, I just have to quote it here (it's most interesting to me that the commenter I like best comes from a background in online fandom cultures):

Link: Matt Cutts: Gadgets, Google, and SEO  Indexing timeline.

Nancy Said,

May 16, 2006 @ 5:16 pm

Matt,

First, I appreciate you maintaining this blog and responding to some of the comments.

I realize you can’t analyze every site, but from what I’ve seen at Webmaster World, the sites you have picked are not very representative of the sites which are having problems with the supplemental index and not being crawled. The sites you have picked are obvious offenders, but sites such as my own and many others have none of these issues. To us, it seems that building a site to the best of one’s ability isn’t good enough; unless you can play the Google game, you’re out of luck. For instance, the inbound link issue. There are only a couple active fansites related to mine (most are no longer updated, and my site is only a few months old). Therefore, I am stuck with a couple inbound links unless I try to contrive inbound links, which I have no desire to do. Of course, the related sites also naturally link back to me - I’m related to them too, after all! Now that’s bad? It’s quite a Catch 22.

I think one should hesitate to imply that all the websites with supplemental problems “deserve it” because they’re all doing something so terribly wrong that they no longer are recognized by the index. There are many sites which do not fit into this penalty schema that have lost pages - too many to blow off as abberations in an otherwise successful change.

[...]

But frankly I am more concerned with the fact that so many pages with good content are being ignored. If I were #105 for my keywords but could look at site:[my site] and see that my pages are indexed, I would be OK with that. At least they’re there, and people who are looking for content unique to my site can find it. However, now, according to Google, only 7 pages on my site are searchable for the average Google user - only seven pages of my site exist in Googleland. I can put exact phrases from supplementally indexed pages in the search engine and get no results returned. With almost nothing indexed, I feel like all my honest efforts are worthless to Google for some mysterious reason.

Yes, it’s your search engine and you may do what you like. However, I’m sure you understand that a search engine that throws out good content is not doing its job. Hopefully, you will not shrug off the numerous legitmate concerns because you were able to find in the vast array of e-mails you received some egregious offenders.

So as I was writing this post, I had another thought. Google is at a choke point of power, literally shaping the virtual landscape of cyberspace by holding the Keys to the Kingdom.

So while this is clearly a bad thing for those of us who need Google to be a reliable source of the Library of Everything, who want indiscriminate indexing and link parsing like Nancy above, who want an uncorrupted data set delivered in the results, IF Google's data set has become corrupted with the advent of Big Daddy, perhaps it means people will be driven to more reliable search engines and the stranglehold on power Google has will finally be broken.

On the other hand, I am starting to suspect that two parallel universes are in operation here: Google crawling and Google indexing. Google's crawling bots are still utter Hoovers, sucking up everything in their path. Reading the tech blogs convinced me of that. SEO folks are in the habit of checking their crawls, and they'd be screaming even more to high heaven if Google were not spidering the Known World in its Entirety, even gmail content etc.

But the Google indexes appear to be operating from a different data set, or different assumptions within the exclusive and secret Google data set. Is there one set of data that Google uses and parses and fine tunes into secret sauce for its own business purposes, just like the NSA would find "uses" for the data it sucks up indiscriminately as well?

And just like with China, is Google creating a separate and unequal parallel universe of indexing data that it allows the public, APIs, and other groups to see, a non-proprietary Google data set?

Who gets to search the REAL and uncorrupted Google data? It seems the rest of us get Big Daddy, but is he sloppy seconds? Those that aren't consigned to the Sandbox, or worse, the bottomless pit with much weeping and gnashing of teeth?

I'm just speculating, of course. Who can see into the secret heart of Google? I don't want Google to be evil more than the next person. And I'm not saying for certain it is.

This poster on the same site also made an interesting point, as I've been victimized by these "scraper sites" as well, but playing copyright police won't stop it (they could just shift the material to a mirror site as fast as blog comment spammers change IP addresses), and hiding the sites by erasing them from the Google index only makes it harder for legit sites to find and protest the plagiarism.

Think of it in terms of college term paper theft. You put your paper online because your teacher requires it. The paper gets hoovered up by some paper mill sites that start selling it to cheaters for a profit. You could try to stop them, but Google blacks them out of the universe, so you don't even know you're being ripped off.

1. Jeff Said,

   May 16, 2006 @ 6:36 pm

   Matt,

   I know google is not giving us webmasters a full picture with the link command. I did the link command on yahoo and msn and I noticed some scraper sites copied my content and added some links to a few of my websites. I have a feeling google is looking at these links as questionable. I am in the process of emailing these scraper sites webmasters and getting the links removed because I did not request to put them there and they violated copywrite by taking our content.

   Since google crawls better than msn and yahoo, will there be a way in the future for us webmasters to see these links? Honestly right now if a competitor wants to silently tank a websites rankings in google all they need to do is drop a bunch of bad links. Without google giving us webmasters the ability to see the links we may never even know this could happen.

 

I'll say it again: the problem is giving Google the Keys to the Kingdom in the first place, because it apparently can literally erase something from cyberspace and render it invisible. Add to that collusion with countries like China (I bet there's politicians in the US who drool over the power Google is allowing the Chinese government), the telcom's bid to turn the Internet into a giant toll road for their cronies, and can fascism be far behind?

Update on the Google Hiccup

In a previous post, I wrote about some of my back blog archives falling out of my custom Google Adsense domain search.

I also opened at Help Ticket at Typepad and submitted a copy of my post to the Tech Support people there. As usual, I did get a helpful response. I've always had good contact with people from Six Apart and I appreciate that. I just wanted to share with others what Typepad had to say about the issue I raised:

Hi Chris,

Thank you for your feedback on this. While we did make the change you're referring to to help with performance, this should not have an adverse affect on how Google indexes older posts on a weblog.

The Recent Posts listing typically links to posts that most users still have on their index page and this is the case for your weblog as well. You may want to try contacting Google about this issue directly as they may have some insight on this for you. We have not made any changes in TypePad that would cause this.

Please let us know if you have any other questions.

This is a bit of a relief, because I want to continue to trust Typepad to do right by me. And second, whenever one has a technical problem, it is good to eliminate the obvious technical glitches first, before jumping to those bigger philosophical issues.

Now I must contact Google with the same questions, probably through the inside of my Adsense account. However, in my response to the Help Ticket above, I thought of another possiblity that may explain why Google would do something so UN-Google-like.

Here's what I wrote:

I appreciate the clarification. I really did not want to believe that the indexing change on Typepad would make permalinks and the keyword indexes disappear from Google.

The thing is, Google is a Hoover. It hoovers up everything. It has designs on hoovering up all gmail email files. It wants a kitchen sink data set. Inclusive, not exclusive. We have to tell Google NOT to hoover up our sites, make our sites invisible to Google, for private family photo albums and such. So why would my back permalinks fall off the Google index if they are still out there for the Google bots to crawl? Could something be blocking Google's bots from crawling down some of Typepad's link paths?

The only other thing I could imagine is if Google thought my sites were a link farm. But my sites don't look anything like link farm sites. They have ideas, strung together, sense to be made, to the best of my ability.

Philosophically, I must admit I'm having this same problem on email, and I've written about it in a past CNN.com column: email spam filters think my emails are spam, and they are not. I'm not even forwarding silly jokes like my mother does. But I can't even send a colleague at work an article that may help him work on a story, because the email filter thinks sending a story is spam and not bona fide research. SINCE WHEN?

The only other thing I can think of is that I've seen link farm sites that are lifting whole posts off my site and republishing them on link farms. The only reason I find them is they show up on Technorati with my words on them. So if Google decided the content of MY POSTS are tainted by the link farms should be nuked them out of their Hoover system, then there have to be a ton of people being plagiarized by link farms who are being INCORRECTLY nuked out of Google.

I wonder if the link farm plagiarizers are themselves hoovering up older site archives, banking on the fact that the owners might not be paying attention, or that the sites might be inactive or abandoned.

An odd Google hiccup...

I've been suspecting this for a while, but I had no documentation. I found a hiccup either in Google's indexing, or in the way Typepad builds its blog content management system.

This may have started back when Typepad's servers got overwhelmed and they altered a few key features that are pretty essential for the way blog menus have traditionally been built. Recent Posts or Recently Commented posts that were more than 30 days old started falling off the side indexes.

Generally, blog software allows you to make a setting for how many items will be displayed in the Recent Posts or Recently Commented, by direct number, not by date backwards in time. It weakened the archive system and the blog ecosystem in Typepad considerably, because the point of Recently Commented is to show where the community action is in the back archived posts. It is also a key feature of self-organizing sites, just like the rating system is in Scoop. Instead of going forward in these features, Typepad took a major step backward.

I've published e-books in blog software, used it as a database, an RSS-based content management system. In short, my use of blog software is not determined by the popularity and now-focused element, that new-new-new blip and burble that dominates much of the blogosphere. I'm more interested in creating deeper discussions, creating posts that are not dependent on timeliness, but rather, become, by blog standards anyway, timeless. Many academic and independent research blogs have this goal, and this is a growing and influential sector of the blogosphere, what I tag as "Public Intellectuals."

I have several posts deep in the archives of some of my older blogs, essays and think-pieces, that get more traffic than the post of the moment.

That was easier to do in 2002 than now. Lots more bloggers now. Or maybe it was because I was in Radio Userland then. Funny thing, though. Those old 2002 and 2003 posts are still pulling hits, same as my 1998 dissertation. This fact alone, I believe, proves my point.

Ultimately, the current, pointer-focused now-now-now attitude of this moment in time in the blogosphere WILL FADE. In a blog-glut, as you sit scanning your news feed reader, you might crawl out of your squirrel wheel and seek out words that have value, value that you want to hold on to, value that isn't based on the fact that it was posted in the last five minutes. Analysis does that. Essays do that. Good writing does that. And people out in the blogosphere are making spaces to post these sorts of things, just as much as they are making spaces for the quick hits and pointers.

Mainstream media focuses exclusively on the quick hits and pointer blogs, perhaps because it is easier to marginalize the blogosphere that way, to dismiss it as insubstantial. Or perhaps because that is the extent of mainstream media's attention span, and it is only capable of looking in a mirror at something that, like for Narcissus, looks like itself. In love with its own reflection, it fails to recognize significant differences in form, presuming ultimate dominance as it does.

But we must not fall into that trap ourselves and let mainstream media and quick hit blogs define us. Juan Cole is not a quick hitter. Riverbend is not a quick hitter. Neither is Glenn Greenwald nor Jay Rosen.

For the first time since being a beta tester for Typepad in 2003, I'm beginning to have second thoughts about the software's apostasy from the blog faith. By downplaying, disempowering, and unhinging key navigation points, Typepad is playing the game, helping to marginalize bloggers as insignificant, rather than developing a tool that works consistently across all types of content, empowering bloggers in this media revolution.

(Yes, I do know that this primarily happened because Typepad grew so fast and its servers were simply overwhelmed with all that rampant indexing. Typepad doesn't have Google's massive server resources, yet. I'm still not happy about it.)

And here's where the Google hiccup comes in.

I have Google ads and customized Google search on all my Typepad blogs. It's just a simple domain search, which is unfortunate, but I'd rather have Google's power, even if I can't at this time limit the searches to subdirectories.

My steady numbers of archive hits have been falling off. Poor me, I thought I was just fading in the ever-expanding blogosphere, an insignificant microbe, according to Truth Laid Bear. It was odd, though. It followed a pattern. Peculiar.

But tonight, on my poetry site, Headpiece Filled With Straw, I wanted to add the Wallace Stevens poem below, "Emperor of Ice Cream." I knew I had another Wallace Stevens poem in my archive of that site, and I wanted to find it using my own customized Google search (which also is supposed to earn me revenue through Google Adsense). When I searched "Wallace Stevens," Google said it had NO MATCHES.

Hey, that usually only happens on Old Media newspaper or TV station sites, with their weird unsearchable obtuse content management systems, where you search for something you know is there, and it doesn't return any results. I can't tell you how many times that has happened to me on the New York Times site, for instance. Old Media CMS's are some of the most poorly designed things out there, but at least now they work with RSS.

Blogs were always superior to those sites in their simple RSS construction, categories, permalinks, etc. PERMALINKS were an important key to that POLITICAL MOVE to stop links from going dead. That's why blogs gained such an edge over Old Media in times of big world news events. It was simply easier to find links that worked to get the information you were looking for, rather than wasting time wrestling with those piss-poor Old Media site search engines and CMS design with no permalinks.

So I tried a few more keywords in my custom site Google search, words that I KNEW were used in topics deep in my archives across all my public, pinging sites. AGAIN, NO MATCHES.

WTF?!

Is it Typepad's fault that my archives have disappeared from the Google crawlers and bots, Google, which intends to INDEX EVERYTHING IN THE WORLD?

Or is Google overwhelmed and deliberately ditching past archives that were already scanned once, perhaps to make room for all the data coming in from its massive Library of Everything book-scanning project?

Is Google imposing subtle hierarchies where a keyword isn't a keyword isn't a keyword, even if it is fully linked and pinged, unless the poster is "important enough" by some other secret, commercial, Old Media measure? Google's strength is the egalitarian nature of its data parsing. If its parsing is corrupted, search data becomes useless, and results delivered based on Google's algorithms aren't accurate.

The hierarchy of Google's world is the link. I accept that hierarchy. I'm willing to live by that hierarchy, with links functioning as dollar votes, so to speak, the coin of this realm. I know link farms and such are working day and night, trying game Google, and I know they will fail, not only because of Google's vigilance, but because the sites AREN'T REAL. Yes, fake link farm blogs are bleeding into Technorati results, but I think that's Technorati's problem, unless the corruption crept into the info coming from Google APIs.

There is a difference between having something to say and just stacking up links. I thought Google's parsing was sophisticated enough for that sort of AI distinction, perhaps a bit of natural language processing, secret sauce, what-have-you.

I DON'T accept other hierarchies of influence corrupting that data set. I want to trust Google's code crunch, if not its people (it's not personal; humans are more fallible than algorithms).

So why are my own archives disappearing from my own custom Google Adsense search?

Has someone judged my words insignificant outside the realm of algorithm-link-crunching-formulas? Has the link crunching formula returned to aspects of the old 1990's HTML Metatag days, where whole pages are no longer scanned for the big crunch, but whole books are?

OR (and I suspect this is the more likely culprit), has the shift in Typepad's server processing made archives older than 30 days fall off the Google universe on subsequent crawls?

I may not be technically adept enough to understand exactly how that might happen, except to guess that in dynamic processing in the CMS (what Six Apart added to Movable Type with 3.0), PERMALINKS are no longer treated as INDIVIDUAL PAGE ARCHIVES. By creating those archive pages only on the fly when someone links to them, does that mean the PERMALINK is virtual? If archives are not "real" Permalinks, would that be why Google thinks they're no longer present on the Internet?

Thing is, my backlist, my archive, is part of my revenue model here. If I'm a Google Adsense user, and I'm directly invoking the LONG TAIL in creating an archive meant to have long-term value, and my archive is being removed from the Internet record, I'm screwed.

Or I have to look for a new blog content management system.

If this problem extends beyond Typepad's borders, then the development is even more ominous for the social phenomenon of the blogosphere. To de-rank older blog archives, to ERASE them, across the board, from the Google crawl of record, is a POLITICAL DECISION TO ATTACK THE LONG TERM INFLUENCE OF SUCH SITES.

In my dissertation and beyond, I started calling this "the politics of deep structure interfaces," where seemingly benign interface design issues have far-reaching political ramifications, just like the  overpasses designed by Robert Moses on roads to Long Island were deliberately made a few inches shorter than the clearance needed for NYC buses, effecting a kind of social segregation for lower income people to be able to travel to those areas.

If Long Tail blog archives disappear from Google, it is effectively the same as if our Blog City Buses are prevented from getting to Media Parity Long Island because of the "benign" technology of Robert Moses's overpass design.

Somebody's lowering the height of the overpasses. Is it Typepad, or Google?

Perhaps someone from Typepad or Google will see this and set me straight. I sure do hope they do it before this post vanishes into search engine oblivion.

The role William Shatner was born to play...

Link: Boston Legal, Tuesday 10/9c on ABC.

The ever-arch Candice Bergen, whom I've long admired as Murphy Brown and her own imitable self, the once-sexier James Spader, who still exudes the attitude of White Palace that Susan Sarandon brought out in him, and good old Oto from "Deep Space Nine." Watching "Boston Legal" is fun any way you look at it.

But William Shatner outshines them all. I couldn't have dreamed up such an amazing role for him. It's postmodern perfection, I think, because Denny Crane is a man who is living as a self-parody, and the role is played by a man who understands his own self-parody on so many levels, right down to the "Saturday Night Live" Sci-Fi Con skit, "Get a Life!" and his peculiar "musical" recordings. Camp camp, campy camp.

You might say Shatner is campier than Tim Curry in "Rocky Horror Picture Show." The subtext romance and deep affection between Denny Crane and Alan Shore just add to the multi-layered nuance of it.

I catch the show whenever I happen to be flipping through. The delicious speeches James Spader delivers put that on the map. Candice and Rene can shoot their well-timed snipe arrows as punctuation.

But William Shatner takes the cake. He's the reason I actually make a personal appointment to watch a TV show, instead of winding down my 500-movie Netflix queue. Jaw-dropping. Aplomb. Audacity. Delivered with the most perfect comic timing. This has got to be the best comedy on television (and I'm the sort of person who WON'T watch a sitcom; they are the most loathsome invention on the planet). But William Shatner in a wet suit? In the cloak room. Larger than life. Smaller than life. With his "little blue pill." Pathetic, sad, laughably lecherous, and proud, all at the same time. Wallace Stevens's roller of big cigars.

Denny Crane. Denny Crane. Denny Crane.

The Emperor of Ice-Cream

Call the roller of big cigars,
The muscular one, and bid him whip
In kitchen cups concupiscent curds.
Let the wenches dawdle in such dress
As they are used to wear, and let the boys
Bring flowers in last month's newspapers.
Let be be finale of seem.
The only emperor is the emperor of ice-cream.

Take from the dresser of deal,
Lacking the three glass knobs, that sheet
On which she embroidered fantails once
And spread it so as to cover her face.
If her horny feet protrude, they come
To show how cold she is, and dumb.
Let the lamp affix its beam.
The only emperor is the emperor of ice-cream.

    -- Wallace Stevens

Shades of the Chinese Cultural Revolution... online?

The sheer numbers of people in China make the prospect of this sort of mob/vigilante action even more frightening, I think. Shirley Jackson's "Lottery" mentality, writ large.

Link: Online Throngs Impose a Stern Morality in China - New York Times.

Online Throngs Impose a Stern Morality in China

By HOWARD W. FRENCH

Published: June 3, 2006

SHANGHAI, June 2 — It began with an impassioned, 5,000-word letter on one of the country's most popular Internet bulletin boards from a husband denouncing a college student he suspected of having an affair with his wife. Immediately, hundreds joined in the attack.

"Let's use our keyboard and mouse in our hands as weapons," one person wrote, "to chop off the heads of these adulterers, to pay for the sacrifice of the husband."

Within days, the hundreds had grown to thousands, and then tens of thousands, with total strangers forming teams that hunted down the student, hounded him out of his university and caused his family to barricade themselves inside their home.

It was just the latest example of a growing phenomenon the Chinese call Internet hunting, in which morality lessons are administered by online throngs and where anonymous Web users come together to investigate others and mete out punishment for offenses real and imagined.

[...]

While Internet wars can crop up anywhere, these cases have set off alarms in China, where this sort of crowd behavior has led to violence in the past. Many draw disturbing parallels to the Cultural Revolution, whose 40th anniversary is this year, when mobs of students taunted and beat their professors. Mass denunciations and show trials became the order of the day for a decade.

In recent years, the government has gradually tightened controls on the Internet, censoring popular search engines, like Google and Technorati; employing thousands of Web police officers; and requiring that customers at Internet cafes provide identification.

There has been recurrent talk by the government of registering all Internet users, and many worry that a wave of online threats and vigilantism could serve as a pretext to impose new limits on users.

[...]

In many countries, electronic bulletin boards hark back to the earliest days of the Internet, before Web browsers were common, and when text messages were posted in static fashion in stark black and white. In today's China, however, bulletin boards, or BBS's, have been colorfully updated and remain at the heart of the country's Internet culture.

"Our Web site is a platform, not a court," said Zeng Liu, a Webmaster for Tianya, which reports 40 million page visits daily and claims to be the world's largest BBS. "We cannot judge who is a good or bad person by some moral standard, but we have our own bottom line. If it's a personal attack on someone, we delete it, but it is very difficult, given that we have 10 million users."

Although concerned about online threats, advocates of free speech say that is no reason for the Chinese authorities to place further limits on the Internet.

"The Internet should be free, and I have always opposed the idea of registering users, because this is perhaps the only channel we have for free discussion," said Zhu Dake, a sociologist and cultural critic at Tongji University, in Shanghai. "On the other hand, the Internet is being distorted. This creates a very difficult dilemma for us."

[...]

What does Florida have against those sweet little League of Women Voters ladies?!

Link: Block the Vote - New York Times.

I've been looking forward to joining the League of Women Voters when I reach my golden years, because I just LOVE those ladies so much, and I think they are just about the coolest people I've ever met.

This is THE most politically-neutral organization on the face of the earth. They were perennial sponsors of political debates because of their famed neutrality (I think that ended in 2004, when the candidates issued ultimatums with too many rules to stack the debate in their favor. Again famously, the League would have none of that crap).

But my experience with the League is more personal. I like to hang out and talk with them at the polls, as they are THE MOST faithful and diligent poll-watchers out there, and so many of these ladies remember well how hard women had to fight to get the right to vote. I'm not saying all the League of Women Voters ladies are elderly, but I am saying the more elderly among them are a national treasure, and everyone should befriend and talk with them, because these are women who understand the importance of democracy above all else, and they've been quietly willing for 67+ years to put their time and energy where their mouths are.

Every time I sit and visit with ladies from the League, I fall in love with them all over again. They've seen politicians come and go, local campaign storms heat up and die, and through it all, they've got one issue: everybody's got a right to vote, and to be informed on what they are voting on.

You wouldn't think that's controversial, but there seem to be people who have a vested interest in making the basic premise of the League seem too radical. Where's MTV with it's "Rock the Vote" campaign? Can't they stand up here with the League?

I wish the League had some clout to fight gerrymandering, but that seems beyond their mission, and it could well be the most serious threat to our voting rights since poll taxes and Jim Crow.

But the ladies are too political for Florida. They want people to vote, to be able to understand their ballots, and Florida just can't have that. The state government of Florida is apparently TOO THREATENED by the League's work. I never would have thought I'd see the day.

Link: Block the Vote - New York Times.

New York Times Editorial

Block the Vote

Published: May 30, 2006

In a country that spends so much time extolling the glories of democracy, it's amazing how many elected officials go out of their way to discourage voting. States are adopting rules that make it hard, and financially perilous, for nonpartisan groups to register new voters. They have adopted new rules for maintaining voter rolls that are likely to throw off many eligible voters, and they are imposing unnecessarily tough ID requirements.

Florida recently reached a new low when it actually bullied the League of Women Voters into stopping its voter registration efforts in the state. The Legislature did this by adopting a law that seems intended to scare away anyone who wants to run a voter registration drive. Since registration drives are particularly important for bringing poor people, minority groups and less educated voters into the process, the law appears to be designed to keep such people from voting.

It imposes fines of $250 for every voter registration form that a group files more than 10 days after it is collected, and $5,000 for every form that is not submitted — even if it is because of events beyond anyone's control, like a hurricane. The Florida League of Women Voters, which is suing to block the new rules, has decided it cannot afford to keep registering new voters in the state as it has done for 67 years. If a volunteer lost just 16 forms in a flood, or handed in a stack of forms a day late, the group's entire annual budget could be put at risk.

[...]

Congress is considering a terrible voter ID requirement as part of the immigration reform bill. Senator Mitch McConnell, Republican of Kentucky, introduced an amendment to require all voters to present a federally mandated photo ID. Even people who have been voting for years would need to get a new ID to vote in 2008. Millions of people without drivers' licenses, including many elderly people and city residents, might fail to do so, and be ineligible to vote. The amendment has been blocked so far, but voting-rights advocates worry that it could reappear.

[...]