Well, I’ve now had a chance to catch my breath after a whirlwind few days in NYC at Gigaom’s Big Data event. So, this morning, I got to thinking, after doing our weekly Minnov8 podcast, that I should try to tell you about some of the high points for me at the event — some (but not all) of which are touched on in the 20 previous posts I did from there… 🙂

It was one awesome conference, for sure. One interesting factoid I learned, which came out right near the end of the event, was something that ties into the news of the day, especially here in Minnesota: a company called Dataminr actually alerted its customers about the Target Stores data breach 14 minutes ahead of the financial wires. Pretty impressive. (Here’s how Dataminr describes what it does: “Using powerful, proprietary algorithms, we transform the Twitter stream into actionable alerts, identifying the most relevant information in real-time for clients in Finance, News, and the Public Sector.”) Perhaps only Brian Krebs knew about the impact of the breach before they did?

How fast can humans and algorithms think? We learned that the limit of human decision-making is 1000 milliseconds — lower, and you’re in the algorithmic ecosystem. And that was the crux of a talk that was the best-of-the-best at this conference for me. It was delivered with mega-gusto by New Zealander (and decathlete) Sean Gourley, founder and CTO of Quid. “How humans and machines can team up to solve big problems.”

I know my data geek friends would especially have liked a panel that included data scientists from three of Silicon Valley’s top companies, LinkedIn, Airbnb, and Uber. Well, here it is, guy-geeks and girl-geeks, for your viewing pleasure: “The best websites aren’t just built, they’re calculated.”

No one would argue that Big Data is a complex topic. But what’s the biggest challenge around it in corporate America today? “I wish the problems were just technical,” said Boyd Davis, a VP in Intel’s datacenter software division. “The biggest challenge is getting senior execs to understand how Big Data impacts business models.”

The most moving talk at Structure Data was an interview of Ramona Pierson, cofounder of Declara, by Gigaom’s Mathew Ingram. She began with her incredible personal story — which, I kid you not, literally took our collective breath away! Here’s a retweet I did that captured the feeling as it happened:

“@BigDataWild +1 “@bmkatz: Wow – gobsmacked at @ramonapierson personal story and how it informs what she does now at @declara #gigaomlive.”

Her company is building an impressive platform that creates a CognitiveGraph™ to give every individual their own personalized learning path. (Investors include Peter Thiel and Data Collective.) Here’s the interview: “Why the future of social search is semantic.”

Some more tidbits from the conference for my data geek friends:

• Spark seems to be the next big thing out there in Big Data after Hadoop. (Hint: because it runs faster.) Gigaom says it’s increasingly replacing MapReduce in next-generation Big Data applications.  Check out DataBricks, the first company to certify applications for Spark.

• Met Life is hiring 1000 data professionals in Raleigh NC (Research Triangle Park). And Ford already has 300 people working in data analytics. (Both companies had on-stage presenters at the conference.)

• Dennis Crowley, CEO of Foursquare, was interviewed by Mathew Ingram. The big question at the end was, "Will we get to the point when we don't even have to check in — just walk into a place?" Crowley's (non) answer: "That's a good question: what will a Foursquare app look like that doesn't require a checkin?" Hmmm. Here’s the full interview: “How does Foursquare think about data, and what are the technical approaches?”

A couple of cool exhibitor companies I talked to:

• Brainspace – A new beta launched at SXSW creates meaningful connections between people and ideas, accelerating the flow of knowledge, discovery, and innovation. Tap into the collective intelligence of the web to discover and connect with the best content and people related to your interests. “Stop stockpiling bookmarks you’ll never see again.” Amen to that!

• SpaceCurve – “If you can’t get the data in, how can you get the answers out?” A real-time platform for organizations that want to turn big data into value as quickly as it is created. It concurrently ingests, fuses, and analyzes Internet of Things, Industrial Internet, spatial, sensor, weather, social media, historical and other data to deliver immediately actionable intelligence at petabyte scales.

How did my conference reporting experiment go?

You know, the one I talked about in my post before I left? ("Big Data Storms the Big Apple This Week") In that, I talked about how I was going to post on my blog, then flip those posts as I wrote them into my Flipboard mag (“Big Data in the Wild”), and then also tweet on my new Twitter account for the magazine (@BigDataWild). I’ve reported from a ton of conferences in my day, but I’d never tried anything as crazy, wacko as this! —- i.e., managing my blog, a magazine, and two Twitter accounts (I also tweeted a bit on my main account), and, um, cross-posting between them. It was all I could do to keep everything straight! But, hey, I like to push the envelope.

Adding to the merriment, I had to use three different devices to pull this off: my iPhone 5S to shoot pix, and my MacBook Air and iPad Air to write the blog posts and tweets. Oh, wait, I also was tweeting from the iPhone. I have an app for my blog platform on the iPad, so I could post from that, and I think I even posted from the same app on my iPhone, too. I had to juggle between devices for another reason: there were no power strips in the main conference room! (Nor the breakout rooms.) That was my only real bitch about this event: all the power strips they had were five or six on one table outside the main room… for 1000+ attendees! What? (Major fail, Gigaom.) But somehow I muddled through, because I had varying degrees of power on my three devices and managed to switch between them. Thank God battery life on the iPad is so good!

As I said above, I did 20 blog posts while there, and I counted about 65 tweets I did on my new account. I now have about 200 followers there, a really great group of data gurus, many of whom were at the event. My Flipboard magazine is rockin’ along, too — now with almost 4800 subscribers, more than 92,000 page flips, and 337 articles.

A couple things I learned: First, posting from my iPhone or iPad blog app, the post title shows up in the Flipboard mag as "Graeme Thickins posted an entry" — what? Bad app!!  Second, using iPhone photos in my blog posts looked fine there, but, even though I chose the largest JPG file size, they looked like hell on the Flipboard mag — they want really high-res files, it seems. So, just for kicks, I switched on the second day to using PNG screen grabs of speaker photos from the Gigaom event site — and, for some weird reason, even though they weren’t big files either, they looked fine in the mag. Go figure.

Speaking of photos…

Here’s my Flick set from the event.

Hell yes, video is data, according to Steve Russell, CEO and Founder, Prism Skylabs. He began by reminding us that there's a ton of video out there — something like 200 times the amount of video that's uploaded to YouTube every day is recorded out in the real world, just for surveillance in industries like retail, for example, every single day.

What are the barriers to more video being uploaded to the Internet and made use of? He cited four: access, computation, privacy, actionability (like mobile apps and data visualization). One application his company is particularly focused on is what in-store video can do for retailers. Gigaom says: "By leveraging customer data, online retailers have vastly improved the shopping experience. Until recently, brick-and-mortar retailers have been unable to similarly optimize at the same speed or scale. With new cloud technology, physical retailers now have the ability to mirror the strategies of their online counterparts, enabling them to critically understand store execution and optimize everything from merchandising to promotions."

Steve Russell is a Silicon Valley veteran with 15 years’ experience in building and managing video technology companies. His latest company is developing applications in marketing, branding, and visual merchandising. He believes privacy and insight can go hand-in-hand using "adaptive computer vision."

He closed with this point: "Sensors are now everywhere and are going to drive more video — it's an exciting future."

Ari Gesher, Engineering Ambassador, Palantir Technologies, just gave an animated talk, starting with the whole history of computing and operating systems (well, compressed a bit), and how it was all so custom (bespoke) — drawing an analogy to where big data is these days. He cited this classic tweet from last year:

@BigDataBorat In Data Science, 80% of time spent prepare data, 20% of time spent complain about need for prepare data.

Here's how Gigaom billed this talk: "The past decade has seen a proliferation of standalone-tools and technology for large scale data processing.  While powerful and transformational, the onus is still on the implementer to do most of the work – 80% of the time is spent on setting up the technology, leaving only a fraction to work on the actual problem at hand.  In the early days of computing, every piece of software had this problem – until operating systems heralded a revolution in building applications cheaply. What does the same innovation look like in the big data space?  How do we get beyond building prototype after prototype?  And what about the elephant that’s not even in the room yet – namely, good user Interface?"

Whew! To get Ari's take, watch the video of his talk on the livestream. And follow Ari on Twitter @alephbass.

This was an amazing session — the speaker's personal story made us all gasp! Wow, follow Ramona Pierson, cofounder and CEO of Declara on Twitter (@ramonapierson). Better than me trying to quickly recap it, watch the video here.

She talks about how machine learning can power platforms that make sure the right people and right content find each other without relying on who they know. Ramona spoke about some great work Declara is doing in Australia right now. (Shout out to my home country!)

Om Malik did a great on-stage interview (as he always does!) of Ben Medlock, CTO at SwiftKey, and Tim Tuttle, CEO, Expect Labs. Check out these two companies — I wish I could type fast enough to cover their insights! (Maybe their technologies can help?)

Here’s how Gigaom billed this session in the program: “Despite the hard work that goes into building systems for deep learning and other methods of understanding human language, users might never know they’re powering their favorite apps. And that’s kind of the point. Hear how voice and text messaging services are learning to predict what users will say to deliver a seamless experience.”

Tuttle especially had some fascinating comments. He said it will only be about six years until we smartphone users have a terabyte on our device! “That changes the game.” No lie — talk about the smartphone as brain…