The current Web has moved past the point where personal blogging is a big focus into the realm of real-time (RT) social interaction. Most content is now generated and shared via RT social networks than on CMS-based systems. However, unlike a CMS’s focus on content, the RT social networks’ focus is on users and their Streams.

No longer is it sufficient to place content at the center of a system’s model. Instead, the RT Social Web demands that users have primacy, that their content preferences, that their Stream choices and channels, are just a part of their overall presence on the Web.

Instead of blogging-1.0 styled CMSs, the paradigm has shifted to what I call User Stream Manager (USM) systems. A USM places the user at the center of the system, not the user’s content.

One such USM is Bottlenose, a self-described tool ”that helps you engage with your streams more intelligently.” Bottlenose was founded by Nova Spivack (T, G+) , a prolific Venture Producer, and Dominiek ter Heide (T, G+), a very talented developer and CTO.

I’ve been privy to Bottlenose’s development for more than a year, watching it evolve into a more robust, well-envisioned Stream management tool — a USM. Although I have not used it on a daily or weekly basis, I’ve kept an eye trained on its progress, stepping back in every so often to test its waters.

I just did so again last week and all I can say is Bottlenose will become my default real-time Stream management system. I’ve grown tired of TweetDeck, HootSuite, and other social media dashboards. Whereas Bottlenose is still a nascent platform and therefore should not be compared apples to apples with the aforementioned dashboards, it’s future prospects are greater in my opinion.

Bottlenose’s foundation embraces Web 3.0; the company lives in what I call the Smartup space. The technology and the company are leading the way toward the era of the User Stream Manager. As consummate Smartup practitioners, they’re aiming at building an ecosystem around their core technology, including the ability for 3rd-party plugins to enhance the Bottlenose experience. Add to this the possibility of an app store, and you have a smartup that not only will monetize in a smart way, but share some of their financial success with other developers.

There is a lot of low-level, amazing, and cool technology that churns underneath Bottlenose’s surface, helping to sort out the flotsam and jetsam in your Stream from the treasures. Their proprietary tech helps extract the signal in your Stream from the noise in your channels. It learns what you like and dislike. It allows you to teach it how you prefer paddling your reach of the RT social river.

Currently, Bottlenose’s oars, rudders, and keel are just in the Twitter and Facebook Streams. As it continues to evolve, hopefully adding additional Streams into the mix (Google Plus, Quora), its utility will only increase and each user’s journey down their RT social Stream will become more meaningful.

If you have not yet tried Bottlenose, it’s time for you to jump on in. The water is fine and the flow is just right!

My Related Articles

1. Flocking To the Stream
2. Building the Social Web: the Layers of the Smartup Stack
3. How Many Streams Can You Kayak At Once?
4. The HyperWeb: it’s All About Connections
5. The first article in my five-part smartup series, Web 3.0: Powering Startups to Become Smartups

As a Social Web architect and an open source advocate I frequently write, think, and promote the notion and ideals of the Open and Social Web. My work in the areas of user-centric control (identity, privacy, data portability, and rights), federated Social Web models, future-of-money projects, and W3C standards groups has shaped my views presented here.

Soon after publishing my 4-part smartup series (almost a year ago), I began to think about key parts of what has become this article. I’ve had bits and pieces of this article jotted down in various places. Over the past three months, the ideas have coalesced into a cohesive framework. With a recent and lengthy process of helping a potential smartup try to find its foundation, I’ve been motivated to assemble, clarify, and share my views on what I call the layers of the smartup stack.

If you’ve carefully read my previous installments in my smartup series you will have discovered–in part–the message that is expressed here. This next installment in the series seeks to clearly present the framework of the smartup stack.

Smartups are Socially Transformative

Smartups look to operate beyond the stale disruptive technology mantra; the smartup vision is not simply a paradigm shift. Instead, smartups are best described as innovating at the intersection of technical, social, and cultural evolution. As such, well thought-out and executed smartups are revolutionary entities. They are socially transformative ecosystems.

The power that a corps of ecosystem partners can bring to your smartup’s success cannot be emphasized enough.

The layers of the smartup stack embrace the uniqueness of each smartup while recognizing the interconnectedness of the greater community. In this regard, smartup’s do not build software. Smartup’s create ecosystems. Like an ecological food web, your smartup can be viewed as an organism that is linked to and interdependent upon other organisms and system services. This mindset requires a broader view of your smartup’s role in society. A smartup’s ultimate goal is to create greater value than is captured.

No matter the grand vision of a given smartup, all smartups share the same DNA at their foundation. They are tech-reliant, Internet-obligate companies. If you need some convincing of this fact, please see my article, Putting the Tech Back into Social Web.

The Layers of the Smartup Stack

Whereas technology is at the center of the smartup stack, you will see in this article that smartups are greater than the sum of their technologies. As we explore each additional layer of the smartup stack, the focus shifts more and more to the outside. Greater emphasis is placed on the social, economic, and cultural frameworks. This will help integrate your vision into the real world. It will help bridge your metaspace creations with their meatspace participants.

Layers can connote horizontal levels upon which other material is placed or stacked. But in the view presented here, layers are rings that surround and bind to any lower and higher concentric-ring partners.

It is practically impossible to singularly architect and build each of the smartup layers without regard to their immediately contiguous layers. However, I will present each layer as if it were a well-defined and self-sufficient entity. The reality is that at all stages of building out your smartup stack, the interconnections to and interdependencies on other layers (inner and outer) must be carefully explored and considered. This is one reason (among many) why your smartup must have in-house technological expertise from the start.

As mentioned above, smartup’s do not build software, they create ecosystems. They recognize that there is great benefit to being linked to and interdependent upon others in a larger system. As many of the system services are outside of a smartup’s immediate control, a smartup must architect its ecosystem to work in symbiotic harmony with the greater Web community.

To that end, a smartup leverages and relies upon open source tools and open Web standards. As we will discuss in the section about the outermost smartup stack layer, smartups also give back to the Open Web movement in order to embrace an ecosystem approach.

The Inner most Layer: the Technology Platform

As mentioned above, technology is at the very core of every Internet-obligate smartup. The center of the smartup stack, then, is the technology platform. There are four pieces that comprise the technology platform. As previous smartup articles discuss two of these pieces in depth, I will not present much additional detail about them.

Each piece of the tech platform layer relies on Open Source tools and standards where ever possible. Although a smartup creates its own technology in aggregate, it leverages code libraries, tools, and standards to help make the process of building out their platform quick and efficient.

At this stage you will be proportioning your smartup’s time between product iteration (which means more coding), marketing your MVP, and customer development. Although you must find the proper balance between these three activities, the primary focus of this process is on building out your smartup’s foundational technology platform.

Here are the four pieces of the tech platform:

1. Schemaless Backend
2. Semantic Web / LOD Stack
3. Responsive Codebase
4. Modern Web Standards

Schemaless Backend

I’ve written an entire smartup article on the virtues of NOSQL versus SQL, so I will not repeat anything here except to say that some smartups may need to use an RDBMS as well for part of their overall data warehousing needs. The main point is that smartups are big-data players and as such they need to utilize the best technology for modeling, capturing, and managing that data. NOSQL databases are, by and large, the preferred choice.

Semantic Web / LOD Stack

I’ve also written an entire smartup article on the Web of Data. Suffice it to say that Semantic Web technologies, which some prefer to refer to as Linked Data technologies, enable the linking of data and allows for the serendipitous discovery of new connections with other datasets.

Smartups understand the value of and participate in the Web of Data. Smartups realize that data is the unit of exchange on the Web, not documents. Instead of Hyperlinks being the engine of exchange, it is Hyperdata. Data is the energy, the food, exchanged between participants in the Social Web. Semantic Web technologies facilitate the flow of information between “habitats”, between communities.

Responsive Codebase

This is the most generic-sounding piece in the tech platform layer. I will not delve too much into this piece of the tech platform layer as it deserves its own full-length article (perhaps the sixth installment in my smartup series).

There is not one preferred or recommended framework, language construct, or codebase that all smartups use. Different smartups use different code-creating tools. They pick those that they are most comfortable with and that serve their particular tech needs. However, there are some clear trends and, therefore, advice that can be offered to each smartup.

The broadest bit of advice is that Internet-coding technologies are evolving to catch up with and meet the needs of a more data-intensive world. Although a smartup CTO should use tools with which he or she feels comfortable, that does not mean that they can be complacent, that they should not spend time exploring and learning some of the newer options.

For instance, a smartup will choose an object-oriented coding style versus a procedural-coding style. But that does not mean that all smartups have to code in PHP, Python, or Ruby. There are some promising, new, convention-breaking language platforms that are the current rage in the Web dev world. One of these is NodeJS—a highly-scalable, high-concurrency, event-driven framework.

Another major smartup trend that epitomizes the Responsive Codebase mantra is moving as much of the processing away from the server side as possible (Web-1.0 and 2.0s thick-server approach). The focus is on creating what are referred to as fat- or thick-client applications. In other words, the browser or mobile device handles considerable more of the processing, relying a lot less on the server.

Another trend is the use of light-weight code libraries. When properly utilized, they allow a smartup to react more quickly and be nimble in their coding practices. As an example, one light-weight code library that my newest smartup uses is JSON-LD. It brilliantly facilitates cross-piece integration and as such can be categorized as falling into both pieces two and three in the tech platform layer.

A final smartup trend is preferred data formats. According to a recent report, 55 percent of all new APIs have support for JSON and a staggering 20 percent of new APIs support only JSON. This demonstrates the quickly-growing trend of utilizing JSON as a preferred data format (see slides 22 & 23). It also indicates that for data interchange, the reliance on XML is fading fast.

Modern Web Standards

Smartups support, adopt, and utilize Web standards. HTML5 and CSS3 are currently among the two most important Web standards. There are of course other standards, whose utility will vary among smartups, but these two should be utilized by all smartups.

The Second Layer: User-control and Economic Engine

The next layer of the smartup stack contains two sublayers that interconnect via their direct connections with the technology platform. Once again, this illustrates the importance of the technology platform as being a fundamental, foundational layer to all smartups.

These two sublayers are:

1. User-centric Rights & Control
2. Future-looking Economic Engine

User-centric Rights & Control

As I have written much about user-centric control over identity, privacy, usage rights, and data portability in the past, I will gloss over most of the details. If you’re interested in learning more about my viewpoints on these topics, simply search my website.

All smartups believe in and understand the importance of returning as much control over data as possible back to the users. They realize that it not only makes sense from the standpoint of being good social stewards, but also it makes good business sense as well.

With support from the smartup’s tech platform, users have significant power over each piece of data that they contribute, that they generate. Further support for users’ rights and control can be provided through novel, user-friendly legal contracts.

Future-looking Economic Engine

I’ve been interested in future-of-money projects and theories for sometime—particularly in how technology, specifically Internet tech, is leading to a revolution in how value is exchanged. This is why I am a charter member of the newly-announced W3C Web Payments Standards Community Group.

I believe that new micropayment frameworks and economic models are essential to not only the healthy growth and long-term viability of a truly Social Web, but also to our greater global society. The future of money and of economic self reliance rests in the emergent properties of the social-driven superorganism. Centrally-controlled currencies will eventually give rise to decentralized currencies and instead of tightly controlled and regulated markets, self-regulation via distributed command and control processes will become the norm.

Smartups are on the bleeding edge of this economic revolution. Smartups thus play an important part in helping to push new payment frameworks and economic models. They are intimately involved in evolving economic models and understanding the need for a universal payment mechanism for the Web—a mechanism that will facilitate the proliferation of alternative currencies, friction-less payments, crowdfunding, and general value exchange.

One payment framework that my smartup will be leveraging is PaySwarm. It is described as, “an open standard that enables web browsers and web devices to perform micropayments and copyright-aware, peer-to-peer digital media distribution.” I believe that PaySwarm can become one of the central pillars to any smartup’s future-looking economic engine.

The Third Layer: the Smartup Social Engine

This layer integrates with the innermost two layers of the smartup stack. The focus is more on the user interface (UI) and the user experience (UX).

When combined with the first two layers, this layer comprises what can best be described as the Smartup’s Social Engine. It is the internal platform that contains any intellectual property (IP). It is the fully-functioning application that provides the smartup’s unique product and service offering.

Although basic UI/UX considerations were made during the initial MVP testing, proving, and refinement phase, it was a Lean UI and Lean UX process. The Social Engine Layer is where a smartup spends considerable time perfecting its full-blown UI and UX. Issues such as tight integration with the the User-centric Rights & Control and Future-looking Economic Engine sublayers are addressed. Issues with proper social interaction flow are addressed.

At this level in the smartup stack, the focus begins to shift more toward the outside, toward the physical usage of the service, and not its technical underpinnings. Toward that end, pathways with which others can interact, integrate, and extend the smartups services are developed and engineered. These become the domain of the next layer.

The Fourth Layer: Outward-facing Connections

A key vision of the smartup model is to encourage and enable outside parties–3rd-party developers and other smartups–to contribute to and expand upon your smartup’s vision. To bring that goal to fruition, a smartup makes anywhere from one to three of the following sublayers available to outside parties. How many sublayers are offered depends on the type of smartup and its overall needs and vision.

The three possible sublayers of the fourth smartup stack layer are:

1. Smartup API Access
2. Smartup Open Source SDK
3. Smartup Standards Group

Smartup API Access

By and large, the vast majority of smartups publish a set of APIs that allow outside parties select access to their datasets. As discussed in the final layer section below, the use of APIs by outside parties can be a major catalyst in a smartup’s growth and success.

Smartup Open Source SDK

The Software Development Kit (SDK) sublayer is more accurately termed an Application Development Kit (ADK) sublayer. The notion behind this sublayer is that there are core codebase modules that may very well be primed for open sourcing. We will see below in the discussion of the final layer of the smartup stack why open sourcing some (or all) of your smartup’s codebase can significantly accelerate the development and evolution of your platform.

Smartup Standards Group

This sublayer is the least-frequently encountered sublayer in the smartup world. The purpose of this sublayer is to standardize key pieces of a smartup’s platform.

Above, in the second layer section, I briefly mentioned PaySwarm. That is a perfect example of a smartup opening up some of its work, exposing their efforts to the open standards process. The newly-announced W3C Web Payments Standards Community Group will focus its efforts around core working technology—mainly PaySwarm.

If your smartup has key technologies that could benefit the greater Social Web by becoming a part of an open standard, then you are encouraged to offer up as much of your technology as possible to make that happen.

The Final Layer: the Smartup Ecosystem

This last layer is perhaps the most difficult one to describe in a few paragraphs. The goal is to freely offer unrelated, 3rd-party smartups and developers tools that they can leverage to help build out, evolve, and expand upon your smartup’s original vision. At the same time, the access that you provide to your smartup’s datasets and technology allows them to create their own paths to success. This is what I term a smartup’s ecosystem.

The sublayer offerings in the fourth layer enable the creation of a motivated, loosely-organized team of volunteer coders that can and will help expand upon and evolve your technology—at least that part of your technology to which you allow 3rd-party access. The power that a corps of ecosystem partners can bring to your smartup’s success cannot be emphasized enough. This is why the ultimate goal of each smartup should be to create more value than is extracted from the ecosystem.

As an example, think of what happened when Automattic–the original makers and copyright holders of WordPress–open sourced the codebase. This led to the eventual, very-large ecosystem of WordPress theme shops, plugin developers, and consultants. It also allowed for Automattic to gain an exceptionally cheap (as in cost) and talented labor force which it continues to use to this day to help it build out the WordPress codebase. That is one of the powers of crowd-sourced software development via open source practices.

Twitter is another great example of the virtues of creating an ecosystem. In its early days, Twitter not only welcomed, but strongly supported and encouraged 3rd-party developers and startups to help expand their ecosystem. They published a rigorous set of APIs that allowed for developers to gain access to many of the datasets Twitter captured. In return, the 3rd-party developers were able to create new features and services that augmented the Twitter experience. This led to a number of successful companies that seemed to pop up over night, swirling around the core of Twitter.

Without these ecosystem partners, Twitter may very well not have succeeded. Unfortunately, as Twitter continues to struggle with figuring out how it can monetize its success, it has cracked down on their ecosystem partners in recent months, making many of them wonder if they can trust Twitter anymore. Twitter’s brilliant ecosystem strategy may be coming to a close.

Facebook was also an early creator of an ecosystem of developers. They offered limited API access, created their Open Graph ontology, and even open sourced a few of their key technologies. However, for the most part, Facebook required (and still does) that the apps of 3rd-party developers live within the siloed confines of the Facebook universe. Facebook is not a proponent of the Open Web, Open Standards, or user-centric control.

Of course, neither Automattic, Twitter, or Facebook are considered smartups. Although they do support–each to differing degrees–some level of open source involvement with their projects, they fail the smartup test with respect to many of the other smartup stack layers detailed above.

Conclusion

You don’t build a startup, you build a company. Whereas the word startup is an enticing concept, it is nothing more than a brand, it connotes nothing more than the early stages of a company. Each stage has its own specific needs and foci. Smartups are no different in this regard.

As mentioned above, many Internet-based startups do not transcend their technology but smartups have a vision beyond their technology. Even so, smartups recognize that–as Internet-obligate entities–they cannot divorce themselves from their technological foundations.

A smartup first builds a strong, foundational layer of technology upon which it then layers on additional functional components. Each of these components–also called sublayers–help push the smartup closer to its vision. To fully actualize its vision a smartup must create the conditions that enable, encourage, and support a system of ecosystem partners. In unison with its ecosystem partners, a smartup works toward providing services that empower users to pursue some of their passions and fulfill some of their goals.

Past Smartup Series Articles

Part 1: Web 3.0: Powering Startups to Become Smartups

Part 2: Web 3.0 Smartups: the Social Web and the Web of Data

Part 3: Web 3.0 Smartups: Moving Beyond the Relational Database

Part 4: Web 3.0 Smartups: the New Web Business Space

</Smartups Series Part 5 of 5>

How to Get Me Involved in Your Smartup

Interested in getting me involved in your smartup? Please see my 7-by-7 rules.

But there are two issues that I wish to address. First, only one of the six listed most influential topics on my Klout profile make sense. Second, without connecting my Facebook account, I’m significantly penalized.

These two issues have big ramifications for those of use trying to build the Social Web.

Luke, Use the Foci

About three or four months ago, I tweeted the same observation concerning the most influential topics list on my Klout profile. Someone from Klout promptly @replied to my tweet stating that with continued use, the algorithms would more accurately determine my most influential topics. But this list has remained unchanged.

I have not bothered to check this assertion, but I believe that sixty percent of my time tweeting and ninety percent of my time writing is spent on the topic of a user-centric, distributed Social Web. This is divided between generic categories of Internet freedoms (GPL, open source in general, net neutrality, data portability, privacy, identity, BuddyPress) and Web 3.0 (Semantic Web/Linked Data, smartups, WebID, FOAF, RDFa).

Yet my Klout profile does not reflect any of that. In fact, “Indiana” is one of my listed most influential topics when I would guess that fewer than 1% of my tweets have used that word.

Of course, I could be wrong. Klout can scour my backtweets a lot more efficiently and effectively than I can. It may be that fewer of my tweets are about those topics than I realize, that few people retweet any of my tweets that are about my primary foci. Perhaps I engage in conversational chatter on Twitter more often than I think.

Whatever the actual truth held within my Twitter data, I’d be very surprised if “Indiana, SEO, Design, Influence, Google” are my most popular topics. I rarely use any of those words in my Tweets or hashtags.

I do not spend my time within the dungeons of Mordor, I mean Facebook.

But it could quite possibly be that Klout is accurate, that those are my most influential topics. If so, that means that the vast majority of my time has been wasted, that I shouldn’t bother tweeting about #privacy, #identity, #opensource, #Web30, #BuddyPress, #SemanticWeb, #WebID…etcetera.

I’m of course being somewhat facetious. The reality is that I already know for which issues most people retweet and @mention me—and it is not the topics that Klout claims I’m most influential on. However, when the vast majority of my tweeting and blogging foci aren’t reflected in the self-proclaimed “Standard for Influence”, I have to wonder whether Klout’s algorithms need some tweaking, or whether I’ve got a clot in my Klout.

Interestingly, and on a side note, influence is one of my most influential topics! In my mind that evokes the image of a self-referential, infinite-looping, circular-referencing maelstrom.

An Audience with Sauron

Without connecting my Facebook profile to my Klout profile, I’m penalized forty percent—whatever that means, I don’t like the sound of it.

Although I do have a Facebook account, I do not use Facebook. I think I have four or so people whose friendship I’ve accepted and who are obviously waiting with baited breath to see what I’m going to say. To date, I have not made a single wall post. So, they will be waiting for a lot longer as I plan to never post any content in Facebook.

In fact, I think in 2010, I may have bothered to login to Facebook three times and that was only to checkout one or two aspects of its interface.

Why don’t I use Facebook? If you’ve been following my tweets and reading my articles posted to my site, you will already understand why. Facebook is antithetical to most of what I believe the Social Web is about. I’m professionally working on helping to bring the concept and infrastructure of the Social Web to fruition. Thus, I do not spend my time within the dungeons of Mordor, I mean Facebook.

Since I don’t use Facebook, some of you may ask why I use Twitter then, as it too is just another private data island. Simple. Although Twitter is a participant in the closed-silo wars, my posted content is accessible to anyone who wants to see it. In other words, they do not have to be logged in to access my Stream. With Facebook, you must not only have an account but also be logged in and be a friend of that person to see their Stream.

As I’m trying to promote the concepts of an open, distributed, user-centric Social Web, Twitter lets me get the message out to everyone—whether they follow me or not, whether they’re logged in or not, whether they have a Twitter account or not.

An Island of Misfit Toys

If Klout is to become the true measure, source, authority, standard for influence across the online world, then it needs to stop living exclusively within the social-networking private clubs. Why? Because influence occurs across the Web-based and mobile-based Internet. It isn’t restricted to closed social-media silos.

What about all of my online activity that occurs within my various blogs? Many people comment on my articles. What about the activity that occurs on forums, like the BuddyPress support forum where I’m a moderator (albeit a very inactive one lately)? What about on identica, or Diaspora, or the various niche BuddyPress sites that are beginning to pop up?

Conversations and influence flow beyond closed private data islands. Much activity occurs across the decentralized Web. In fact, for those of us fighting for the Social Web, we envision a day when most of the social activity will occur across a user-centric, decentralized, distributed architecture. The exclusive walled gardens of the Web will become a relic of the bygone, archaic Web-2.0 days.

Right now, a Klout score may best be summarized as a subjective measure of influence within a few select groups of data clubs. If a Klout score is to become the “standard measure of online influence” as proclaimed on their Twitter profile, then a user’s influence across the entire Social Web must be objectively calculated.

Shouldn’t my activity across the entire Internet be factored into my influence? Shouldn’t the activity of all of those who spend their time outside the Islands of Misfit Toys count for something? Of course.

Does this really matter? Yes.

As companies are beginning to use Klout scores to assess a potential candidate’s job application, to determine if a particular person has sufficient influence to be awarded a consulting contract, and for other inevitably unknown purposes, the accuracy and integrity of that score becomes paramount to all the Web’s citizens. Presently, too much importance is being placed on a Klout store, most likely to the unfair detriment of some.

As some company is going to win the influence-measurement war, if you care about your clout and how it is determined on Klout, provide them with feedback and the ways in which you think they could improve their service.

Update

December 22, 2010: Less than 12 hours after posting this article, my Klout Most Influential Topics list has changed since the first time I can remember. It’s still very inaccurate, but notice that the topic “Google” has been replaced with “Open Source”. A coincidence? I sure hope so. Otherwise I’m going to have to keep writing articles to get this all straightened out.

December 22, 2010 (Update 2): Joe Fernandez, the Cofounder and CEO of Klout, contacted me via Twitter this afternoon asking if I’d like to talk. We chatted for awhile about the issues I brought up in this article and some of the directions in which Klout will be heading in the near future. After talking with Joe–a very nice guy by the way–it is clear to me that they are thinking about the larger ramifications of their service and what it truly means to be the standard measure of online influence. Their task is very challenging indeed. It’s nice to know that Joe cares enough about their users to take the time to listen to feedback and communicate their dedication to continually evolving and improving their platform.

Related Outside Articles

The Read Write Web article that has brought some visitors to the post and triggered a small discussion, 4 Ways Klout Can Evolve.

An interesting though piece by Justin Goldsborough, Why Klout scares me; Hint: It’s not the tool itself

This article is my exhaustive study of what I thought was a simple little function—the do_action function. It details how WordPress action hooks really work. It is a long, detailed article. If you want to understanding the inner works of the do_action function, then it will be worth your time. Although this article only briefly mentions the apply_filters function, the lessons learned about the do_action function apply equal as well as the coding of these two functions is nearly identical.

This is not a beginners guide to WordPress action and filter hooks. To benefit from this article, you need to understand what hooks are, why you use them, and how to use them.

Hooks, Barbs, and Snags

Before we start, I need to define two terms. You may be wondering about the title of this article, “WordPress Hooks, Barbs, and Snags.” You should already know the definition of a WordPress hook. But what are barbs and snags?

These are two terms I created to classify the following:

Barbs: salient insights about the functioning of WordPress hooks that stick out and grab your attention. Understanding these barbs help you avoid snags. In this sense, a barb is a good thing.

Snags: WordPress hooks that do not execute as expected, not because the do_action or apply_filters functions are misbehaving, but rather because you do not understand how these functions actually work.

The Hook Loop or Where the Issue Gets Complex

When it comes to code, loops can be tricky constructs to properly implement. What seems like a simple, innocuous operation may actually be more complex than meets the eye. This is the case with the do loops in the do_action and apply_filters functions.

There are two key arrays that WordPress uses to determine the firing sequence of actions and filters— the $wp_actions and $wp_filter arrays. These arrays get built throughout the execution of each page load and are not fully populated until page load is complete and PHP stops execution, waiting for its next command.

The $wp_filter Array

The $wp_filter array is where all added action functions and filter functions are stored for future reference and processing by both the apply_filters function and the do_action function. This array is a very-large, multidimensional array nested to four levels. See the snapshot of the partial contents of this massive array.

Here is a snapshot of what is generated in the $wp_filter array just by navigating to the homepage in my particular setup. Your results will vary depending on which plugins you have activated, which theme you’re using, whether you’re running WordPress in single-site or multi-site mode, whether you’re running BuddyPress, whether a user is logged into your site, and of course which versions of WP / BP you currently have installed.

As you can see, there are a lot of action functions and filter functions listed, each of which result in the grabbing and storing of added actions and filters. Not all of these are necessarily triggered. It is up to the apply_filters and do_action functions to determine which of the referenced functions will be fired.

The $wp_action Array

The $wp_actions array holds all of the current actions invoked by do_action events in the files that are processed when a given page is loaded. Like the $wp_filter array, the contents of this array obviously depend on the page to which a user navigates.

Here is a snapshot of what is generated in the $wp_actions array just by navigating to the homepage in my particular setup.

But read on as knowing about these two arrays is only one aspect to fully understanding the underlying processes, in figuring out the entire puzzle.

Populating the Arrays, Firing the Hooks

Here’s another piece to this complex puzzle. We know that the array $wp_filter holds an array listing all of the currently active action functions and filter functions. But how is this array built?

It’s simple. All action and filter functions that are not enclosed within a function, a loop, or any type of conditional clause currently not triggered, are added sequentially to this array as a given file is executed after it’s loaded (via either an include, include_only, require, or require_only function call). Therefore, as soon as a file is loaded, execution passes to that file and all directly executable add_action, add_filter, remove_action, and remove_filter function calls are processed.

This means that PHP is furiously processing function calls from these four functions, resulting in data being added or removed from the $wp_filter array. When an add_action call is triggered, the parameters are sent to the add_action function in the WP plugin API. When an add_filter call is triggered, the parameters are sent to the add_filter function—and so on.

However, one additional function is called when an add_action call is processed. As soon as the add_action function gets called, it immediately passes on the responsibility to the add_filter function. This means that WordPress looks at action functions and filter functions as the same. This is another key insight and is why both action functions and filter functions are referenced within the same array— the $wp_filter array.

Once the add_filter function gets control, it adds the calling function to the growing list of actions and filters that can be triggered by a given do_action or apply_filters event. This process quickly results in the $wp_filter array growing into a massive, multidimensional array. Of course, action and filter functions can get removed from the array via calls from the remove_action and remove_filter functions.

The reality of code execution is of course more complex. The $wp_filter array is still being populated with action and filter functions when the first do_action events are triggered. This means that it is possible to have a situation where an action or filter that is hooked to a given do_action or apply_filters event will not get triggered if the file in which the hooked function is located has not been loaded before the hook is triggered.

This last point is exactly what was happening in BuddyPress when the bp_init action hook fired. This is the issue that was puzzling me and resulted in this exhaustive study and creation of my new WordPress Hook Sniffer plugin. You can read this BuddyPress Trac ticket to learn more.

A Detailed Analysis to Shed Some Light

Here is a breakdown and partial analysis of the $wp_actions array, the array that holds all of the current actions invoked by do_action events in the files that are processed when the homepage is loaded. We will use this array along with the $wp_filter array to figure out what is really going on.

The contents of this array obviously depends on the page to which a user navigates. This is just what happens when a non-logged-in user visits the homepage of the default BuddyPress theme in my WPMU setup. The only other plugin that is activated is my BuddyPress Privacy Component. For these tests, I am running WPMU 2.9.2 and BuddyPress 1.2.3.

NOTE: To fully understand what is going on, you need to make sure that you are looking at the completed arrays and not the partial arrays grabbed partway through page-load.

Let’s look at the order in which do_action events will be triggered. We will analyze what is happening in the first four elements of the $wp_actions array (index value of “0” through “3”). How is each action triggered and what are the results? This will start off as a detailed look into each hook’s actions, but will provide fewer details with each additional event as the basic processes become evident.

The First Four Triggered Action Events

[0] => muplugins_loaded

[3] => bp_setup_root_components


[0] => muplugins_loaded

When a user navigates to the homepage of a default BuddyPress install running on top of WPMU, the first action hook that is invoked is muplugins_loaded. This is directly triggered on line 547 in wp-settings.php. Any added actions that are tied to this event will next be triggered.

Looking at the $wp_filter array, you will see that there are no actions tied to this hook. This means that there is no subarray element with a key name of muplugins_loaded. In other words, you will not find any reference to that action event within the $wp_filter array. So, this action event does not result in any action functions getting fired.

NOTE: A triggered hook may or may not have actions tied to it. If no actions are tied to the hook, the do_action routine passes by that hook, checking the next invoked hook for added action functions to fire.

Before the next action event is triggered, something important happens.

Execution of the wp-settings.php file continues. On line 703 of wp-settings.php, all the active plugins are loaded—both blog specific plugins and site-wide plugins.

If you look at that line, $current_plugins can be modified by any added filters. That is what happens. On line 2348 of wpmu-functions.php, an add_filter call triggers function mu_filter_plugins_list. This filter modifies the contents of $current_plugins, adding any MU plugins that have been activated site wide. Processing is returned wp-settings.php, resulting in the main file of each activated plugin getting loaded.

There is an interesting item to notice within function mu_filter_plugins_list. On line 2345 of that function, the two arrays $active_plugins and $active_sitewide_plugins are merged and the resultant new array is sorted.

What does this mean? It means that when it comes to the issue of action and filter sequence firing, it does not matter which plugins you activate first. The order in which plugins are added to each of the serialized objects active_plugins and active_sitewide_plugins is immaterial. The sort function will alphabetically reorder the contents of the merged array.

NOTE: Each WPMU blog has it’s own listing of active plugins that are stored in a serialized object in the blog’s wp_x_options table. Look for the entry “active_plugins” within the option_name field of that table. The corresponding active plugins are in the option_value field of that record. All plugins that are to be activated site wide within WPMU are stored in a serialized object in the wp_sitemeta table. Look for the entry “active_sitewide_plugins” within the meta_key field of that table and the corresponding plugins that are to be activated site wide in the meta_value field of that record.

So, in our particular case with two site-wide plugins installed and activated (buddypress and bp-authz), the order in which they were activated is immaterial in determining which file is included (loaded) first by the conditional loop code starting on line 704 of wp-settings.php.

If you look in the wp_sitemeta table for the meta_key “active_sitewide_plugins”, you will see this:

a:2:{s:24:"buddypress/bp-loader.php";i:1270239897;s:28:"bp-authz/bp-authz-loader.php";i:1270396554;}

BuddyPress appears first because I activated it first—as is necessary. But, when the references to these two main plugin files are processed through the mu_filter_plugins_list function, they get sorted. This results in bp-authz-loader.php getting loaded before bp-loader.php.

This could be a big issue as all BP-dependent plugins rely on BuddyPress to be installed, activated, and loaded. Any BP-dependent plugin that comes alphabetically before “buddypress” will have their main file loaded before BuddyPress’ main file.

Barb 1: As a BuddyPress plugin developer, this is the first major insight, the first important barb that can prevent your code from hitting a snag with regards to action and filter execution.

Fortunately, there is an easy way to prevent this barb from snagging you and those that use your plugin. Make sure that BuddyPress is active before your plugin loads its core files.

Since that is exactly what I’ve done with my BuddyPress Privacy Component, there are no issues and execution quickly passes back to the loop and the next file in the $current_plugins array is loaded. This is the main BuddyPress file, bp-loader.php.

[1] => bp_core_loaded

Execution of bp-loader.php begins. On line 22 of bp-loader.php, the first custom BuddyPress action event is invoked:

do_action( 'bp_core_loaded' );

It is directly invoked, directly triggered. It is not enclosed within a function, a loop, or any type of conditional clause. This invoked event is what appears in the second element of the $wp_actions array.

Since there are currently no tied-in actions to this particular action event (as can be easily determined by looking at the $wp_filter array), the invocation of this action does not result in further actions being triggered.

The execution of bp-loader.php continues. Right after line 22, a series of conditional clauses include (load) all enabled BuddyPress modules—activity, blogs, forums, friends, groups, messages, xprofile.

As each of these files are included (loaded), any directly invoked (directly triggered) action events within each of them will also be invoked. Since there are no additional, directly invokable do_action calls within any of these files, execution is passed back to wp-settings.php.

[2] =>plugins_loaded

On line 735 in wp-settings.php, the do_action event for plugins_loaded is directly triggered.

do_action('plugins_loaded');

This is when things really start to accelerate. Up until this time, only a few lines of php code within BuddyPress have actually been run.

With the invocation of the plugins_loaded event, we now have our first action hook that has actions tied to it. Understanding what happens next leads us to the second major insight, the second important barb.

To understand what happens next, we need to inspect the elements within the plugins_loaded array of the $wp_filter array. I have extracted just that subarray element below:

plugins_loaded:
Array
(
[0] => Array
(

)


This array element is itself a multidimensional nested array. In fact, the first element (with an index value of “0”) has two action functions referenced.

An important point: each subarray of the $wp_filter array is a numeric array. Read that again. This means that the second level of the nested $wp_filter array contains numeric arrays.

You might be wondering how the index values for these numeric arrays are set. This is simple.

The key, the index value, of each 2nd-level array element is determined by the priority assigned to the event in the add_action and add_filter call. Thus, the priority set for a given action or filter function becomes the key of each element in a given hook’s multidimensional array. If a priority is not set for a given add_action or add_filter call, it is automatically defaulted to “10”. This means that any and all added actions or filters that do not specify a priority are automatically assigned to the proper action array element with an index value of “10”.

As an example, if the priority for a given added action is set to “16”, then that particular action function will be located in the array element for that action with an index value of “16”. Multiple action functions can be added under the same index value, creating an array of action functions associated with the same numeric key element. The order in which they appear is determined by the order in which they were executed.

Barb 2: This is the second major insight, the second important barb that can prevent your code from hitting a snag with regards to action and filter execution.

Now, the next piece of the puzzle is this. The order in which action functions are added to a given action array element is determined by the order in which they are encountered, the order in which their corresponding add_action or add_filter calls were fired. As we will see below, this is what makes determining the firing order of added actions and filters difficult.

Barb 3: This is the third major insight, the third important barb that can prevent your code from hitting a snag with regards to action and filter execution.

Barb 4: The higher the index key, the later the action or filter functions within that index-key grouping will get fired. Therefore, using the example above, if we set a priority of “16” for a given action function, it would get fired only after all the other action functions with a lower priority have been fired.

We have now reached the final piece of the puzzle!

Near the beginning of this article, the complexity of the do loops within the do_action and apply_filters functions was briefly mentioned. And it is within those loops that this story really finally gets resolved.

If you look at lines 166 (within the apply_filters function) and line 339 (within the do_action function) of the wp-includes/plugin.php file, you will find the code lines where the actual hooked functions are fired. Do you see the call_user_func_array() function call? That is a core PHP function. It is what will actually fire the next action or filter function in line, as determined by the data in the $wp_filter array.

The action of the call_user_func_array() function nested within the do loop is not as simple as it may seem. Why is this? Well, currently we are discussing the plugins_loaded action event and all of the tied into functions that get fired.

Look at the next action event to get triggered within the $wp_actions array. It is the action event bp_setup_root_components. But wait. Why is there another action event being triggered? We have not yet finished iterating through all of the added action functions to the plugins_loaded action event.

Once again, the reason is simple. Triggered action functions can themselves have action events. Inspecting the plugins_loaded subarray in the $wp_filter array, we see that the bp_setup_root_components action function is invoked by the plugins_loaded event via an added action call on line 2021 in bp-core.php. Within that action function there is an action event called bp_setup_root_components.

So, the sequential processing of all the added action functions with the plugins_loaded subarray of the $wp_filter function has come across a new action event to trigger. It now gets priority and further processing of the plugins_loaded subarray is put on hold until the bp_setup_root_components action event is finished processing its added action functions.

Barb 5: This can quickly result in a nested grouping of action events with the firing of their corresponding action functions. This is the final piece to the puzzle and the last and final barb that can prevent your code from hitting a snag with regards to action and filter execution.

As you are reading this, the juxtaposed actions of the do loop with the call_user_func_array() function more than likely seem obvious. It may seem this is not unexpected behavior or difficult to understand.

Whereas that is the case when all of the data is clearly laid out before you, the complex actions of these intertwined functions are not apparent when you are coding a plugin. There is no practical way to peek inside this action-event, action-function feedback loop.

This is were the true value of the WordPress Hook Sniffer plugin is revealed.

[3] => bp_setup_root_components

Here’s where we understand the last piece to this confusing puzzle.

Inspecting the $wp_filter array for this action event you will see that there are four action functions that are hooked to this action event: bp_activity_setup_root_component, bp_blogs_setup_root_component, bp_forums_setup_root_component, groups_setup_root_component.

These action functions are fired in that sequence. Since they do not have any action events which they trigger, control is passed back to the plugins_loaded event and the next action function in the plugins_loaded subarray is fired.

Applying the Lessons

When interpreting the results of the WoodPress Hook Sniffer plugin, you need to look at the output of the $wp_actions array and then search out the hooked actions to each of those items (if any) in the $wp_filter array. This will give you a general idea of what gets fired when. You can then experiment and see how changes to action priorities affect the firing sequence of a given added action.

You need to carefully think about the execution timing of each action or filter you add, looking at the corresponding triggering event for the hook or hooks to which it’s tied. Remember, you can hook a function to more than one hook which means that it is possible, if you are not careful, that some of them may fire whereas others may not. It’s all a matter of whether or not the reference to the action or filter function was successfully added to the $wp_filter array before the associated hook is triggered.

Looking at the Action Event Firing Sequence and the Filter Event Firing Sequence will help you precisely tune your added hooks and filters. Remember that execution my temporarily pass to another action or filter event. The output of these two WordPress Hook Sniffer arrays will show you when that happens.

Finally, there is no requirement that you place your add_action and add_filter calls right below the functions in which they are associated. Whereas doing so does facilitate code understanding, you could instead place them all at the top of the file (or the bottom) in which the associated functions are located. Remember, each add_action and add_filter function call that is directly executable is processed as soon as a file is loaded. So it does not matter where within the file those calls are located—with the possible exception of fine tuning the execution order of a particular function.

Final Notes on the $wp_filter and $wp_actions Arrays

The arrays $wp_filter and $wp_actions are repopulated from scratch each time a page is loaded. They do not contain a running accounting of all action and filter functions and all action and filter hooks. The contents of both of these arrays may vary depending on which page you navigate. The contents will depend on the files that are loaded (via include, include_only, require, require_only) when a given page you are on is displayed.

The reality is that, when visiting a given page, there will always be more hooks, action, and filter functions buried deep inside non-loaded files or within function blocks that are not directly hooked. The first is a result of the add_action, remove_action, add_filter, or remove_filter calls not getting fired until that file is loaded. At file load, any directly-executable do_action or apply_filters will be triggered as well. The second is the result of functions that have hooks that only get triggered if the file is loaded and the function is fired.

Although the WP Hook Sniffer will locate all action and filter hooks, and action and filter functions no mater the source–core files, custom themes, or 3rd-party plugins–what is captured depends on one simple rule—that the hooks and function calls are actually fired during page load.

As an example, WP Hook Sniffer cannot locate add_action calls that get fired on a different page than the currently loaded page. It can also not locate add_action calls that are buried inside a function that does not get triggered unless a special criteria is met. To sniff out those added actions, you have to navigate to that page so that those calls get fired or cause the special criteria to be met so that the function containing the buried action call is triggered.

Remember, the way that the WordPress Plugin API processes added and removed actions and filters is by storing (or deleting) a reference to them in the $wp_filter array. But that only happens for hooked functions that are actually encountered during page load. The $wp_filter array is wiped clean with each page load and rebuilt from scratch. The same holds true for the $wp_actions array and the action hooks it tracks.

Therefore, if you have action or filter functions that are supposed to be fired on a given page but you are not seeing that they’ve been fired, then that indicates that they are hooked to an action or filter event that finishes firing before the file in which the target action or filter functions are located even gets loaded. This is exactly what happened to me and caused me to write WP Hook Sniffer.

Summary of Barbs: the Key Insights

1. When it comes to the issue of action function and filter function sequence firing, it does not matter which plugins you activate first.

2. The key, the index value, of each 2nd-level array element within the $wp_filter array is determined by the priority assigned to the event in the add_action and add_filter call.

3. The order in which action functions and filter functions are added to a given action or filter subarray element in the $wp_filter array is determined by the order in which they are encountered during code execution.

4. The higher the index key, the later the action or filter functions within that index-key grouping will get fired.

5. Action hooks (events) –> trigger action functions –> which can have additional action hooks –> which can trigger additional action functions –> until control is passed back to the the originally-triggering action hook.

This is the action-event, action-function feedback loop caused by the juxtaposed actions of the do loop with the call_user_func_array() function.

During the chat, Andy Peatling (lead BP developer and Automattic employee), presented an idea about breaking up development work into component teams:

I’d like to start breaking BP down into chunks, and find people that are really interested in specific features…so for example if you really love the activity stream functionality you could focus specifically on that, and stick to patching just this area…so the long term goal is to get teams on components and have that transition into core commit teams.

This has merit. As the complexity of the BuddyPress codebase expands, it will be increasingly difficult for a one- or two-person team to do all the core lifting. BuddyPress is a complex suite of plugins. It is a social-network-creating ecosystem full of hundreds of functions and classes. Breaking the workload into project teams is a sensible approach.

More Hands to Watch

But, this notion of modularizing BuddyPress core development made me realize that a single guy–that would be me–cannot effectively continue to maintain and update the BuddyPress Privacy Component. It is impractical.

As you already may know from my very successful fundraising drive for my BuddyPress Privacy Component, keeping the BP Privacy plugin up to snuff with each new release of BP is quite challenging. In effect, I have to be an expert on all the BuddyPress components.

If there will be project teams managing the future development of the BuddyPress suite of components, that means two things: 1) there will be too much information created by too many hands on which I need to stay caught up; 2.) there’s an opportunity to streamline privacy filtering.

Enter the BuddyPress Privacy API

Privacy should be a core feature of any social network. BuddyPress is no exception to this rule. So, I’m now thinking that the best approach to privacy in BuddyPress is via a Privacy Layer that provides a basic Privacy API which any and all components can access.

I’m now investigating how practical and possible it will be to create a Privacy Layer using my current privacy codebase. If it is something that can successfully be created without a significant amount of additional work, I will switch my efforts toward creating the BP Privacy Layer.

This means, that going forward, it will be up to each BuddyPress component development team to utilize the Privacy Layer (if they choose to), to tie their component into the Privacy API, and provide privacy filtering. That way, providing privacy will become a team effort and not just one guy playing catch up, running behind Andy, jjj, and all the component-team members who are furiously evolving the BuddyPress codebase.

Do you think a BuddyPress Privacy Layer is the best way to ensure that privacy becomes a core element of each component? Do you think a BuddyPress Privacy API is a desirable feature?