Grid Computing Looking Forward

With the enormous flexibility and reliability afforded by computing grids, it may seem surprising that they not more pervasive today. The primary explanation is that grids exist in the context of a large ecosystem. It is not possible to go to a store and purchase a grid. Roadblocks to wider adoption are both technical and business-oriented in nature. From a technical perspective, it is safe to assume that applications not designed in multiprocessor environments are by default uni-processor applications. They can be executed on a multiprocessor node, but they will not use more than one processor, even if more are available, and hence the total run time won’t be shorter.
From a cost perspective, it might be attractive to share resources across organizations, including different companies, even in different countries. Doing so implies additional overhead to ensure data integrity, security, and resource billing. The technology to support these functions is still evolving. The lack of precedents makes potential users squeamish about trusting their code and data to be executed by someone else in a shared resource environment represented by a grid. Therefore, few grids today cross company boundaries. The largest user communities for grids today belong to government and academic research. This challenge translates directly into opportunity for those solution providers and system integrators that can overcome them. As the ecosystem of solutions for grid computing
continues to evolve, adoption is likely to increase by private companies that seek to harness the power and cost advantages of grid computing.

My perspective is that the real business value in grid computing has not even been defined, let alone achieved. The "commercial grid" -- as opposed to the scientific/technical uses of today's High Performance Computing grids -- is where the money will be made. The white paper in this blog defines a rather obvious architecture of nodes, clusters, and grids. The billion-dollar market questions are:
1) How will enterprise software tools be grid-enabled so any knowledge worker can begin to ask the questions no one can get answers to today? Look to the data warehousing market for a historical analog.
2) Can hardware vendors like IBM and Sun convince customers to rent grid capacity, ala electricity or will enterprises (continue to) build-their-own from excess capacity?
3) How can we harness the existing grid infrastructure available at every enterprise today: desktop PCs? Or will grids have to be built out on one capital asset requisition at a time over decades?
4) Will commercial grids lead or follow the trends towards virtualization? In a virtual computing world, the application becomes much simpler but the infrastructure has to be much smarter.
5) What happens to security in a node-cluster-grid world? The infrastructure of a grid world will make virus and trojan propagation possible at lightening speeds -- and it will mostly occur behind the enterprise firewall.
6) Are there synergies outside the grid that might make economic and societal sense? These might include electricity load shifting to do seriuos grid computing at night when electrical generation capacity is available at lower unit costs of production. Another would be geographical load-shifting work from the U.S. to Europe when that continent is sleeping and network and system traffic are lower.
7) Commercial grid computing will need to borrow a lot from existing high availability technology. Some grid applications will be mission critical in a 24/7 world. Others will only need checkpointing and restart at a checkpoint on failure.

Lots to ponder as we think about the computing world in 2015.

-- Peter S. Kastner