UCD, UCD+P, ICO, and PureApp

As I mentioned on my last post, lots of changes happening. I have continued to change my role and have now landed as a cloud advisor. I am excited but enough about me.

What my new role has afforded me is to continue to explore and understand the various IBM cloud software solutions out there. It is an interesting landscape and is changing faster than ever. This post delves into IBM cloud provisioning and deployment solutions but leaves the on-premises/off-premises concept OFF the table. For the most part this discussion is concerned about the ability to automatically stand up an application environment regardless of its physical proximity to developer. So before I even dive into specifics, we can talk some general capabilities. The following list is not exhaustive and there are many more capabilities that the products I talk about are capable of. These are just the ones that I am interested in for this post. 

For the purpose of this post, the definition of the deployment stack, shown here, is the total of all the layers of technology needed to be created in order for a full application to execute.

stack2

Provisioning – As I said before, in this post I am interested only in the ability to automatically stand up new environments. This concept also assumes some type of pattern or infrastructure-as-code concept to enable the automation.

Deployment – For the sake of this post, deployment refers to the automated deployment of an application including its specific application configuration applied to a provisioned environment.

GO LIVE tasks – Those tasks that must occur for an application to GO LIVE that are above and beyond simply provisioning an environment and deploying the application. Tasks such as appropriate approvals, insure the app is properly monitored, backups are in place, property security and end point compliance policies, notifications are setup when things go wrong, etc. These important tasks are part of every operations teams set of responsibilities and have a large impact on production deployments.

Pattern – The ability to capture some part or all of the stack definition in a reusable artifact.  There are two pattern capabilities that we will talk about, vSYS (virtual system) patterns and HEAT.

Let’s now take a look at the IBM tools currently in this space. Big disclaimer here. There are many many additional solutions from the IBM portfolio many of which are highly customized and include a services component. These types of solutions are highly desirable in a hybrid cloud scenario where you need brokerage services to not only serve the line of business but also the ability to manage the provisioned environments across your hybrid landscape and the ability to manage cost and charge back across that same landscape. There are outsourced solutions from IBM that target unique cloud platforms.  For the purposes of our conversation here today, we assume we have a IaaS cloud solution that we want to take advantage of. In lieu of a big description of each product, I will simply list the capabilities that it provides (there are many more but these are the ones of interest for this post).

IBM Cloud Orchestrator – provisioning, patterns, deployment, go live tasks, BPM

UrbanCode Deploy – deployment

UrbanCode Deploy with Patterns – provisioning, patterns, deployment

PureApplication – provisioning, patterns, deployment

So that doesn’t help. Lots of overlap and obviously lots of details under the covers. Each product does a thing or two very well, so let’s look at the list again and I will expand on each capability highlighting its strength(s).

IBM Cloud Orchestrator

provisioning – ICO currently can provision nodes based on both types of patterns.  ICO can provision to many virtual environment both OpenStack based and not.
patterns – ICO currently supports 2 pattern technologies, HEAT and vSYS (virtual system). The vSYS patterns are the legacy pattern type. HEAT patterns are based on OpenStack and therefore require an OpenStack implementation. ICO has full editing capabilities for vSYS patterns, however ICO does not provide an editor for HEAT patterns.
deployment – while ICO doesn’t have a separate deployment capability, you are able to build into your vSYS patterns application components and Chef scripts that can ultimately deploy applications as part of the provisioning process. However this capability is not very scalable and is precisely why deployment tools like UrbanCode Deploy were created. HEAT patterns as defined by OpenStack definition do not contain deployment-specific capabilities (more details below).
GO LIVE tasks – ICO has a large list of pre-configured integrations to common operations tools to manage your go-live tasks.
BPM – ICO has a robust BPM engine allowing you to craft an detailed process that can be initiated through a self-serve portal. This allows you to string together your provisioning, deployment, and GO LIVE tasks into a single user-driven process.

UrbanCode Deploy

deployment – UCD’s strength is application deployment including environment inventory and dashboarding.

UrbanCode Deploy with Patterns

deployment – UCD+P includes UrbanCode Deploy and relies on it to deploy application components.
patterns – UCD+P is a full HEAT pattern visual/syntax editor. UCD+P also has encorporated HEAT engine extensions that allow the heat pattern and the engine to not only provision nodes but also execute Chef recipes from a Chef server and also deploy applications using UrbanCode Deploy. The resulting HEAT pattern is truly a full stack representation as represented by the picture above in a single pattern artifact.

PureApplication

provisioning – PureApplication software has the ability to provision nodes (I will leave it at that without going into all the different flavors. For the purpose of this discussion we are interested in PureApplication software that manages PureApplication systems.)
patterns – PureApp has numerous vSYS patterns that help to automate the installation and configuration of IBM middleware. The provisioning engine is robust and can orchestrate the configuration of multiple nodes that would make up an IBM middleware topology.
deployment – in the same sense as ICO, you can add application deployment information into your patterns but the same limitations apply.

So if we cherry pick the best capabilities out of each tool, we would grab the go-live tasks and BPM from ICO, the app deployment from UCD, the HEAT pattern editing and HEAT engine extensions from UCD+P, and the IBM middleware patterns from PureApp. There, we are done. Maybe at some point in the future this will be a single PID that you can buy. But until then, is it possible to string these together in some usable way?

Again, a big disclaimer here. I am not an expert here on this entire stack but hope to drive conversation.

OK, to begin let’s take advantage of the BPM capabilities of ICO and drive everything from here. The BPM capabilities allow us to construct a process that executes the provisioning tasks and the go-live tasks with the appropriate logic. You can envision a self-serve portal with a web page that asks for specific information for a full stack deployment. Things like the app, the type of topology to deploy the app on top of, the name of the new environment, etc. ICO would need the appropriate pattern to then use to provision. Here is where ICO can “grab” the HEAT pattern from UCD+P via an integration. It will then execute the provisioning via the OpenStack HEAT engine. This HEAT engine must have the UCD+P HEAT engine extensions applied to it. Since these extensions contain application deployment capabilities, the provisioning process will also utilize UrbanCode Deploy to deploy the application components to the appropriate provisioned nodes based on the HEAT pattern. The BPM process can also call the appropriate operations products to execute the go-live tasks either before or after the provisioning step in the process. Whew!!

So what is missing? It would be great to take advantage of the PureApp IBM middleware patterns which are bar none the best and most robust installation/configuration patterns available. A general solution here would be to include the appropriate Chef recipes as part of your HEAT pattern to get the middleware installed and configured and for non-IBM middleware solutions this is your best bet. But there is a lot of orchestration involved in setting up WebSphere clusters, for example, that are not easily accomplished using Chef or Puppet. PureApp has this capability in spades. The best practice in using PureApp today is to use UrbanCode Deploy to deploy the app to a PureApp provisioned environment as the patterns in PureApp are not HEAT-based. A lot has been invested in the PureApp patterns and what the future holds as far as other pattern technologies here is uncertain today. It is important to know that PureApp is the preferred solution when it comes to provisioning systems that will utilize IBM middleware.

This is the story so far and I am sure I have holes in my description above. I welcome any feedback and experiences.

Advertisements

Innovate Here We Come

I have been so busy lately and with Innovate next week I have had even less time to post. But if you are lucky enough to go to Innovate, here are a few items you should check out.

1.  Plugin development Birds-of-a-Feather – Come to my plugin development BoF on Monday and let’s talk plugins.  I am heading up the effort to get the word out on plugin development for UrbanCode Deploy.

2.  Check out the new announcements in the DevOps space.  Some pretty cool stuff coming out.

Look me up if you are there.

See you in Orlando.

UrbanCode Deploy and WebSphere – an Ever Growing Relationship

My experience with WebSphere Application Server started when IBM acquired Rational Software some 11(?) years ago.  WebSphere Studio Application Developer fell under the Rational umbrella and I was one that tried to tackle its capabilities.  And for the longest time dealing with WebSphere Application Server meant no more than “right-mouse-click -> Run on Server.”

I will admit that I am not a WAS admin.  It is a career unto itself and those that are good at it are worth keeping.  But over the years Rational continued to challenge me in the WebSphere arena with the release of Rational Automation Framework for WebSphere (RAFW).  RAFW codified 100s of common WebSphere “gestures” that could be rolled together into hard working jobs that did much of the heavy lifting in managing and maintaining WebSphere Application Servers.  It also had the ability to auto-discover WebSphere configurations and push those configurations to other servers that even may be running different versions of WAS.

Then along comes UrbanCode Deploy and we now have a deployment engine that could sure use some if not all of the capabilities of RAFW.  We are now beginning to see some being rolled into UrbanCode Deploy.  This post is the first in a series examining the Middleware Configuration for WebSphere Application Server (MCWAS) plugin for UrbanCode Deploy.

To begin, there are actually two plugins that come into play.  The MCWAS plugin has commands that are all about capturing, comparing, and using WebSphere configurations.  Some of the RAFW capabilities are captured in the MCWAS plugin (but not all. Stay tuned to future UCD releases for improvements).  A pdf file also exists in the plugin zip file that has details about how to accomplish what we are doing in this post.  The Application Deployment for WebSphere plugin contains 64 commands (as of version 71) that do the bulk of the work in updating and tweaking WebSphere configuration items.  The number is not even close to what RAFW supported, but it is getting there. By the way, you can always get up-to-date plugins from IBM here.  Both of these plugins need to be installed into your UCD server.

Next, let’s examine my test environment.  First, I have a UCD server and agent running on my laptop.  And like I said before, I am not a WAS admin, so I simply have WAS 8.5 running on my laptop with the Plants By WebSphere sample application installed and running.  I can follow directions so I got that app up and running pretty quickly.  Our use case is that I want to capture the configuration of that WAS instance and re-use it to deploy to another server.  We will use the MCWAS capabilities in UCD to do that.

1.  First we need to do some setup in UCD.  We need to setup our agent so that it can auto-discover WebSphere.  If WebSphere is installed in its default location, we don’t have to do anything.  If not, then we need to make sure UCD knows how to find WebSphere.  We add a wsadmin.path property to the agent with its value the path to the wsadmin.bat (or wsadmin.sh) file.  Also, we need a MCWAS component that will be used to hold the configuration.  I created a new component and based it on the Middleware Configuration for WebSphere template.  This component is the center of the deployment process.

2.  Next we create our resources.  You are free to create your own resource tree to represent whatever environment you have, but mine is simple.

one

3.  Next we add the agent to the resource.  If we wait for a few seconds and refresh, we see that we our agent has auto-discovered WebSphere and we get an additional resource below that agent that represents our WebSphere cell.  Magic!!!  We didn’t have to do anything to tell UCD, it just went out and found it for us.  Cool.

two

4.  The cell resource also has a bunch of role properties defined.  We need to fill those in to help UCD auto-discover what is underneath that cell.  If we look at the properties of the WebSphereCell resource, we see this list.  Actually in my case all I need to fill in is the WebSphere profile path.  This tells the UCD which profile I am interested in (there is a way to auto-discover multiple cells but we will keep it simple here).  We can leave the rest blank, especially the cell name.  We will have auto-discovery get that for us.

four

5.  Back to the resource tree, I can now tell UCD to auto-configure the cell.  In the Actions menu of the cell resource, select Auto Configure.

five

6.  A dialog pops up asking us to specify the auto-configure steps.  We don’t have any steps configured yet, so click on the “No auto configure steps for resource” link (yes it is a link.  UCD needs a bit of user interface improvement here).

six

7.  A new dialog is presented that let’s us pick an auto-configure step.  Lo and behold there is currently only 1 choice.  Select WebSphere Topology Discovery and click OK.

seven

8.  Back to the Auto Configure dialog, we now have 1 step defined.  Click Save.

eight

9.  If we wait a few seconds (in my case it was about 15.  I am sure in more complicated environments it might be a minute or so) and refresh our resources, more magic happens!!!  UCD has now interrogated our WAS cell and found the node name and server name below it.  You can also go back into the cell properties and see that the cell name property has been filled in.

nine

This structure should represent your true WAS configuration.  For those of you out there that are real WAS admins and have some real servers running with real configurations, try this and see if UCD gets it right.

Our next post in this series will now take advantage of what we have auto-discovered and use it in a deployment.  Stay tuned.

Dynamically creating a property

There are times in a deployment process that you may want to create a property dynamically and use it in a later process step.  Fortunately, Urbancode Deploy provides a mechanism to do this.

Let’s use the following scenario.  As part of your process, you want to create a tar file backup of a directory structure.  You wish to create a tar filename that includes a date stamp.  You therefore have to get the current date and time and format it to your liking. Here is how to do it.

First, create a Groovy process step.  There is a Groovy plugin that allows you to create Groovy scripts and have them executed as a process step.   We will use a simple script like below to grab the date and time and format it.

import java.text.SimpleDateFormat
def today = new Date()
def formattedDate = new SimpleDateFormat("ddMMyyyy, Ka").format(today)
outProps.put("date",formattedDate)

The Groovy step will look like this:

datetime

Notice the name of the Groovy step as we will use it later.

Also, notice the last line of the Groovy script. This exports the formattedDate as a property.  We can now use that property in subsequent steps.

To reference this property, use the following syntax:

${p:set-date/date}

“set-date” is the name of the step that set the property and “date” is the property name.  You can now use this variable to create your tar file, maybe like this:

tar cvf /tmp/${p:set-date/date}.tar .


					

Updating a plugin – the upgrade.xml file

When updating a plugin, the update process that the uDeploy server goes through is to scan every process that has been created on the server (component and generic) and upgrade any step that is provided by the plugin that you are upgrading.  The upgrade.xml file tells the uDeploy server how to update a step or a property.

update

Included at the bottom is the XSD file that defines the upgrade.xml file.

1.  Update a script – if there are no changes to the plugin.xml file (no API changes) but you simply want to update one of the scripts that are run that implement the logic of a step, no change is required to any of the plugin configuration files.  You can simply reload the plugin zip file and the scripts will be replaced.
Note:  Probably not a good practice to follow.  If you are updating your “official” plugin, update the revision number.

2. Propagating a step to the next version – any time you upgrade a plugin, you must include a <migrate-command> element for every step that you want to include in the new version.

<migrate to-version="7">
    <migrate-command name="Step Name"/>
</migrate>

3.  Deleting a step – if you want to delete a step, then simply don’t include a <migrate-command> element for that step.

4.  Adding a step – you don’t need to do anything in the upgrade.xml file for this new step as there is nothing to upgrade.  The process of applying the new plugin will add the new step.

5.  Rename a step – the <migrate-command> element provides a way to rename a step by including an “old” attribute.

<migrate to-version="7">
    <migrate-command name="New Step Name" old="Old Step Name"/>
</migrate>

6.  Adding a step property – there a few situations here that need to be taken into consideration.

a.  if you are adding a new property that does not have a default value, then obviously your script will have to take that into account.  In this situation, you don’t need to do anything special to the upgrade.xml file as long as you have a <migrate-command> element for the property’s step.  All existing process steps will get the new property but since there is no default value there is no need to migrate any existing properties.   If the new property is required, you will be asked to enter a value when the process is requested.

b.  If you are adding a new property that does have a default value, required or not, then you should add a <migrate-properties> element as follows.  This will insure that the properties of any existing use of the step will get migrated and the default value of the property will get assigned to the new property.  If you don’t include the <migrate-properties> element, the default value will not be applied to the property as no property migration will occur.  This will be a problem if your script assumes a default value.

<migrate to-version="7">
   <migrate-command name="Step Name">
      <migrate-properties>
         <migrate-property name="New Property Name" default="New Property Default Value" />
      </migrate-properties>
   </migrate-command>
</migrate>

7.  Deleting a step property – nothing needs to be done in the upgrade.xml if you are deleting a step property.  The process of upgrading the plugin will remove the property as long as you have a <migrate-command> element to migrate the step.

8.  Renaming a step property – a capability similar to renaming a step exists to rename a step property using the <migrate-properties> element.

<migrate to-version="7">
   <migrate-command name="Step Name">
      <migrate-properties>
         <migrate-property name="New Property Name" old="Old Property Name" />
      </migrate-properties>
   </migrate-command>
</migrate>

9.  Changing a step property’s default value – the <migrate-property> element is also used to change its default value.

<migrate to-version="7">
   <migrate-command name="Step Name">
      <migrate-properties>
         <migrate-property name="Property Name" default="New Default Value" />
      </migrate-properties>
   </migrate-command>
</migrate>

10.  Steps 9 and 10 can be combined if you are both renaming a property and re-defining its default value.

<migrate to-version="7">
   <migrate-command name="Step Name">
      <migrate-properties>
         <migrate-property name="New Property Name" old="Old Property Name" default="New Default Value" />
      </migrate-properties>
   </migrate-command>
</migrate>

update.xsd.docx

DevOps

My name is Darrell Schrag, and I work for IBM as a member of the Rational Emerging Technologies team.  As a member of this team, I am planning on sharing my thoughts and insights about DevOps in this blog.  I hope to provide my readers with some valuable information, as well as links to where the best information on this subject exist.  Keep watching this blog for more content in the coming weeks and months.