DataStage

Calling Python code from IBM InfoSphere Streams (DataStage) Part 1

The Python programming language is a popular choice among enterprise developers to quickly put together working solutions. Many companies adopt Python to build IT assets for regular use. IBM InfoSphere® Streams is a middleware product designed for implementing logic directly in C++ and Java™ technology. Read on to learn how to call Python code directly from IBM InfoSphere Streams applications.

If you want to Gain In-depth Knowledge on DataStage, please go through this link DataStage Online Training

Calling Python code from IBM InfoSphere Streams (DataStage) Part 1📷
📷

Overview

IBM InfoSphere Streams is high-performance real-time event processing middleware. Its unique strength lies in its ability to ingest structured and unstructured data from a variety of data sources for performing real-time analytics. It does this through a combination of an easy-to-use application development language called Streams Processing Language (SPL) and a distributed runtime platform. This middleware also provides a flexible application development framework to integrate code written in C++ and Java into Streams applications. In addition to C++ and Java, many developers who build real-world IT assets also use dynamic programming languages. With its strength in system integration capabilities, Python is a viable option for many companies to quickly build solutions. For those with existing assets written in Python, there is a way to integrate Python code inside Streams applications. This article explains the details about doing that through a simple Streams application example.

This article assumes familiarity with InfoSphere Streams and its SPL programming model. Working knowledge in C++ and Python is also needed to understand the programming techniques. For indepth details about InfoSphere Streams and Python.

InfoSphere Streams is a key component in IBM’s big data platform strategy. Many of IBM’s current and prospective customers with Python assets and skills can take advantage by mixing it with InfoSphere Streams. This article is targeted at readers whose technical focus is big data applications, including application designers, developers, and architects.

Example scenario

In order to explain the nitty-gritty technical details involved in calling Python code from a Streams application, we will stick to a simple example. This scenario involves reading the names of a few web addresses from an input CSV file and calling a simple user-written Python function that will

return the following details as its result. We will then write the result for each web address into a separate output CSV file:

  • Primary hostname of the URL
  • List of alternate hostnames for the URL
  • List of IP addresses for the URL
  • Company name specified in the URL string

Prerequisites

Code snippets used below explain the implementation details for the scenario explained above. This example code can also be downloaded so you can run it on your own IBM InfoSphere Streams installation. The example code was tested in the following environment:

  • RedHat Enterprise Linux 6.1 or above (or an equivalent CentOS version)
  • gcc version 4.4.5 20110214 (Red Hat 4.4.5–6) (GCC)
  • Python 2.6.6 (r266:84292, Apr 11 2011, 15:50:32, shipped with RHEL6)
  • /usr/lib/libpython2.6.so
  • /usr/include/python2.6 directory with Python.h and other include files
  • IBM InfoSphere Streams 3.x configured with a working Streams instance

The same techniques could work in slightly different environments (e.g., RHEL 5.8 and Streams 2.0.0.4) with some tweaks to the code or environment setup

High-level application components

In our simple example scenario, there are three major components. Each component is independent enough to be in its own project because of the natural separation by the programming language used in each of them:

  • UrlToIpAddress Python script
  • StreamsToPythonLib C++ project
  • streams-to-python SPL project

UrlToIpAddress is a Python script with simple logic that uses Python APIs to get IP address and hostname information for a given web address. This script can be tested independently using the Python interpreter. This tiny script plays a major part in this article in demonstrating how to call functions in a Python script from a Streams application.

StreamsToPythonLib is a C++ project. Inside of it, source code for the SPL native function logic is included. Primarily, source code here uses the Python/C API to embed Python code during the execution of C++ code. Embedding Python in C++ code is well described as part of the Python documentation. This project contains a Wrapper include (.h) file, which is an important one and this file provides an entry point for a Streams SPL application to call any C++ class method. All the C ++ logic in this project will be compiled into a shared object library (.so) file and made available to the SPL application.

Take your career to new heights of success with a DataStage Training

streams-to-python is a Streams SPL project. Inside of it, we have a basic SPL flow graph to make a call chain (SPLC++Python). This SPL code reads URLs from an input file in the

data directory, calls the C++ native function to execute the Python code, receives the results, and writes it to an output file in the data directory. Inside the SPL project directory, a native function model XML file outlines the meta information needed to directly call a C++ class method from SPL. This detail covers the C++ wrapper include file name, C++ namespace containing the wrapper functions, C++ wrapper function prototype expressed using SPL syntax/types, name of the shared object library created from the C++ project, location of the shared object library, location of the wrapper include file, etc.

In the following sections, we will dive deep into each of these three application components and explain the Python, C++, and SPL code in a detailed manner.

Python logic

Listing 1 shows the Python code. This is the business logic we want to call from Streams.

Listing 1. UrlToIpAddress.py

import re, sys, socket def getCompanyNameFromUrl(url): # Do a regex match to get just the company/business part in the URL. # Example: In “www.ibm.com", it will return “ibm”. escapedUrl = re.escape(url) m = re.match(r’www\.(.*)\..{3}’, url) x = m.group(1) return (x) def getIpAddressFromUrl(url): # The following python API will return a triple # (hostname, aliaslist, ipaddrlist) # hostname is the primary host name for the given URL # aliaslist is a (possibly empty) list of alternative host names for the same URL # ipaddrlist is a list of IPv4 addresses for the same interface on the same host # # aliaslist and ipaddrlist may have multiple values separated by # comma. We will remove such comma characters in those two lists. # Then, return back to the caller with the three comma separated # fields inside a string. This can be done using the Python # list comprehension. return(“,”.join([str(i).replace(“,”, “”) for i in socket.gethostbyname_ex(url)])) if ((__name__ == “__main__”) and (len(sys.argv) >= 2)): url = sys.argv[1] # print(“url=%s” % (url, )) print “IP address of %s=%s” % (url, getIpAddressFromUrl(url)) print “Company name in the URL=%s” % repr(getCompanyNameFromUrl(url)) elif ((__name__ == “__main__”) and (len(sys.argv)

It is evident from Listing 1 that the Python code is deliberately kept simple for clarity. This has two Python functions followed by a code snippet that will run when the Python script is executed using a Python interpreter. To verify that the code works as expected, this script can be run from a shell window: python UrlToIpAddress.py.

At the top of the file, Python modules, such as regular expression and socket, are imported. The first function is getCompanyNameFromUrl, which takes a web address as input. It does a regular expression match to parse the company name from the web address and returns the company

name to the caller. Next function is getIpAddressFromURL. It also takes a web address as input. It calls a Python socket API to get the IP address of the given web address. In particular, this Python API (gethostbyname) returns a tuple with three elements in it. These three elements provide hostname of the server for the given web address, alternate hostnames if any, and one or more IP addresses for that server. Instead of returning the tuple type to the caller, this function flattens the three tuple elements into a Python string by inserting a comma after each element. Then it returns the result as a string to the caller.

The purpose of this example is to learn about calling those two Python script functions from within a Streams application. We will focus on that in the following sections.

C++ logic

InfoSphere Streams allows for the inclusion of code written in C++ in two ways. One way is to build primitive Streams operators in C++, thereby incorporating the business logic written in C++. The other option is to execute any arbitrary C++ class methods directly from SPL as native functions. In this exercise, we will use the native function approach. To do that, we will create a separate C+ + project named StreamsToPythonLib, in which we will write the necessary code to call the Python functions we covered in the previous section. Then we will create a shared object (.so) library to make this C++ code available to the Streams SPL application.