CapeSoft.Com
Clarion Accessories
NetTalk
Doc Index
Creating Web Services
CapeSoft Logo

Creating Web Services

Download Latest Version
Installed Version Latest Version

Creating Web Services

Note: NetTalk Web Services requires xFiles

Introduction

There are a variety of common features that programs can offer that collectively are known as Web Services. NetTalk 8 introduced two new Procedure templates, called NetWebService and NetWebServiceMethod to assist with creating web services.

A NetWebService is the name for a collection of service methods.

In that application tree you can have multiple NetWebServiceMethods for a single NetWebService. The NetWebService is just a container, and does not include much more than just a list of the methods.

Your application can contain multiple NetWebService procedures (each one with one or more methods.)

All methods support a variety of encodings, for input and for output. Possible input encodings are XML (using xFiles), JSON (using jFiles) or Form-URL-Encoded. Possible output encodings are XML or JSON.

Supported Web Service standards include WSDL, SOAP and REST.

Example

In NetTalk 7 and earlier a hand-coded approach to web services was demonstrated in an example called SOAPServer (42). This example is included for reference purposes.

However a new example, WebService (77) is provided from NetTalk 8 and as it makes use of the new templates, is a better example for developing future WebServices.

Documentation

The service, and each method is self documenting. From any browser you can see the documentation for the service and for each method. The documentation includes as much information as possible so that other developers can easily make use of your service.

To see the documentation in your browser navigate to one of the following links;

www.whatever.com/servicename
www.whatever.com/servicename?help
www.whatever.com/methodname?help

www.whatever.com/servicename?methodname

WSDL

A WSDL file is a formal, computer-readable way of documenting a web service. Using this file, other programmers can write programs that make use of your service, and the existence of this file makes their lives much easier.

NetTalk automatically (and dynamically) generates this file for you when it is requested. A WSDL file for your service can be retrieved by using one of these links;

www.whatever.com/servicename?wsdl or
www.whatever.com/methodname?wsdl



Method Data

The method template has a place where you can set the incoming data fields, and the returning data fields. These fields need to be declared in your application either as tables, table fields, global data or local data.

Incoming fields can be any simple data type or a TABLE, GROUP or  QUEUE structure. (If a QUEUE structure is used then declaring it as a local queue is recommended - global queues are strongly discouraged.)

Like a normal Clarion procedure the method can take multiple incoming parameters and (depending on your code) the caller may only need to send some of them - in other words some of the incoming parameters may be optional.

If a Table is selected as an incoming parameter, then another parameter (tablename_action) is also expected from the caller. The action parameter informs the server of the nature of the change or fetch.

The template includes validation options for the parameters, but you can also add your own hand-code validation where necessary.

Unlike a normal Clarion procedure, the method can return multiple variables. Return values can be simple fields (including Local or Global variables), Queues, Groups, Tables and Views.

All methods can return one or more errors in place of the declared return values. If any errors exist, then  none of the other declared variables are returned. For more on errors see Errors.

All methods can also make use of the standard ServiceResult queue. The template code will especially make use of this when an incoming Table parameter is used. For more on standard results see Results.

Fields in Returning VIEWs

On of the Return types supported is a VIEW. Views are particularly useful here because they allow you to send a subset of a table (ie rows, or columns in the table can be suppressed.)

By default all [1] the fields in the Table are included in the view. You can override this by adding only the fields you want to export to the View Fields list in the template. The fields will be formatted according to the dictionary settings, unless this feature is disabled for this method, or globally.

[1] All is perhaps not quite true. You can set fields in the dictionary so they will not be included.

If the Field User Option NETAPI exists, and is set to 0, then this field will not be included. If it is set to 1 then the field is included (even if it is over another field.)

Using the NETAPI switch allows you to expressly exclude the "parent" field of the OVER, and expressly include the "child" field of the OVER. It can be very important to do this if you have a GROUP over a STRING.

Consider the following case;

ArriveStamp STRING(8) 
ArriveStampGroup GROUP,OVER(ArriveStamp) 
  ArriveDate DATE 
  ArriveTime TIME 
END 


In this situation, by default, the ArriveStamp field would be automatically included. This is almost certainly not desirable though because "Binary" data is not allowed in XML (it's ok in Json) and because unless the client is a Clarion program this string is meaningless anyway.

In this case it would be advantageous to set NETAPI to 0 for ArriveStamp.

Tables

Probably the most common use for a WebServiceMethod is to read and/or write database records. Since this is a common use-case, the templates are aware of this situation, and creating these sorts of methods is very straightforward.
  1. Create a NetWebServiceMethod procedure. Add it to a NetWebService as normal.
  2. Create an incoming parameter of type TABLE, and select a Table. Tick on the database actions you want to allow (Inserts, Updates etc)
  3. If the method allows the user to read the table then create a return value, of type VIEW. Give the view a name, tick on GenerateViewStructure and select the Table to view. If you leave the Fields list blank then all the fields from the table will be exported. Alternatively you can select a subset of fields to send to the client. Also be sure to enter an appropriate filter here.
The service will look for an additional parameter, called TableName_Action. This parameter is a string and should contain one of insert, update, get or delete.

Wizard

The wizard can now generate a web service and a web service method for each table that you select.

Calling Methods from a Client program

The above calls are typically made from a browser, and the goal is to determine what the service can do, and how to use it. To actually make use of the service though you need to call one of the methods.

A service is a collection of one or more methods. You can create your own methods using the NetWebServiceMethod procedure type. You can add as many methods as you like to a service, and your application can contain as many services (ie collections of methods) as you like.

Methods have a name, a list of incoming parameters, and a list of returned data. While a Clarion procedure only returns a single piece of data, a web method can return any amount of data, including GROUP and QUEUE structures.

The NetWebServiceMethod procedure type supports a number of calling techniques;

  1. Normal HTTP GET. The URL is of the form /service/method. Incoming parameters are passed as part of the URL or as cookies. The returned result is a simple XML structure. For example;

    GET /School/GetSchoolTeacher?FromTeacherID=value&ToTeacherId=value&Authenticate=value

    It's also possible to call the method using the PUT or DELETE commands instead of GET. For example;

    PUT /School/GetSchoolTeacher?TeacherID=value&Teacher
    Name=value&


    In the method itself you can determine which Verb was used by checking p_web.RequestMethodType. Typical possible values are one of NetWebServer_GET, NetWebServer_PUT, NetWebServer_POST and NetWebServer_DELETE.
  2. Normal HTTP POST. The URL is of the form /service/method. Incoming parameters are passed as Post Data, but in addition to the URL and Cookies, data can be passed in as plain "POST Data". For example;

    POST /School/GetSchoolTeacher HTTP/1.1
    Content-Type: application/x-www-form-urlencoded
    Content-Length: length

    FromTeacherID=value&ToTeacherId=value&Authenticate=value


  3. SOAP 1.1. The URL is of the form /service. In this case a POST is used, but the incoming Post Data is formatted as XML, and (optionally) wrapped in a SOAP envelope. A HTTP Header called SOAPAction: is also set to the method name.

    POST /School HTTP/1.1
    Host:
    somehost
    Content-Type: text/xml
    Content-Length:
    length
    SOAPAction: /GetSchoolTeacher

    <?xml version="1.0" encoding="utf-8"?>
    <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
      <soap:Body>
        <GetSchoolTeacher xmlns="https://www.capesoft.com">
          <FromTeacherID>value</FromTeacherID>
          <ToTeacherId>value</ToTeacherId>
          <Authenticate>
            <User>value</User>
            <Password>value</Password>
          </Authenticate>
          <AsAtDate>value</AsAtDate>
          <AsAtTime>value</AsAtTime>
        </GetSchoolTeacher>
      </soap:Body>
    </soap:Envelope>


  4. SOAP 1.2. The URL is of the form /service. the request is very similar to a SOAP 1.1 request, although the SOAPAction header is not included. the content-type of the SOAP 1.1 request is text/xml whereas the content-type for a SOAP 1.2 request is application/soap+xml. For example;

    POST /School HTTP/1.1
    Host:
    somehost
    Content-Type: application/soap+xml
    Content-Length:
    length

    <?xml version="1.0" encoding="utf-8"?>
    <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
      <soap:Body>
        <GetSchoolTeacher xmlns="https://www.capesoft.com">
          <FromTeacherID>value</FromTeacherID>
          <ToTeacherId>value</ToTeacherId>
          <Authenticate>
            <User>value</User>
            <Password>value</Password>
          </Authenticate>
          <AsAtDate>value</AsAtDate>
          <AsAtTime>value</AsAtTime>
        </GetSchoolTeacher>
      </soap:Body>
    </soap:Envelope>

CORS

If the API will be consumed by another web page, in other words consumed by JavaScript, then CORS will come into play. CORS (Cross Origin Resource Sharing) is a mechanism whereby a server "tells" a browser whether the resource can be consumed by the browser.

This only affects browsers - other clients will not be affected by this.

In order for your server to allow CORS requests, from other browsers, you will need to set the Access-Control-Allow-Origin header.


Your Code in the method

Services can include automatically generated methods (more on that in a moment) but a NetWebServiceMethod procedure doesn't actually do anything unless you add the necessary code.

In this sense a method can do "anything" that you code it to do. The template will parse the incoming request, and place it in the Parameter data structures. It will also format the return structures into XML or JSON, and return them to the caller. Your job is to write the code that populates the return structures.

You add your code to a routine called ServiceMethod. By the time this routine is called the parameter structures have been primed. Once this code is completed another (generated for you) routine will turn the result data structures into XML (or JSON) - you don't need to worry about that, you just write the code to populate the return structures with the correct value. Your code should not need to care about the technique used to call the method.

You add normal Clarion code here, opening tables, reading data and performing calculations, just like you would in any Clarion procedure.

You have access to the SessionQueue here for fetching and storing information using the normal p_web.GetSessionValue and p_web.SetSessionValue methods. However the incoming request will be bound to the session ONLY if the sessionID cookie is set in the request. Since the web client accessing this method is usually not a browser, the cookie may not automatically be set.

Aside: If you are using a Clarion program as a client, and you are using the NetWebClient class, then you can set the NetWebClient.OptionAutoCookie property so that multiple requests will preserve the cookies, and hence the session ID. If you are using a different tool for the client, then you will need to research that tool to determine how to send the cookie.

If your code does not interact with the session queue, then you don't need to worry about this.

Your code can (and should) generate Errors when things go wrong. See the next section for more on adding errors to your code.

Results

NetTalk contains a generic Results queue (p_web.ServiceResultQueue) which is used to pass information back to the caller. This queue is populated when the caller is adding, editing or removing records in the database, using the generated template code.

The Queue contains four fields;You can add your own information to the queue if you wish. The method to call is

p_web.AddServiceResult (action, tablename, recordId, description)

Errors

A method may fail for any number of reasons. Services should return detailed, meaningful error information to the client wherever possible. To this end NetTalk Service Methods include a built-in, automatic, always consistent queue of errors.

Errors can be generated automatically by the validation template settings, or they can be generated in your method code. To Generate a method simply call;

p_web.AddServiceError (Number, Position, RecordID, Description, Recommendation)

You are free to pass whatever you like to the AddServiceError method, but the more information passed to the client the better.

If any errors are added to the queue in this way, then only the errors list will be returned to the caller. None of the other return values will be included in the reply.

The ServiceErrorQueue is automatically added as a possible reply to the generated WSDL file for all methods in the service.

SOAP versus REST

As you can see from the above your method will happily accept incoming requests formatted as a SOAP XML packet. It is equally happy though to receive the request as a simple GET, PUT, POST or DELETE command. This is sometimes known as a REST request.

Your embed code does not change greatly between creating a SOAP service, or a REST service. REST is basically the same as SOAP without all the SOAP wrapping. Typically a REST client will also use the different HTTP verbs (GET, POST, PUT and DELETE) to match up to regular file activities. A SOAP service on the other hand will typically use a parameter to determine the file action to take.

The template generated code can easily handle both, for example in your embed code you might have some code like this;

(In this example a parameter called ACTION is assumed. If the parameter is sent, then loc:act is set from that, if the parameter is omitted, or set to 0, then the HTTP verb is checked and the loc:act based on that.)

loc:act = action
if loc:act = 0
    case p_web.RequestMethodType
    of 'GET'
        loc:act = Net:ViewRecord
    of 'DELETE'
        loc:act = Net:DeleteRecord
    of 'PUT'
        loc:act = Net:ChangeRecord
    of 'POST'
        loc:act = Net:InsertRecord
    end
end
   

To be a truly RESTful method you should not need to access the SessionQueue in order for the method to work. Ideally the client should pass you all the information you need in order for the method to work.

Return Format Rules

NetTalk API's can return results in either XML or JSON format. NetTalk applies a number of rules to decide which format to use. As soon as a rule is met then the decision is made, and no further rules are tested. The rules are as follows;
  1. If JSON is not supported by the method then return XML.
  2. If XML is not supported by the method then return JSON.
  3. If the incoming ACCEPT header specifies XML then return XML.
  4. If the incoming ACCEPT header specifies JSON then return JSON.
  5. If the incoming request type is SOAP 1.1 or SOAP 1.2 then return XML.
  6. If the incoming CONTENT-TYPE header is XML then return XML.
  7. If the incoming CONTENT-TYPE header is JSON then return JSON.
  8. If the local template setting is not DEFAULT then use local template setting
  9. Use global template setting

Overriding XML or JSON methods

The NetWebServiceMethod will generate two objects[1] in the procedure. The one (xml) supports XML structures, the other (json) supports JSON structures.

xml  xFileXML
json JSONClass


These are simple object declarations based on the xFiles xFileXML and the jFiles JSONClass classes respectively. These objects are used internally to parse incoming requests, and to create outgoing answers.

There may be times when it is desirable to override one or more methods in one, or both, of these objects. For example when parsing xml input it may be necessary to add embed code to the xFileXML.AssignField method (as described in the xFiles documentation.) To expose the embed points it is necessary for the object to rather be generated by the appropriate extension template.

For XML, go to the Extensions tab and add the xFiles IncludexFilesObject extension. Set the object name to xml. (This is important, it MUST have this name.) Then go to the settings for the method (Properties / Actions), to the General tab, and turn off the Generate XML Object option there. Doing these two steps means the xml object declaration is generated by the xFiles template, and not generated by the NetTalk template. Once this is done all the embed points for the xml object will be defined and you can embed code in them.

For JSON you follow the same process as for XML, but add the jFiles IncludejFilesObject extension and set the object name to json. Then on the General tab turn off the Generate JSON Object setting.

[1] The JSON object is only generated if the jFiles global extension has been added to the app.

Formatting Fields in the Response

From build 9.17, support for automatic formatting and deformatting of parameter and return fields is included.

This feature can be activated and deactivated globally. (It is on by default.)
It can be overridden at the field level in the Method.

Specifically fields which are stored as "not String" and have a picture, are automatically converted to strings (using the picture) and vice-versa.

For Parameters this feature is limited to parameter types TABLE, DATE, TIME, NUMBER and STRING.  GROUP, QUEUE, STRINGTHEORY and FILE are not supported.

For Return values this feature is limited to FIELD (from the dictionary, not local data), TABLE and VIEW return types. For VIEWs only Generated View fields are done automatically.

Embed points exist for both XML and JSON which allow you to format, and deformat fields in hand-code.
FormatTypeMethodCode Example
xmlparameterxml.AssignField Self.CurrentField = Deformat(pString,'@d6')
jsonparameterjson.DeformatValue Return Deformat(pValue,'@d6')
xmlreturn valuexml.SaveCurrentFieldToXML self.FormatCurrentField('@d6')
jsonreturn valuejson.FormatValue pLiteralType = json:string
Return clip(left(format(pValue,'@d6')))
For disconnected sync  the server, and the desktop need to be speaking the same language. For this reason the Desktop Sync client now defaults to auto formatting. If you disable auto formatting on the server side, then it needs to be disabled on the Desktop Client as well.

Currently the JavaScript sync is always formatted, so it is recommended that sync methods leave auto formatting on.

For more information on formatting the output of a field in xFiles, see here
https://www.capesoft.com/docs/xFiles/xfiles.htm#FormattingSaveField

For more information on formatting the output of a field in jFiles see here
https://www.capesoft.com/docs/jfiles/jfiles.htm#FormattingSavedValues

Field Prefixes

Field prefixes are a thorny problem for a couple of reasons. Firstly, they're a foreign concept in most languages, so they are often not desirable in a public API at all. Secondly they make use of a colon separator, which is not a valid character in a variable name in most languages.

For this reason you have the option to include, or exclude prefixes from parameters and return values. In most cases having the prefix OFF seems the better option.

For builds prior to 9.17 the default for prefixes was ON. For parameters and values added using 9.17 or later the default value is OFF. the setting for existing parameters and returns are not changed by the update, however support for the prefix support in the system has been overhauled so definitely check that your methods are still working the way you expect.

When writing a web app, colons are translated into a double underscore to make the names compatible with HTML. For ServiceMethods though, colons are translated into a single underscore. In builds 9.16 and earlier this was slightly inconsistent as simple GET and POST calls still made use of the double underscore, where XML and JSON used a single underscore. From 9.17 this has been made consistent so that all requests, and all responses use a single underscore if prefixes are ON.

Authentication

It is probable that many of the services you are providing will require that the user authenticate themselves in order for the action to complete.

As with normal Web apps, the method of authentication is left largely under your control - you can determine the best approach that suits your situation. Some approaches include (but are not limited to);
  1. Pass the Login and Password as fields on all incoming packets. These fields are then first checked in your code to verify the request before any other action is taken. The login and password could be passed as data in the request, or in the  HTTP Authentication header. If using headers then be sure to add code to your WebHandler procedure in the Authenticate method.
  2. Allow the user to log in with one request and set the session as logged in (as you would in a normal web app). Then further requests from the client can use the SessionID cookie with further requests. Note however that normal session timeout rules apply here - if no traffic from the service is received for a pre-determined period of time, then the session will timeout.
Bear in mind that because the service will typically be used by some service other than a browser more complicated authentication schemes are possible.

Basic Authentication

Basic Authentication is preferred over Digest Authentication when used over a TLS connection. To activate Basic Authentication in your application;
  1. Go to the WebServer procedure, Extensions, Settings, Security tab.
    Turn on the option Suggest Basic Authentication for Logged In Pages.
  2. Go to the WebHandler procedure, to the .Authenticate method. After the parent call add code to test the pUser and pPassword parameters. For example;

    Access:Users.Open()
    Access:Users.UseFile()
    User:Login = pUser
    If Access:Users.Fetch(User:Key) = level:Benign
      if pPassword = User:Password
        ReturnValue = true
      end
    End
    Access:Users.Close()

    NOTE: The above code is a simplification, Passwords should be stored as SALTed, HASHed values whenever possible.
While this code is ideal for an API, it is also possible to call .Authenticate from a LoginForm procedure, thus allowing this code to work for an API or for a Web App.

Digest Authentication

Digest Authentication is preferred to Basic Authentication when used over a non-secure connection. It does not transmit the password in plain-text, but rather a hash of the password. However it requires the server to store the actual password, not a hash of the password, which is not ideal. Then again, using any login over non-secure connections is not ideal. If you are considering Digest authentication rather consider a TLS connection, and if you have a TLS connection to the server use Basic rather than Digest authentication.

To implement Digest authentication;
  1. Go to the WebServer procedure, Extensions, Settings, Security tab.
    Turn on the option Suggest Digest Authentication for Logged In Pages.
  2. Go to the WebHandler procedure, to the .GetPassword method. After the parent call add code to fetch, and return, the users password, based on the pUser parameter. For example;

    Access:Users.Open()
    Access:Users.UseFile()
    User:Login = pUser
    If Access:Users.Fetch(User:Key) = level:Benign
        ReturnValue = User:Password
      end
    End
    Access:Users.Close()

Sessions

Authenticating a user on each request can be expensive. A SessionID is one way of "remembering" the user and "knowing" they are already logged in. NetTalk automatically creates sessions for all connections, and if the client makes use of the Session then (even if they pass authentication information) then the authentication code is not re-run.

Session ID's are included, as a SetCookie header, in every result returned by the server. If a client respects this cookie setting, and returns the cookie with each request, then no authentication is required, and the response will hence be faster. For this reason it is recommended that clients respect the cookie field and return the cookie.


[End of this document]
Return to NetTalk Documentation Index