The mamaDateTime API is limited to representing data ranges between 1970 and 2106 due to its underlying data structure and several of its methods.
The portion of the mamaDateTime data structure which represents seconds is used to calculate dates and stores the seconds elapsed since the Unix epoch as an unsigned 32 bit integer. As this value is unsigned, it is incapable of representing dates before 1970 and as the value is 32-bit, it has a maximum value of 4,294,967,295 which translates to early in 2106.
There are also several mamaDateTime methods which use unsigned 32 bit integers in their interface and therefore suffer from the same range limitation.
The most prevalent use case is needing to representing bonds which may have been issued before 1970 or which may not mature until after 2106.
There are several requirements which any solution being considered must satisfy:
One common theme for each of the design approaches considered below is the underlying data format used by mamaDateTime. Traditionally this has been an unsigned 64 bit integer, but all the solutions outlined below are based on the premise that this data structure will be replaced with a struct rather than try and hack the existing unsigned 64-bit integer to support extended data. Further details on the new data structure is provided later in the document.
This involves considering every MAMA datetime method available today, and if the method does not currently supported extended ranges (due to an insufficient function prototype or method signature), create new methods which do.
This involves adding the following methods in C:
mamaDateTime_setEpochTimeExmamaDateTime_setEpochTimeMillisecondsExmamaDateTime_setEpochTimeMicrosecondsExmamaDateTime_setWithHintsExmamaDateTime_setWithPrecisionAndTzmamaDateTime_setTimeExmamaDateTime_setTimeWithPrecisionAndTzmamaDateTime_getEpochTimeExmamaDateTime_getEpochTimeMicrosecondsExmamaDateTime_getEpochTimeMicrosecondsWithTzExmamaDateTime_getEpochTimeMillisecondsExmamaDateTime_getEpochTimeMillisecondsWithTzExmamaDateTime_getWithHintsExAnd the following methods in C++:
MamaDateTime::setEpochTimeMamaDateTime::setEpochTimeMillisecondsMamaDateTime::setEpochTimeMicrosecondsMamaDateTime::setWithHintsMamaDateTime::setMamaDateTime::setTimeMamaDateTime::getEpochTimeMicrosecondsNo other changes are required at C# or Java layer on the interface side.
This would involve adding extended methods but only for two methods so that there is some way available to read and write extended date times.
This means adding the following methods in C:
mamaDateTime_setEpochTimeExmamaDateTime_getEpochTimeExAnd adding the following methods in C++:
MamaDateTime::setEpochTimeMicrosecondsMamaDateTime::getEpochTimeMicrosecondsNo other changes are required at C# or Java layer on the interface side.
Note this was formerly related to using timeval, until we spotted some problems on windows
A method already exists to allow a timespec struct to be populated from a mamaDateTime object (mamaDateTime_getStructTimeSpec.
This approach involves adding a corresponding “setter” for this method, so that it can be used instead.
This means adding the following methods in C:
mamaDateTime_setFromStructTimeSpecNo other changes are required at C# or Java layer on the interface side.
struct timespec typically uses a 32 bit signed int
to represent seconds since epoch.At the interface level, adding timespec methods is the most straight forward, in that it simply involves the addition of a single new interface. It is also the method with the lowest effort involved and the method which conforms closest with POSIX standard conventions.
In this model, the existing mamaDateTime accessor methods would not be changed. Those which are capable at present of representing wider date
ranges will continue to do so, while those with a more limited range will continue to provide only values within that range. If an accessor is
called against a mamaDateTime structure which cannot be represented by that accessor, OpenMAMA will report an error condition.
C++ will be extended to also add support for a timespec accessor as a parallel to the underlying C addition. Java’s setTime and
set methods will need enhancement to support use of long second parameters, while enhancements to the underlying JNI interface
will be required to facilitate the propagation of extended range values. C# will similarly require some enhancements to support the
extended format internally, though the exposed C# DateTime format should already support larger values.
As all of the other language bindings are built on top of the C layer, this area will be documented the most heavily.
Each of the current C level accessors will need to be rewritten to parse the updated structure in order to return values consistent with the current implementation as well as extended data types.
At the interface level, adding timespec methods simply involves the addition of a two new functions,
mamaDateTime_setFromStructTimeSpec and mamaDateTime_getFromStructTimeSpec. The function prototypes will look like this:
MAMAExpDLL
extern mama_status
mamaDateTime_setFromStructTimeSpec(const mamaDateTime dateTime,
struct timespec* inputTimeSpec);
MAMAExpDLL
extern mama_status
mamaDateTime_getStructTimeSpec(const mamaDateTime dateTime,
struct timespec* result);
This will allow client applications to access extended time range values via the POSIX standard timespec structure. On each of OpenMAMA’s supported
platforms, timespec uses a time_t for storing second values, and a second signed long is used for microseconds. As such it is usually capable of
representing date ranges far in excess of those required by this RFC.
However, due to the size of time_t on 32 bit systems, client applications
on those systems will not be able to retrieve values within the extended date range via epoch-based methods. They can still use other non-epoch
based methods though such as mamaDateTime_getStructTm, mamaDateTime_getDateAsString, mamaDateTime_getMonth etc. without being impacted.
Internally the structure utilized by OpenMAMA for the storage of Date Time data will be modified from the existing 64 bit unsigned integer, to a new extended structure which is capable of handling the extended time range. The format of this structure will be completely abstracted by the accessor methods, and therefore the changes should be invisible to existing client applications.
The current mamaDateTime underlying data structure is a unsigned 64 bit int which is split out as follows:
| Data Content | Size | Comments |
|---|---|---|
| Seconds | 32-bit | Needs to be bigger to support times beyond 2106 and also has no ability to represent times before 1970. |
| Microseconds | 20-bit | Maximum value is 1,048,575 - if nanosecond support is ever required (planning ahead), this would be too small (you’d need another 10 bits) |
| Precision | 4-bit | Used to store decimal precision. |
| Hints | 4-bit | Used to store information on whether or not to display as date / time (one of these bits going spare too, could expand to 2 with changes) |
| Spare (idle) | 4-bit | Could be 6 if you count the values spare in hints |
It is usually heap-allocated by mamaDateTime_create(), but because it’s typedef’d to a mama_u64_t*, bridges tend to use direct access
to grab a serialized version of the data structure.
typedef mama_u64_t* mamaDateTime;
Using a mama_u64_t for datetime always felt like we were going to unnecessary lengths to treat the MAMA Date Time object like a serialized format,
when realistically it’s a mapping format which acts as an intermediary between the payload bridge and MAMA which tends to end up dealing
with mediating variables anyway. You would also need some pretty creative bit shifting to make use of those spare bits and it would only
realistically buy us some short term time until nanosecond or TZ support is required. As a further side effect, it’s very convoluted to
deal with which makes it difficult to debug.
Therefore the existing data structure is not being considered for this solution.
To make the new data structure easier to maintain, the following members are proposed:
| Data Content | Data Type | Comments |
|---|---|---|
| Seconds | int64 (yes - signed) | Slightly wasteful if you’re trying to make every bit earn its keep, but easy to work with and will support times prior to 1970. Note that C11 timespec struct uses time_t which I would have preferred but looks like it is only guaranteed to be 64 bit on 64 bit platforms. |
| Nanoseconds | long int | Note nanoseconds as a whole number is native to C11 timespec which uses a long int to store nanoseconds, so suggesting that data format here |
| Precision | uint8 | Again, some bits going spare, but easy to work with. |
| Hints | uint8 | Again, some bits going spare, but easy to work with. |
So total data size is 112 bytes. Note that the format is reasonably close to timespec, and when all supported systems are 64-bit, we may modify this to simply
use a timespec as a member.
The mamaDateTime object will also be made opaque to avoid the temptation to attempt direct access to the underlying data type:
typedef void* mamaDateTime;
There are currently over 100 unit tests around MAMA Date Time. As each method will effectively be getting re-written to use the new data structure, each function changed will need to be accompanied with a corresponding set of unit test changes to cover:
The methods concerned here are very self contained and should be covered entirely by unit tests. As a result of this, nothing beyond standard regression testing should be required.
A single new method is required here - MamaDateTime::setFromStructTimeSpec. Everything else should remain unchanged in the eyes of the application. The method signature
will be:
setFromStructTimeSpec (struct timespec& value);
As this is built on top of the C layer, no additional changes are required here.
As each method will effectively be getting re-written to use the new data structure, each function changed will need to be accompanied with a corresponding set of unit test changes to cover:
The methods concerned here are very self contained and should be covered entirely by unit tests. As a result of this, nothing beyond standard regression testing should be required.
Code changes will be required in the native layer to integrate with the new C methods for extended ranges.
No method changes are required as the interface uses standard C# DateTime format which already supports extended ranges.
As this is built on top of the C layer, no additional changes are required here.
Each function will need to be accompanied with a corresponding set of unit test changes to cover:
Code changes will be required in the native layer to integrate with the new C methods for extended ranges.
MamaDateTime.setTime and MamaDateTime.set will require a new signature to allow for seconds as a long. Otherwise no
other changes are required.
As this is built on top of the C layer, no additional changes are required here.
Each function will need to be accompanied with a corresponding set of unit test changes to cover:
The methods concerned here are very self contained and should be covered entirely by unit tests. As a result of this, nothing beyond standard regression testing should be required.
New methods available as documented in the summary details for users who require access to the extended datetime format.
As this change will effectively break the bridge interface for any payload bridges which use direct access to the underlying mama_u64_t data type. This means that the minor version in OpenMAMA will need to change to reflect this as per our versioning standard.
The payload bridge will be responsible for ensuring that:
mamaDateTime_setFromStructTimeSpec is avoided in favour of non-epoch based methods such as mamaDateTime_setDate and
mamaDateTime_setTime. This will ensure that no gibberish time fields get generatedAfter the upgrade, bridge developers will need to stop doing shortcut hacks like this:
void setFieldValue (payloadlibField field, mamaDateTime value)
{
/* dereferencing mamaDateTime to access underlying u64 */
payloadlib_setWireFormatPtr (field, *value);
}
And instead do something more like this to honor the opacity of mamaDateTime as many payload bridges already do today:
void setFieldValue (payloadlibField field, mamaDateTime value)
{
payloadlib_setWireFormatDateStamp (field, value.tv_sec, value.tv_usec);
}
(note the same applies to all mamaDateTime methods invoked from the bridge).
No other internal API components are impacted by these changes.