All major classes, like those responsible for persistence, business logic, and presentation, are mapped as Spring managed beans. “Bean” is Spring terminology and simply refers to a class that is instantiated, assembled, and otherwise managed by the Spring IoC container. Dependencies between beans are injected by the IoC container, which allows for loose coupling, re-configuration and testability. For documentation on Spring, please refer to springframework.org.
The services found in the dhis-service-core project basically provide methods that delegate to a corresponding method in the persistence layer, or contain simple and self-explanatory logic. Some services, like the ones found in the dhis-service-datamart, dhis-service-import-export, dhis-service-jdbc, and dhis-service-reporting projects are more complex and will be elaborated in the following sections.
The JDBC service project contains a set of components dealing with JDBC connections and SQL statements.
Fig. JDBC BatchHandler diagram
The BatchHandler interface provides methods for inserting, updating and verifying the existence of objects. The purpose is to provide high-performance operations and is relevant for large amounts of data. The BatchHandler object inserts objects using the multiple insert SQL syntax behind the scenes and can insert thousands of objects on each database commit. A typical use-case is an import process where a class using the BatchHandler interface will call the addObject( Object, bool ) method for every import object. The BatchHandler will after an appropriate number of added objects commit to the database transparently. A BatchHandler can be obtained from the BatchHandlerFactory component. BatchHandler implementations exist for most objects in the API.
The JdbcConfiguration interface holds information about the current DBMS JDBC configuration, more specifically dialect, driver class, connection URL, username and password. A JdbcConfiguration object is obtained from the JdbcConfigurationProvider component, which currently uses the internal Hibernate configuration provider to derive the information.
The StatementBuilder interface provides methods that represents SQL statements. A StatementBuilder object is obtained from the StatementBuilderFactory, which is able to determine the current runtime DBMS and provide an appropriate implementation. Currently implementations exist for PostgreSQL, MySQL, H2, and Derby.
The IdentifierExtractor interface provides methods for retrieving the last generated identifiers from the DBMS. An IdentifierExtractor is obtained from the IdentifierExtractorFactory, which is able to determine the runtime DBMS and provide an appropriate implementation.
Fig. JDBC StatementManager diagram
The StatementHolder interface holds and provides JDBC connections and statements. A StatementHolder object can be obtained from the StatementManager component. The StatementManager can be initialized using the initalise() method closed using the destroy() method. When initialized, the StatementManager will open a database connection and hold it in a ThreadLocal variable, implying that all subsequent requests for a StatementHolder will return the same instance. This can be used to improve performance since a database connection or statement can be reused for multiple operations. The StatementManager is typically used in the persistence layer for classes working directly with JDBC, like the DataMartStore.
The import-export project contains classes responsible for producing and consuming interchange format files. The import process has three variants which are import, preview and analysis. Import will import data directly to the database, preview will import to a temporary location, let the user do filtering and eventually import, while the analysis will reveal abnormalities in the import data. Currently supported formats are:
-
DXF (DHIS eXchange Format)
-
IXF (Indicator eXchange Format)
-
DHIS 1.4 XML format
-
DHIS 1.4 Datafile format
-
CSV (Comma Separated Values)
-
PDF (Portable Document Format)
Fig. Import-export service diagram
The low-level components doing the actual reading and writing of interchange format files are the converter classes. The most widely used is the XmlConverter interface, which provides a write( XmlWriter, ExportParams ) and a read( XmlReader, ImportParams ) method. Most objects in the API have corresponding XmlConverter implementations for the DXF format. Writing and reading for each object is delegated to its corresponding XmlConverter implementation.
The ExportParams object is a specification which holds the identifiers of the objects to be exported. The converter retrieves the corresponding objects and writes content to the XmlWriter. XmlConverter implementations for the DXF format exist for most objects in the API. For instance, the write method of class DataElementConverter will write data that represents DataElements in DXF XML syntax to the XmlWriter.
The ExportService interface exposes a method InputStream exportData( ExportParams ). The ExportService is responsible for instantiating the appropriate converters and invoke their export methods. To avoid long requests prone to timeout-errors in the presentation layer, the actual export work happens in a separate thread. The ExportService registers its converters on the ExportThread class using its registerXmlConverter( XmlConverter ) method, and then starts the thread.
The ImportParams obect contains directives for the import process, like type and strategy. For instance, the read method of class DataElementConverter will read data from the XmlReader, construct objects from the data and potentially insert it into the database, according to the directives in the ImportParams object.
The ImportService interface exposes a method importData( ImportParams, InputStream ). The ImportService is responsible for instantiating the appropriate converters and invoke their import methods. The import process is using the BatchHandler interface heavily.
The ImportExportServiceManager interface provides methods for retrieving all ImportServices and ExportServices, as well as retrieving a specific ImportService or ExportService based on a format key. This makes it simple to retrieve the correct service from using classes since the name of the format can be used as parameter in order to get an instance of the corresponding service. This is implemented with a Map as the backing structure where the key is the format and the value is the Import- or ExportService reference. This map is defined in the Spring configuration, and delegates to Spring to instantiate and populate the map. This allows for extensibility as developing a new import service is simply a matter of providing an implementing the ImportService interface and add it to the map definition in the Spring configuration, without touching the ImportExportServiceManager code.
Fig. Import-export converter diagram
Functionality that is general for converters of all formats is centralized in abstract converter classes. The AbstractConverter class provides four abstract methods which must be implemented by using converters, which are importUnique( Object ), importMatching( Object, Object), Object getMatching() and boolean isIdentical( Object, Object ). It also provides a read( Object, GroupMemberType, ImportParams ) method that should be invoked by all converters at the end of the read process for every object. This method utilizes the mentioned abstract methods and dispatches the object to the analysis, preview or import routines depending on the state of the object and the current import directives. This allows for extensibility as converters for new formats can extend their corresponding abstract converter class and reuse this functionality.
The data mart component is responsible for producing aggregated data from the raw data in the time and space dimension. The aggregated data is represented by the AggregatedDataValue and AggregatedIndicatorValue objects. The DataSetCompletenessResult object is also included in the data mart and is discussed in the section covering the reporting project. These objects and their corresponding database tables are referred to as the data mart.
The following section will list the rules for aggregation in DHIS 2.
-
Data is a aggregated in the time and space dimension. The time dimension is represented by the Period object and the space dimension by the OrganisationUnit object, organised in a parent-child hierarchy.
-
Data registered for all periods which intersects with the aggregation start and end date is included in the aggregation process. Data for periods which are not fully within the aggregation start and end date is weighed according to a factor “number of days within aggregation period / total number of days in period”.
-
Data registered for all children of the aggregation OrganisationUnit is included in the aggregation process.
-
Data registered for a data element is aggregated based on the aggregation operator and data type of the data element. The aggregation operator can be sum (values are summarized), average (values are averaged) and count (values are counted). The data type can be string (text), int (number), and bool (true or false). Data of type string can not be aggregated.
-
Aggregated data of type sum – int is presented as the summarized value.
-
Aggregated data of type sum – bool is presented as the number of true registrations.
-
Aggregated data of type average – int is presented as the averaged value.
-
Aggregated data of type average – bool is presented as a percentage value of true registrations in proportion to the total number of registrations.
-
-
An indicator represents a formula based on data elements. Only data elements with aggregation operator sum or average and with data type int can be used in indicators. Firstly, data is aggregated for the data elements included in the indicator. Finally, the indicator formula is calculated.
-
A calculated data element represents a formula based on data elements. The difference from indicator is that the formula is on the form “data element * factor”. The aggregation rules for indicator apply here as well.
Fig. Data mart diagram
The AggregationCache component provides caching in ThreadLocal variables. This caching layer is introduced to get optimal caching [9]. The most frequently used method calls in the data mart component is represented here.
The DataElementAggregator interface is responsible for retrieving data from the crosstabulated temporary storage and aggregate data in the time and space dimension. This happens according to the combination of data element aggregation operator and data type the class represents. One implementation exist for each of the four variants of valid combinations, namely SumIntDataElementAggregator, SumBoolDataElementAggregator, AverageIntDataElementAggregator and AverageBoolAggregtor.
The DataElementDataMart component utilizes a DataElementAggregator and is responsible for writing aggregated data element data to the data mart for a given set of data elements, periods, and organisation units.
The IndicatorDataMart component utilizes a set of DataElementAggregators and is responsible for writing aggregated indicator data to the data mart for a given set of indicators, periods, and organisation units.
The CalculatedDataElementDataMart component utilizes a set of DataElementAggregators and is responsible for writing aggregated data element data to the data mart for a given set of calculated data elements, periods, and organisation units.
The DataMartStore is responsible for retrieving aggregated data element and indicator data, and data from the temporary crosstabulated storage.
The CrossTabStore is responsible for creating, modifying and dropping the temporary crosstabulated table. The CrossTabService is responsible for populating the temporary crosstabulated table. This table is used in an intermediate step in the aggregation process. The raw data is de-normalized on the data element dimension, in other words the crosstabulated table gets one column for each data element. This step implies improved performance since the aggregation process can be executed against a table with a reduced number of rows compared to the raw data table.
The DataMartService is the central component in the data mart project and controls the aggregation process. The order of operations is:
-
Existing aggregated data for the selected parameters is deleted.
-
The temporary crosstabulated table is created and populated using the CrossTabService component.
-
Data element data for the previously mentioned valid variants is exported to the data mart using the DataElementDataMart component.
-
Indicator data is exported to the data mart using the IndicatorDataMart component.
-
Calculated data element data is exported to the data mart using the CalculatedDataElementDataMart component.
-
The temporary crosstabulated table is removed.
The data element tables are called “aggregateddatavalue” and “aggregatedindicatorvalue” and are used both inside DHIS 2 for e.g. report tables and by third-party reporting applications like MS Excel.
The reporting project contains components related to reporting, which will be described in the following sections.
The ReportTable object represents a crosstabulated database table. The table can be crosstabulated on any number of its three dimensions, which are the descriptive dimension (which can hold data elements, indicators, or data set completeness), period dimension, and organisation unit dimension. The purpose is to be able to customize tables for later use either in third-party reporting tools like BIRT or directly in output formats like PDF or HTML inside the system. Most of the logic related to crosstabulation is located in the ReportTable object. A ReportTable can hold:
-
Any number of data elements, indicators, data sets, periods, and organisation units.
-
A RelativePeriods object, which holds 10 variants of relative periods. Examples of such periods are last 3 months, so far this year, and last 3 to 6 months. These periods are relative to the reporting month. The purpose of this is to make the report table re-usable in time, i.e. avoid the need for the user to replace periods in the report table as time goes by.
-
A ReportParams object, which holds report table parameters for reporting month, parent organisation unit, and current organisation unit. The purpose is to make the report table re-usable across the organisation unit hierarchy and in time, i.e. make it possible for the user to re-use the report table across organisation units and as time goes by.
-
User options such as regression lines. Value series which represents regression values can be included when the report table is crosstabulated on the period dimension.
Fig. Report table diagram
The ReportTableStore is responsible for persisting ReportTable objects, and currently has a Hibernate implementation.
The ReportTableService is responsible for performing business logic related to report tables such as generation of relative periods, as well as delegating CRUD operations to the ReportTableStore.
The ReportTableManager is responsible for creating and removing report tables, as well as retrieving data.
The ReportTableCreator is the key component, and is responsible for:
-
Exporting relevant data to the data mart using the DataMartExportService or the DataSetCompletenessService. Data will later be retrieved from here and used to populate the report table.
-
Create the report table using the ReportTableManager.
-
Include potential regression values.
-
Populate the report table using a BatchHandler.
-
Remove the report table using the ReportTableManager.
The Chart object represents preferences for charts. Charts are either period based or organisation unit based. A chart has tree dimensions, namely the value, category, and filter dimension. The value dimension contains any numbers of indicators. In the period based chart, the category dimension contains any number of periods while the filter dimension contains a single organisation unit. In the organisation unit based chart, the category dimension contains any number of organisation units while the filter dimension contains a single period. Two types of charts are available, namely bar charts and line charts. Charts are materialized using the JFreeChart library. The bar charts are rendered with a BarRenderer [2], the line charts with a LineAndShapeRenderer [2], while the data source for both variants is a DefaultCategoryDataSet [3]. The ChartService is responsible for CRUD operations, while the ChartService is responsible for creating JfreeCharts instances based on a Chart object.
Fig. Chart diagram
The purpose of the data set completeness functionality is to record the number of data sets that have been completed. The definition of when a data set is complete is subjective and based on a function in the data entry screen where the user can mark the current data set as complete. This functionality provides a percentage completeness value based on the number of reporting organisation units with completed data sets compared to the total number of reporting organisation units for a given data set. This functionality also provides the number of completed data sets reported on-time, more specifically reported before a defined number of days after the end of the reporting period. This date is configurable.
Fig. Data set completeness diagram
The CompleteDataSetRegistration object is representing a data set marked as complete by a user. This property holds the data set, period, organisation unit and date for when the complete registrations took place. The CompleteDataSetRegistrationStore is responsible for persistence of CompleteDataSetRegistration objects and provides methods returning collections of objects queried with different variants of data sets, periods, and organisation units as input parameters. The CompleteDataSetRegistrationService is mainly delegating method calls the store layer. These components are located in the dhis-service-core project.
The completeness output is represented by the DataSetCompletenessResult object. This object holds information about the request that produced it such as data set, period, organisation unit, and information about the data set completeness situation such as number of reporting organisation units, number of complete registrations, and number of complete registrations on-time. The DataSetCompletenessService is responsible for the business logic related to data set completeness reporting. It provides methods which mainly returns collections of DataSetCompletenessResults and takes different variants of period, organisation unit and data set as parameters. It uses the CompleteDataSetRegistrationService to retrieve the number of registrations, the DataSetService to retrieve the number of reporting organisation units, and performs calculations to derive the completeness percentage based on these retrieved numbers.
The DataSetCompletenessExportService is responsible for writing DataSetCompletenessResults to a database table called “aggregateddatasetcompleteness”. This functionality is considered to be part of the data mart as this data can be used both inside DHIS 2 for e.g. report tables and by third-party reporting applications like MS Excel. This component is retrieving data set completeness information from the DataSetCompeletenessService and is using the BatchHandler interface to write such data to the database.
The Document object represents either a document which is uploaded to the system or a URL. The DocumentStore is responsible for persisting Document objects, while the DocumentService is responsible for business logic.
Fig. Document diagram
The PivotTable object represents a pivot table. It can hold any number of indicators, periods, organisation units, and corresponding aggregated indicator values. It offers basic pivot functionality like pivoting and filtering the table on all dimensions. The business logic related to pivot tables is implemented in Javascript and is located in the presentation layer. The PivotTableService is reponsible for creating and populating PivotTable objects.
The LocationManager component is responsible for the communication between DHIS 2 and the file system of the operating system. It contains methods which provide read access to files through File and InputStream instances, and write access to the file system through File and OutputStream instances. The target location is relative to a system property “dhis2.home” and an environment variable “DHIS2_HOME” in that order. This component is used e.g. by the HibernateConfigurationProvider to read in the Hibernate configuration file, and should be re-used by all new development efforts.
The ConfigurationManager is a component which facilitates the use of configuration files for different purposes in DHIS 2. It provides methods for writing and reading configuration objects to and from XML. The XStream library is used to implement this functionality. This component is typically used in conjunction with the LocationManager.
The system support project contains supportive classes that are general and can be reused througout the system.
The deletion manager solution is responsible for deletion of associated objects. When an object has a depdency to another object this association needs to be removed by the application before the latter object can be deleted (unless the association is defined to be cascading in the DBMS). Often an object in a peripheral module will have an associations to a core object. When deleting the core object this association must be removed before deleting the core object.The core module cannot have a dependency to the peripheral module however due to the system design and the problem of cyclic dependencies. The deletion manager solves this by letting all objects implement a DeletionHandler which takes care of associations to other objects. A DeletionHandler should override methods for objects that, when deleted, will affect the current object in any way. The DeletionHandler can choose to disallow the deletion completely by overriding the allowDelete* method, or choose to allow the deletion and remove the associations by overriding the delete* method. Eg. a DeletionHandler for DataElementGroup should override the deleteDataElement(..) method which should remove the DataElement from all DataElementGroups. If one decide that DataElement which are a member of any DataElementGroups cannot be deleted, it should override the allowDeleteDataElement() method and return false if there exists DataElementGroups with associations to that DataElement.
First, all DeletionHandler implementations are registered with the DeletionManager through a Spring MethodInvokingFactoryBean in the Spring config file. This solution adheres to the observer design pattern.
Second, all method invocations that should make the DeletionManager execute are mapped to the DeletionInterceptor with Spring AOP advice in the Spring config file. The DeletionInterceptor in turn invokes the execute method of the DeletionManager. First, the DeletionManager will through reflection invoke the allowDelete* method on all DeletionHandlers. If no DeletionHandlers returned false it will proceed to invoke the delete* method on all DeletionHandlers. This way all DeletionHandlers get a chance to clean up associations to the object being deleted. Finally the object itself is deleted.